hi all,

great to read about this, I'd like to join in! Can I just join using the
zoom link tomorrow or do I need an invitation? (If I do need one, please
invite me :))

cheers


On Wed, Apr 14, 2021 at 8:15 PM Daniel Imberman <daniel.imber...@gmail.com>
wrote:

> Thank you Ian,
>
> I’ve invited everyone on this thread to the meeting with that zoom link.
> Anyone else who wants to join can add the calendar event here
> calendar.google.com/event?action=TEMPLATE&tmeid=Mm4zN2Q3MnFwNnBqbW9hMmNocXMyNzJpdHYgZGFuaWVsQGFzdHJvbm9tZXIuaW8&tmsrc=dan...@astronomer.io
> <https://calendar.google.com/event?action=TEMPLATE&tmeid=Mm4zN2Q3MnFwNnBqbW9hMmNocXMyNzJpdHYgZGFuaWVsQGFzdHJvbm9tZXIuaW8&tmsrc=daniel%40astronomer.io>
>
> On Wed, Apr 14, 2021 at 11:05 AM, Ian Buss <ianjb...@gmail.com> wrote:
>
> If this works for everyone, here's a zoom link for Thursday 8AM PST:
> https://cloudera.zoom.us/j/99928254235?pwd=VTFlQk4vQjQ5Z2JzUDM3ZWZKKy9MQT09
>
> Happy to move or use an alternate method as needed.
>
> On Wed, Apr 14, 2021 at 6:58 PM Daniel Imberman <daniel.imber...@gmail.com>
> wrote:
>
>> Thursday works for me!
>>
>> On Wed, Apr 14, 2021 at 10:05 AM, Ian Buss <ianjb...@gmail.com> wrote:
>>
>> Hi all,
>>
>> I actually can’t do Wednesday next week as I’m moving house :) Any chance
>> we could do Thursday or Friday at the same time?
>>
>> Cheers
>>
>> Ian
>> On 14 Apr 2021, 17:49 +0100, Kaxil Naik <kaxiln...@gmail.com>, wrote:
>>
>> Just few comments here:
>>
>> Currently -- atleast for the foreseeable future Airflow workers will need
>> access to the DAG Files, so workers can not run using the Serialized DAGs.
>>
>> Also serialized DAGs do not even have all the info needed for it to run
>> it. Currently the serialization happens in the parsing process in the
>> scheduler which can be in future separated as a separator "parsining"
>> component, but that won't solve the "isolation" problem you are trying to
>> solve. The only current way it can be solved is pickling -- and we have
>> strictly decided against using pickling for DAGs.
>>
>> The idea in Statement (2) & (3) would help solve the isolation problem in
>> (1) and can be done with some work now.
>>
>> Happy to talk about it in more detail here or on call, the time Daniel
>> suggested works for me.
>>
>> Regards,
>> Kaxil
>>
>> On Wed, Apr 14, 2021 at 5:35 PM Daniel Imberman <
>> daniel.imber...@gmail.com> wrote:
>>
>>> How about Wednesday, April 21 at 8:00AM PST?
>>>
>>> On Wed, Apr 14, 2021 at 9:33 AM, Xinbin Huang <bin.huan...@gmail.com>
>>> wrote:
>>>
>>> I am available any days.
>>>
>>> On Wed, Apr 14, 2021, 9:32 AM Daniel Imberman <daniel.imber...@gmail.com>
>>> wrote:
>>>
>>>> Hi everyone!
>>>>
>>>> Would people be available around 8AM/9AM PST some point next week? I’m
>>>> in PST and Ian is UTC+1 so would be great to find a timezone that
>>>> accomodates everyone.
>>>>
>>>> Daniel
>>>> On Wed, Apr 14, 2021 at 6:26 AM, Ryan Hatter <ryannhat...@gmail.com>
>>>> wrote:
>>>>
>>>> I’d also like to be added please :)
>>>>
>>>> On Apr 13, 2021, at 21:27, Xinbin Huang <bin.huan...@gmail.com> wrote:
>>>>
>>>> 
>>>> Hi Daniel & Ian,
>>>>
>>>> I am also interested in the idea of a serialization representation that
>>>> can be executed by workers directly. Can you also add me to the call?
>>>>
>>>> Thanks
>>>> Bin
>>>>
>>>> On Tue, Apr 13, 2021 at 2:49 PM Ian Buss <ianjb...@gmail.com> wrote:
>>>>
>>>>> Daniel,
>>>>>
>>>>> Thanks for your warm welcome and quick response and the advice on
>>>>> providers! Will certainly check out the examples you sent.
>>>>>
>>>>> 1. An "airflow register" command definitely sounds promising, would
>>>>> love to collaborate on an AIP there so let's set something up.
>>>>> 2. We use KubernetesExecutor exclusively as well. We've noticed
>>>>> significant additional load on the metadata DB as we scale up task pods so
>>>>> I've also thought about an API-based approach. Such an API could also open
>>>>> up the possibility of per-task security tokens which are injected by the
>>>>> scheduler, which should improve the security of such a system. Food for
>>>>> thought at least. I will start putting some of these thoughts down on 
>>>>> paper
>>>>> in a sharable format.
>>>>>
>>>>> Ian
>>>>>
>>>>> On Tue, Apr 13, 2021 at 7:46 PM Daniel Imberman <
>>>>> daniel.imber...@gmail.com> wrote:
>>>>>
>>>>>> Hi Ian,
>>>>>>
>>>>>>
>>>>>> Firstly, welcome to the Airflow community :). I'm glad to hear you've
>>>>>> had a positive experience so far. It's great to hear that you want to
>>>>>> contribute back, and I think that multi-tenancy/DAG isolation is a pretty
>>>>>> fantastic project for the community as a whole (a lot of things are are
>>>>>> things we want but are limited by hours in a day).
>>>>>>
>>>>>>
>>>>>> 1. I've personally been kicking around some ideas lately about an
>>>>>> "airflow register" command that would write the DAG into the metadata DB 
>>>>>> in
>>>>>> a way that could be "gettable" by the workers via the API. This work is
>>>>>> very early. I'd love to get some help on it. Perhaps we can set up a zoom
>>>>>> chat to discuss drafting an AIP?
>>>>>>
>>>>>>
>>>>>> 2. Limiting worker access to the DB is not only good security
>>>>>> practice; it also opens up the door to a lot of valuable features. This
>>>>>> feature would be especially close to my heart as it would make the
>>>>>> KubernetesExecutor significantly more efficient. It should be possible to
>>>>>> set up a system where the workers only ever speak to an API server and
>>>>>> never need to touch the DB.
>>>>>>
>>>>>>
>>>>>> 3. This is not something I personally have insight into, but I think
>>>>>> it sounds like a good idea.
>>>>>>
>>>>>>
>>>>>> Finally, addressing your question about a Cloudera provider. If
>>>>>> anything, it would probably give the provider _more_ legitimacy if you
>>>>>> hosted it under the Cloudera GitHub org (we very purposely created the
>>>>>> provider packages with this workflow in mind). There are multiple places
>>>>>> where we can work to surface this provider so it is easy to find and use.
>>>>>>
>>>>>>
>>>>>> Astronomer has a pretty good sample provider here
>>>>>> <https://github.com/astronomer/airflow-provider-sample>. One example
>>>>>> of it running in the wild is the Great Expectations provider here
>>>>>> <https://github.com/great-expectations/airflow-provider-great-expectations>.
>>>>>> I'd also be glad to get you in contact with people who have built 
>>>>>> providers
>>>>>> in the past to help you with that process.
>>>>>>
>>>>>>
>>>>>> Looking forward to seeing some of these things come to fruition!
>>>>>>
>>>>>>
>>>>>> Daniel
>>>>>>
>>>>>> On Tue, Apr 13, 2021 at 9:43 AM, Ian Buss <ianjb...@gmail.com> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> First a quick introduction: I'm an engineer with Cloudera working on
>>>>>> our Data Engineering product (CDE). Airflow is working great for us so 
>>>>>> far.
>>>>>> We've been looking into how we can enhance the multi-tenancy story of
>>>>>> Apache Airflow as we currently deploy it. We have the following areas 
>>>>>> which
>>>>>> we'd like (with community consensus) to work on and contribute back to
>>>>>> Apache Airflow to enhance the isolation between tenants in a single 
>>>>>> Airflow
>>>>>> deployment.
>>>>>>
>>>>>> 1. Isolating code execution and parsing of DAG files. At the moment,
>>>>>> DAG files are parsed in a few locations in Airflow, including the 
>>>>>> scheduler
>>>>>> and in tasks. There is already the concept of DAG serialization (and 
>>>>>> we're
>>>>>> using that for the web component) but we'd be interested to see if we can
>>>>>> sandbox the execution of arbitrary user code to a locked down
>>>>>> process/container without full access to the metadata DB and connection
>>>>>> secrets etc. The idea would be to parse and serialize the DAG in this
>>>>>> isolated container and pass back a serialized representation for
>>>>>> persistence in the DB. Has anyone explored this idea?
>>>>>>
>>>>>> 2. Limiting task access to the metadata DB. It would be great if we
>>>>>> could remove the requirement for tasks to have full access to the 
>>>>>> metadata
>>>>>> DB and to report task status in a different (but still scalable) way. 
>>>>>> We'd
>>>>>> need to tackle access or injection of connection, variable and xcom data 
>>>>>> as
>>>>>> well for each task naturally.
>>>>>>
>>>>>> 3. Finer-grained access controls on connection secrets. Right now,
>>>>>> although there are nice at-rest encryption options with Fernet or Vault,
>>>>>> IIUC any DAG can access any connection (and thus any secret). Since the
>>>>>> "run as" user is largely defined within the DAG and its tasks, this is
>>>>>> challenging for a multi-tenant environment (see caveat below)
>>>>>>
>>>>>> Caveat: It's definitely noted that to some extent we should assume
>>>>>> that an Airflow deployment is a "trusted" environment and that best
>>>>>> practices such as git+PR workflows are the gold standard and that any
>>>>>> malicious code and dependencies should be identified through this 
>>>>>> process.
>>>>>> Also that there is a clear admin role for connection management etc.
>>>>>>
>>>>>> We have some ideas informally sketched out as to how to address the
>>>>>> above but would be keen to hear the community opinion on this and to see 
>>>>>> if
>>>>>> anyone is keen to collaborate on designs and implementation, or to hear 
>>>>>> if
>>>>>> anything is already in the works. In particular I noticed that the very
>>>>>> first improvement proposal (AIP-1) addresses much of the above :). 
>>>>>> However,
>>>>>> it seems fairly dormant at the moment.
>>>>>>
>>>>>> One other question: we have a provider (operators and hooks) for
>>>>>> interacting with Cloudera components that we'd like to contribute to the
>>>>>> project. The provider FAQs indicate that new provider contributions are
>>>>>> still welcome in the project in 2.x, is that accurate?
>>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>> Ian
>>>>>>
>>>>>>

Reply via email to