Hi all, First a quick introduction: I'm an engineer with Cloudera working on our Data Engineering product (CDE). Airflow is working great for us so far. We've been looking into how we can enhance the multi-tenancy story of Apache Airflow as we currently deploy it. We have the following areas which we'd like (with community consensus) to work on and contribute back to Apache Airflow to enhance the isolation between tenants in a single Airflow deployment.
1. Isolating code execution and parsing of DAG files. At the moment, DAG files are parsed in a few locations in Airflow, including the scheduler and in tasks. There is already the concept of DAG serialization (and we're using that for the web component) but we'd be interested to see if we can sandbox the execution of arbitrary user code to a locked down process/container without full access to the metadata DB and connection secrets etc. The idea would be to parse and serialize the DAG in this isolated container and pass back a serialized representation for persistence in the DB. Has anyone explored this idea? 2. Limiting task access to the metadata DB. It would be great if we could remove the requirement for tasks to have full access to the metadata DB and to report task status in a different (but still scalable) way. We'd need to tackle access or injection of connection, variable and xcom data as well for each task naturally. 3. Finer-grained access controls on connection secrets. Right now, although there are nice at-rest encryption options with Fernet or Vault, IIUC any DAG can access any connection (and thus any secret). Since the "run as" user is largely defined within the DAG and its tasks, this is challenging for a multi-tenant environment (see caveat below) Caveat: It's definitely noted that to some extent we should assume that an Airflow deployment is a "trusted" environment and that best practices such as git+PR workflows are the gold standard and that any malicious code and dependencies should be identified through this process. Also that there is a clear admin role for connection management etc. We have some ideas informally sketched out as to how to address the above but would be keen to hear the community opinion on this and to see if anyone is keen to collaborate on designs and implementation, or to hear if anything is already in the works. In particular I noticed that the very first improvement proposal (AIP-1) addresses much of the above :). However, it seems fairly dormant at the moment. One other question: we have a provider (operators and hooks) for interacting with Cloudera components that we'd like to contribute to the project. The provider FAQs indicate that new provider contributions are still welcome in the project in 2.x, is that accurate? Thanks in advance! Ian
