My 2 cents. Roughly based on conversations with Airflow users internal and external.
* Platformization and multi-tenancy has been hard to solve on Airflow. Large deployments of Airflow (multi-cluster on-prem or cloud) have a need to platformize, manage/govern and vend Airflow as a service to their data engineering communities internally. The feature/means to achieving this will be through 1/ DAG manifests and making DAGs a top level tracked/guid-ed entity that has permissions and context models associated with it so that multiple tenants can operate out of the same Airflow environment. * Expanding DAG folders into workspaces/profiles. Users will benefit from a high level construct to group DAGs. This will unlock more opportunities on permission scoping and templatization of access across all dags in that profile/folder. This might also have some UX benefits as a side effect – users dislike seeing a thousand DAGs on the list view. * Modernizing the DAGProcessing->Executor loop in order to support remote DAG fetcher, synced DAGs and similar features. This will lessen the reliance to on-disk files as source of truth. While DAG processing can become faster purely based on DAG Manifest + serialized version of DAG. The secondary process can asynchronously work to update the manifest and serialized DAG based on either filesystem (default) or any configured remote DAG fetcher. From: Ash Berlin-Taylor <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, June 15, 2021 at 2:55 AM To: "[email protected]" <[email protected]> Subject: [EXTERNAL] Roadmap ideas for Airflow 2.2 and beyond CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi everyone, As I'm sure many of you are aware I (along with Aizhamal) am giving the opening keynote at this year's Airflow summit, and I'm covering "what's next after 2.0" -- essentially what is the roadmap for Airflow for the next 12-18 months. Since Airflow is a community project first and foremost I'd like to get all your ideas, no matter how off the wall :) I've got my own ideaas, and 2.2 is fairly firm already (AIPs 39 and 40), but 2.3 and beyond starts to get less clear, so if you have something that you'd like to see Airflow be able to do or do better, now is the time to speak up. You don't have to have a solution, just "I find doing X hard/annoying/difficult" is enough. (And a general reminder: the roadmap is a statement of intent, not a promise of timeline or even that a feature will actually be implemented) To keep this thread manageable, please can we avoid discussions _in this thread_ about ideas and keep +1/me too's to a minimum. Cheers, -ash
