> What’s the added benefit of implementing this pattern? Is the main value prop that DAGs can be defined in new ways and parsed with a specific interface? Because Airflow already supports parsing YAML files with dag-factory.
One primary goal of this AIP is to make support for solutions like DAG Factory Airflow-native. One user pain point (which we've observed first-hand on multiple occasions with our Managed Airflow) when using DAG Factory & similar solutions is the necessity to maintain a Python "bridge" between the declarative format and Airflow. That's because Airflow's DAG processor is purely Python-file oriented, and most optimizations are tailored only for those files. So, as number of DAGs in a single Airflow deployment grows, users must manage both dimensions: - DAG processor(s) itself (which is already quite complex) - Individual files for populating DAGs based on the declarative format. As the number of declarative DAGs grows, the naive approach of loading all configs from a single Python file quickly stops working well (especially in more dynamic development Airflow deployments). So, users must either complicate their CI by populating additional 1-to-1 Python files for declarative configs or devise elaborate sharding schemas. I believe Data Engineers (or their coding agents :) ) shouldn't have to care about the latter dimension in the first place. With the DAG importer they should be able to use off-the-shelf solutions and fully benefit from existing DAG processor optimizations as it will ingest DAGs from individual YAML files (or maybe not even files) without needing to introduce any additional Python code in-between. Another benefit is that it will introduce a clear entry point for unblocking the authoring of DAGs written in non-Python languages by providing the interface to abstract away the individual language runtime complexity (see also discussion in comments of https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-108+Language+Task+SDK+and+the+Language+Coordinator+Layer ). On Fri, Jun 5, 2026 at 5:23 PM Vikram Koka <[email protected]> wrote: > I just left a few comments on the Confluence page as well, but in the > interest of visibility, sharing them here as well. > > What are the actual deliverables of this AIP (in it's revised form)? > > Is this purely an internal code refactor with no immediate user facing > benefits? > I ask because this refactored AIP also refers to an already merged PR 60127 > > If this for performance reasons - mentioned above, but not yet precisely > defined with respect to before / after. > > Are there any user facing benefits as a result of this AIP? > > On Fri, Jun 5, 2026 at 6:27 AM Jake McGrath via dev < > [email protected]> wrote: > >> Quick question from my side; what’s the added benefit of implementing >> this >> pattern? Is the main value prop that DAGs can be defined in new ways and >> parsed with a specific interface? Because Airflow already supports parsing >> YAML files with dag-factory, and things like JSON with homegrown DAG >> generators. Is there a performance component? >> >> >> On Jun 5, 2026 at 7:43:11 AM, Jarek Potiuk <[email protected]> wrote: >> >> > Generally lack of comments or even 'looks good' means that no-one had >> time >> > to look at it - and not that they have no objections. No objections >> > assumption is reserved for LAZY CONSENSUS when consensus seems to be >> > achieved >> > >> > On Fri, Jun 5, 2026, 13:40 Jarek Potiuk <[email protected]> wrote: >> > >> > To be honest - I do not think there was enough of a discussion on it to >> > >> > start vote thread. And certainly not recently. No-one had any comments >> on >> > >> > it after scoping down and placing it in Confluence >> > >> > >> > J. >> > >> > >> > On Fri, Jun 5, 2026, 13:23 Dilnaz Amanzholova via dev < >> > >> > [email protected]> wrote: >> > >> > >> > > Hi all, >> > >> > > >> > >> > > I’m calling vote on AIP-85: DAG importer >> > >> > > https://cwiki.apache.org/confluence/x/_Q7OEg >> > >> > > >> > >> > > You can also review the design document here >> > >> > > < >> > >> > > >> > >> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 >> > >> > > > >> > >> > > . >> > >> > > >> > >> > > Current discussion thread (after scoping down): >> > >> > > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt >> > >> > > Original discussion thread (with wider "extendable DAG parsing >> controls" >> > >> > > title): >> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl >> > >> > > >> > >> > > The vote will run for 5 days, closing on Wednesday, 10th June 2026, at >> > >> > > 10:00 UTC. >> > >> > > >> > >> > > Everyone is encouraged to vote, but only PMC members and Committers' >> > votes >> > >> > > are considered binding. Please vote accordingly. >> > >> > > >> > >> > > [ ] +1 Approve >> > >> > > [ ] +0 no opinion >> > >> > > [ ] -1 disapprove with the reason >> > >> > > >> > >> > > Kind regards, >> > >> > > Dilnaz Amanzholova >> > >> > > >> > >> > >> > >> >
