Hi TP, I agree with placing it under sdk/execution_time/importers and it makes sense to have a similar consensus as with coordinators.
I will update the description of AIP-85 based on the comments provided above and check the AIP-108 discussion thread to align them. Thanks! Dilnaz Amanzholova On Mon, Jun 8, 2026 at 1:58 PM Tzu-ping Chung <[email protected]> wrote: > I think it should be sdk/execution_time/importers similar coordinators. > Direct descendants under sdk should be used only for DAG AUTHORING > interfaces. Dag importers are not used in dag files, but more like core > plugins, so they should be either in core or execution_time. (Ideally they > should probably be hosted under a different mechanism, but we probably > don’t want to add too much complexity up front (similar to the coordinator > package discussion we had for AIP-108). > > TP > > > > On 8 Jun 2026, at 19:26, Dilnaz Amanzholova via dev < > [email protected]> wrote: > > > > Hi Jason, > > > >> For the location where DagImporter lives, would it be better to keep it > > in Task-SDK (`sdk/importers`)? > > > > Yes, as of now the AbstractDagImporter is implemented under `/sdk` and I > > will ensure the modifications to it do not include Core modules. > > > >> Registering the importer in provider metadata > > > > Makes sense. I actually already tested this scenario locally (using > > import_string()); thank you for informing me about the community > consensus. > > > >> need to include providing the "source code" to be displayed in "Code" > tab > > > > Adding get_source_code(self, file_path) -> str directly to the > > AbstractDagImporter interface makes total sense. It cleanly delegates the > > responsibility of source-text retrieval to the importer itself. This > > ensures that whether it's a native YAML tree or source code managed by a > > language SDK, the Web UI has a standardized, non-Python way to fetch and > > display it. Thank you! Overall, when implementing the proof of concept > > locally, the YAML definition was already showing without an additional > > method because I was setting the `dag.fileloc` to the `file_path`. > However, > > the above method seems like a cleaner and more functional way to > implement > > it. > > > > > > On Sat, Jun 6, 2026 at 6:01 PM Zhe-You(Jason) Liu <[email protected]> > > wrote: > > > >> Hi all, > >> > >> For the location where DagImporter lives, would it be better to keep it > in > >> Task-SDK (`sdk/importers`)? In the long-term of Dag-processing, the > >> "parsing" should be treated on the Task-SDK side. Additionally, the > >> mentioned "unblocking the authoring of DAGs written in non-Python > >> languages" means that Coordinator will build on top of DagImporter. The > >> Coordinator lives in Task-SDK now, so the DagImporter _needs_ to live in > >> Task-SDK as well to make this happen. (We shouldn't import any module > from > >> Core in Task-SDK) > >> > >>> Registering the importer in provider metadata > >> > >> Would it be better to just dynamically import the class-path specified > in > >> the Airflow config instead of overwhelming the ProvidersManager > interface? > >> Same as the recent discussion on community consensus regarding the > >> Coordinator interface feature. Since we're able to dynamically import > the > >> class-path with `import_string` anyway, we can free the ProviderManager > >> entirely. > >> > >> > >>> This would need to include providing the "source code" to be displayed > in > >> "Code" tab - an improvement > >>> From Jens' comment on the Confluence page > >> > >> Yes, it's a necessary functional requirement for the "defining whole > Dag in > >> lang-sdk" that we defer in the multi-lang feature. The coordinator > >> interface back then supported `get_code_from_file(file_path)` [1] In > other > >> words, we implement the concept of DagImporter as a shallow interfaces > on > >> the coordinator then. > >> > >> IMHO, we should support the `get_source_code(self, file_path) -> str` > >> method for the `AbstractDagImporter` interface. What I imagine for the > >> further Coordinator interface is to add a class attribute that specifies > >> the DagImporter (e.g. `JavaCoordinator.dag_importer = JavaDagImporter`). > >> > >> [1] > >> > >> > https://github.com/jason810496/airflow/blob/task-sdk/feature/coordinator-interface-backup-with-pure-java-dag-feature/sdk/coordinators/java/src/airflow/sdk/coordinators/java/coordinator.py#L75-L88 > >> > >> Thanks. > >> > >> Best regards, > >> Jason > >> > >> On Sat, Jun 6, 2026 at 2:04 AM Igor Kholopov via dev < > >> [email protected]> > >> wrote: > >> > >>>> What’s the added benefit of implementing this pattern? Is the main > >> value > >>> prop that DAGs can be defined in new ways and parsed with a specific > >>> interface? Because Airflow already supports parsing YAML files with > >>> dag-factory. > >>> > >>> One primary goal of this AIP is to make support for solutions like DAG > >>> Factory Airflow-native. One user pain point (which we've observed > >>> first-hand on multiple occasions with our Managed Airflow) when using > DAG > >>> Factory & similar solutions is the necessity to maintain a Python > >> "bridge" > >>> between the declarative format and Airflow. That's because Airflow's > DAG > >>> processor is purely Python-file oriented, and most optimizations are > >>> tailored only for those files. So, as number of DAGs in a single > Airflow > >>> deployment grows, users must manage both dimensions: > >>> - DAG processor(s) itself (which is already quite complex) > >>> - Individual files for populating DAGs based on the declarative format. > >> As > >>> the number of declarative DAGs grows, the naive approach of loading all > >>> configs from a single Python file quickly stops working well > (especially > >> in > >>> more dynamic development Airflow deployments). So, users must either > >>> complicate their CI by populating additional 1-to-1 Python files for > >>> declarative configs or devise elaborate sharding schemas. > >>> I believe Data Engineers (or their coding agents :) ) shouldn't have to > >>> care about the latter dimension in the first place. With the DAG > importer > >>> they should be able to use off-the-shelf solutions and fully benefit > from > >>> existing DAG processor optimizations as it will ingest DAGs from > >> individual > >>> YAML files (or maybe not even files) without needing to introduce any > >>> additional Python code in-between. > >>> > >>> Another benefit is that it will introduce a clear entry point for > >>> unblocking the authoring of DAGs written in non-Python languages by > >>> providing the interface to abstract away the individual language > runtime > >>> complexity (see also discussion in comments of > >>> > >>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-108+Language+Task+SDK+and+the+Language+Coordinator+Layer > >>> ). > >>> > >>> On Fri, Jun 5, 2026 at 5:23 PM Vikram Koka <[email protected]> > wrote: > >>> > >>>> I just left a few comments on the Confluence page as well, but in the > >>>> interest of visibility, sharing them here as well. > >>>> > >>>> What are the actual deliverables of this AIP (in it's revised form)? > >>>> > >>>> Is this purely an internal code refactor with no immediate user facing > >>>> benefits? > >>>> I ask because this refactored AIP also refers to an already merged PR > >>> 60127 > >>>> > >>>> If this for performance reasons - mentioned above, but not yet > >> precisely > >>>> defined with respect to before / after. > >>>> > >>>> Are there any user facing benefits as a result of this AIP? > >>>> > >>>> On Fri, Jun 5, 2026 at 6:27 AM Jake McGrath via dev < > >>>> [email protected]> wrote: > >>>> > >>>>> Quick question from my side; what’s the added benefit of implementing > >>>>> this > >>>>> pattern? Is the main value prop that DAGs can be defined in new ways > >> and > >>>>> parsed with a specific interface? Because Airflow already supports > >>> parsing > >>>>> YAML files with dag-factory, and things like JSON with homegrown DAG > >>>>> generators. Is there a performance component? > >>>>> > >>>>> > >>>>> On Jun 5, 2026 at 7:43:11 AM, Jarek Potiuk <[email protected]> wrote: > >>>>> > >>>>>> Generally lack of comments or even 'looks good' means that no-one > >> had > >>>>> time > >>>>>> to look at it - and not that they have no objections. No objections > >>>>>> assumption is reserved for LAZY CONSENSUS when consensus seems to > >> be > >>>>>> achieved > >>>>>> > >>>>>> On Fri, Jun 5, 2026, 13:40 Jarek Potiuk <[email protected]> wrote: > >>>>>> > >>>>>> To be honest - I do not think there was enough of a discussion on it > >>> to > >>>>>> > >>>>>> start vote thread. And certainly not recently. No-one had any > >> comments > >>>>> on > >>>>>> > >>>>>> it after scoping down and placing it in Confluence > >>>>>> > >>>>>> > >>>>>> J. > >>>>>> > >>>>>> > >>>>>> On Fri, Jun 5, 2026, 13:23 Dilnaz Amanzholova via dev < > >>>>>> > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>> > >>>>>>> Hi all, > >>>>>> > >>>>>>> > >>>>>> > >>>>>>> I’m calling vote on AIP-85: DAG importer > >>>>>> > >>>>>>> https://cwiki.apache.org/confluence/x/_Q7OEg > >>>>>> > >>>>>>> > >>>>>> > >>>>>>> You can also review the design document here > >>>>>> > >>>>>>> < > >>>>>> > >>>>>>> > >>>>>> > >>>>> > >>> > >> > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 > >>>>>> > >>>>>>>> > >>>>>> > >>>>>>> . > >>>>>> > >>>>>>> > >>>>>> > >>>>>>> Current discussion thread (after scoping down): > >>>>>> > >>>>>>> https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt > >>>>>> > >>>>>>> Original discussion thread (with wider "extendable DAG parsing > >>>>> controls" > >>>>>> > >>>>>>> title): > >>>>> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl > >>>>>> > >>>>>>> > >>>>>> > >>>>>>> The vote will run for 5 days, closing on Wednesday, 10th June > >> 2026, > >>> at > >>>>>> > >>>>>>> 10:00 UTC. > >>>>>> > >>>>>>> > >>>>>> > >>>>>>> Everyone is encouraged to vote, but only PMC members and > >> Committers' > >>>>>> votes > >>>>>> > >>>>>>> are considered binding. Please vote accordingly. > >>>>>> > >>>>>>> > >>>>>> > >>>>>>> [ ] +1 Approve > >>>>>> > >>>>>>> [ ] +0 no opinion > >>>>>> > >>>>>>> [ ] -1 disapprove with the reason > >>>>>> > >>>>>>> > >>>>>> > >>>>>>> Kind regards, > >>>>>> > >>>>>>> Dilnaz Amanzholova > >>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >
