I think it should be sdk/execution_time/importers similar coordinators. Direct descendants under sdk should be used only for DAG AUTHORING interfaces. Dag importers are not used in dag files, but more like core plugins, so they should be either in core or execution_time. (Ideally they should probably be hosted under a different mechanism, but we probably don’t want to add too much complexity up front (similar to the coordinator package discussion we had for AIP-108).
TP > On 8 Jun 2026, at 19:26, Dilnaz Amanzholova via dev <[email protected]> > wrote: > > Hi Jason, > >> For the location where DagImporter lives, would it be better to keep it > in Task-SDK (`sdk/importers`)? > > Yes, as of now the AbstractDagImporter is implemented under `/sdk` and I > will ensure the modifications to it do not include Core modules. > >> Registering the importer in provider metadata > > Makes sense. I actually already tested this scenario locally (using > import_string()); thank you for informing me about the community consensus. > >> need to include providing the "source code" to be displayed in "Code" tab > > Adding get_source_code(self, file_path) -> str directly to the > AbstractDagImporter interface makes total sense. It cleanly delegates the > responsibility of source-text retrieval to the importer itself. This > ensures that whether it's a native YAML tree or source code managed by a > language SDK, the Web UI has a standardized, non-Python way to fetch and > display it. Thank you! Overall, when implementing the proof of concept > locally, the YAML definition was already showing without an additional > method because I was setting the `dag.fileloc` to the `file_path`. However, > the above method seems like a cleaner and more functional way to implement > it. > > > On Sat, Jun 6, 2026 at 6:01 PM Zhe-You(Jason) Liu <[email protected]> > wrote: > >> Hi all, >> >> For the location where DagImporter lives, would it be better to keep it in >> Task-SDK (`sdk/importers`)? In the long-term of Dag-processing, the >> "parsing" should be treated on the Task-SDK side. Additionally, the >> mentioned "unblocking the authoring of DAGs written in non-Python >> languages" means that Coordinator will build on top of DagImporter. The >> Coordinator lives in Task-SDK now, so the DagImporter _needs_ to live in >> Task-SDK as well to make this happen. (We shouldn't import any module from >> Core in Task-SDK) >> >>> Registering the importer in provider metadata >> >> Would it be better to just dynamically import the class-path specified in >> the Airflow config instead of overwhelming the ProvidersManager interface? >> Same as the recent discussion on community consensus regarding the >> Coordinator interface feature. Since we're able to dynamically import the >> class-path with `import_string` anyway, we can free the ProviderManager >> entirely. >> >> >>> This would need to include providing the "source code" to be displayed in >> "Code" tab - an improvement >>> From Jens' comment on the Confluence page >> >> Yes, it's a necessary functional requirement for the "defining whole Dag in >> lang-sdk" that we defer in the multi-lang feature. The coordinator >> interface back then supported `get_code_from_file(file_path)` [1] In other >> words, we implement the concept of DagImporter as a shallow interfaces on >> the coordinator then. >> >> IMHO, we should support the `get_source_code(self, file_path) -> str` >> method for the `AbstractDagImporter` interface. What I imagine for the >> further Coordinator interface is to add a class attribute that specifies >> the DagImporter (e.g. `JavaCoordinator.dag_importer = JavaDagImporter`). >> >> [1] >> >> https://github.com/jason810496/airflow/blob/task-sdk/feature/coordinator-interface-backup-with-pure-java-dag-feature/sdk/coordinators/java/src/airflow/sdk/coordinators/java/coordinator.py#L75-L88 >> >> Thanks. >> >> Best regards, >> Jason >> >> On Sat, Jun 6, 2026 at 2:04 AM Igor Kholopov via dev < >> [email protected]> >> wrote: >> >>>> What’s the added benefit of implementing this pattern? Is the main >> value >>> prop that DAGs can be defined in new ways and parsed with a specific >>> interface? Because Airflow already supports parsing YAML files with >>> dag-factory. >>> >>> One primary goal of this AIP is to make support for solutions like DAG >>> Factory Airflow-native. One user pain point (which we've observed >>> first-hand on multiple occasions with our Managed Airflow) when using DAG >>> Factory & similar solutions is the necessity to maintain a Python >> "bridge" >>> between the declarative format and Airflow. That's because Airflow's DAG >>> processor is purely Python-file oriented, and most optimizations are >>> tailored only for those files. So, as number of DAGs in a single Airflow >>> deployment grows, users must manage both dimensions: >>> - DAG processor(s) itself (which is already quite complex) >>> - Individual files for populating DAGs based on the declarative format. >> As >>> the number of declarative DAGs grows, the naive approach of loading all >>> configs from a single Python file quickly stops working well (especially >> in >>> more dynamic development Airflow deployments). So, users must either >>> complicate their CI by populating additional 1-to-1 Python files for >>> declarative configs or devise elaborate sharding schemas. >>> I believe Data Engineers (or their coding agents :) ) shouldn't have to >>> care about the latter dimension in the first place. With the DAG importer >>> they should be able to use off-the-shelf solutions and fully benefit from >>> existing DAG processor optimizations as it will ingest DAGs from >> individual >>> YAML files (or maybe not even files) without needing to introduce any >>> additional Python code in-between. >>> >>> Another benefit is that it will introduce a clear entry point for >>> unblocking the authoring of DAGs written in non-Python languages by >>> providing the interface to abstract away the individual language runtime >>> complexity (see also discussion in comments of >>> >>> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-108+Language+Task+SDK+and+the+Language+Coordinator+Layer >>> ). >>> >>> On Fri, Jun 5, 2026 at 5:23 PM Vikram Koka <[email protected]> wrote: >>> >>>> I just left a few comments on the Confluence page as well, but in the >>>> interest of visibility, sharing them here as well. >>>> >>>> What are the actual deliverables of this AIP (in it's revised form)? >>>> >>>> Is this purely an internal code refactor with no immediate user facing >>>> benefits? >>>> I ask because this refactored AIP also refers to an already merged PR >>> 60127 >>>> >>>> If this for performance reasons - mentioned above, but not yet >> precisely >>>> defined with respect to before / after. >>>> >>>> Are there any user facing benefits as a result of this AIP? >>>> >>>> On Fri, Jun 5, 2026 at 6:27 AM Jake McGrath via dev < >>>> [email protected]> wrote: >>>> >>>>> Quick question from my side; what’s the added benefit of implementing >>>>> this >>>>> pattern? Is the main value prop that DAGs can be defined in new ways >> and >>>>> parsed with a specific interface? Because Airflow already supports >>> parsing >>>>> YAML files with dag-factory, and things like JSON with homegrown DAG >>>>> generators. Is there a performance component? >>>>> >>>>> >>>>> On Jun 5, 2026 at 7:43:11 AM, Jarek Potiuk <[email protected]> wrote: >>>>> >>>>>> Generally lack of comments or even 'looks good' means that no-one >> had >>>>> time >>>>>> to look at it - and not that they have no objections. No objections >>>>>> assumption is reserved for LAZY CONSENSUS when consensus seems to >> be >>>>>> achieved >>>>>> >>>>>> On Fri, Jun 5, 2026, 13:40 Jarek Potiuk <[email protected]> wrote: >>>>>> >>>>>> To be honest - I do not think there was enough of a discussion on it >>> to >>>>>> >>>>>> start vote thread. And certainly not recently. No-one had any >> comments >>>>> on >>>>>> >>>>>> it after scoping down and placing it in Confluence >>>>>> >>>>>> >>>>>> J. >>>>>> >>>>>> >>>>>> On Fri, Jun 5, 2026, 13:23 Dilnaz Amanzholova via dev < >>>>>> >>>>>> [email protected]> wrote: >>>>>> >>>>>> >>>>>>> Hi all, >>>>>> >>>>>>> >>>>>> >>>>>>> I’m calling vote on AIP-85: DAG importer >>>>>> >>>>>>> https://cwiki.apache.org/confluence/x/_Q7OEg >>>>>> >>>>>>> >>>>>> >>>>>>> You can also review the design document here >>>>>> >>>>>>> < >>>>>> >>>>>>> >>>>>> >>>>> >>> >> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 >>>>>> >>>>>>>> >>>>>> >>>>>>> . >>>>>> >>>>>>> >>>>>> >>>>>>> Current discussion thread (after scoping down): >>>>>> >>>>>>> https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt >>>>>> >>>>>>> Original discussion thread (with wider "extendable DAG parsing >>>>> controls" >>>>>> >>>>>>> title): >>>>> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl >>>>>> >>>>>>> >>>>>> >>>>>>> The vote will run for 5 days, closing on Wednesday, 10th June >> 2026, >>> at >>>>>> >>>>>>> 10:00 UTC. >>>>>> >>>>>>> >>>>>> >>>>>>> Everyone is encouraged to vote, but only PMC members and >> Committers' >>>>>> votes >>>>>> >>>>>>> are considered binding. Please vote accordingly. >>>>>> >>>>>>> >>>>>> >>>>>>> [ ] +1 Approve >>>>>> >>>>>>> [ ] +0 no opinion >>>>>> >>>>>>> [ ] -1 disapprove with the reason >>>>>> >>>>>>> >>>>>> >>>>>>> Kind regards, >>>>>> >>>>>>> Dilnaz Amanzholova >>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
