Thanks a lot for being so responsive to the feedback on the AIP Confluent page, Dilnaz. Since yesterday, you've added file_dependencies to DagImportResult, introduced a DagSource class that carries the source language for the Code tab, and made the importer precedence order explicit (bundle explicit mapping > global config > defaults) - all great improvements.
I agree with Jarek on reconciling the merged foundation with the AIP (particularly the per-bundle registry) before re-running the vote. And yeah, explicitly stating that a declarative DAG file carries the same trust level as the Python code is important - in our experience, users often think YAML is just data, so we should make this explicit. On the zip question, composite importers feel like the right path forward. Once we get non-Python formats treated as first-class citizens, it feels like the next natural step is mixed content. Delegating to inner importers handles that cleanly, whereas I feel that extending PythonDagImporter wouldn't scale as well. It does feel we should wrap up the bigger questions in [DISCUSS] - this will allow maintainers to VOTE on the proposed design with confidence, and the AIP will be stronger with it. With the pace I've observed since yesterday, it doesn't feel like we're far off - looking forward to seeing it back into VOTE. Kind regards, Tatiana On Wed, 8 Jul 2026 at 13:24, Jarek Potiuk <[email protected]> wrote: > Hi Dilnaz, and indeed - thanks for pushing this forward. > > The scoped-down "DAG importer" framing is much clearer, and > native non-Python DAGs are a genuinely good direction. +1 on the goal. > > I am happy to make comments and suggestions or even fill-in some blanks > (security especially) in the AIP proposal if you are fine with me doing it > - see > some of the comments below. > > Before I can vote on the design, I'd like to raise one structural concern > plus a couple of smaller ones. My main point is that the foundation already > merged in https://github.com/apache/airflow/pull/60127 > contradicts the design the AIP describes, and I think we > should reconcile the two before the vote closes rather than after. > > Some of these points are already explicitly stated, either in the AIP or > opened > questions raised by Tatiana - I just want to reiterate them here and see if > we can > reach consensus on the shape of those changes. > > Concretely, in airflow-core/src/airflow/dag_processing/importers/: > > 1. Registry scope — global singleton vs. per-bundle. The AIP describes > per-bundle importer mapping with a bundle -> global -> default resolution > order (and Tatiana's Q4/Q7 ask for the same). But the merged > DagImporterRegistry is a process-wide singleton keyed by extension, "last > registered wins, one importer per extension." A global extension->importer > map structurally can't give two bundles different importers for the same > extension, so as it stands the foundation can't support the resolution > model the AIP promises. If per-bundle is the intended end state, I think we > want that in the interface now, because swapping a singleton out later is > expensive. > > 2. Zip handling. The AIP describes a generic ZipImporter that delegates to > inner-format importers. The merged PythonDagImporter instead hard-owns both > .py and .zip, with a docstring saying YAML-in-zip would require extending > it > or writing a composite importer — the opposite of the AIP. I think we > should > decide on one or the other direction, I like the composite importer more - > even > if it is more complex, it allows for greater flexibility if we have > composite > importers. But I would love to hear what others think. > > 3. DagImportResult.file_dependencies is in the AIP dataclass but not in the > merged one — and this is more than cosmetic. Declarative DAGs routinely > share a defaults.yaml; the DFP decides what to re-parse from per-file > mtime, > so without importer-declared dependencies a change to a shared file won't > re-parse its dependents (a staleness bug). This is Tatiana's Q5, and I'd > treat it as a correctness requirement, not a nice-to-have. Worth spelling > out how it interacts with AIP-66's git-sha bundle versioning too. > > On security, I'd like the AIP to state explicitly that a declarative DAG > file is fully code-equivalent trust: a YAML importer resolving operators by > classpath has the same power as arbitrary Python. People tend to treat YAML > as "data" and .py as "code," and I want to make sure nobody builds a > "review-free YAML bundle" path on the back of this. I'd also like it stated > as a hard constraint that importers run only in the DAG processor, > in-process, under the same parse timeout, and never touch the metadata DB — > consistent with the AIP-66/AIP-72 boundary. And a quick confirmation that > the old "push DAGs over API / ingester" idea stays out of scope, since that > one genuinely changes the threat model. > > Again, for security, if you are happy with it, I would love to be able to > fill > all those security blanks - this can be described on high level in the AIP, > and when implementation progresses, it would turn into security model > updates (I'm also happy to help there). > > Two smaller things: > > - The safe_mode/might_contain_dag heuristic ("file contains the strings > airflow + dag") currently lives in the abstract list_dag_files, but it's > Python-specific and will mis-handle YAML — it should be delegated > per-importer (Tatiana's Q3). > > - It'd be good to confirm the registry indirection adds no measurable > regression on the plain .py path - some performance tests with basic > big-numbers-of-dags scenarios would be great to do as part of the > implementation. > > None of this is a blocker on the idea — I'm supportive. But given the open > design questions and that the vote has been quiet, It reaffirms we treat > this > as still in [DISCUSS], reconcile the AIP with the merged foundation > (especially point 1), and re-run the vote once the interface reflects the > intended end state. > > Happy to help review the per-bundle registry shape. > > Best, > Jarek > > On Tue, Jul 7, 2026 at 4:45 PM Tatiana Al-Chueyr Martins < > [email protected]> wrote: > > > Hi Dilnaz, > > > > Thanks a lot for pushing this forward, and for the updates after the > > earlier feedback - AIP 85 looks much clearer than before. > > > > I'm one of the maintainers of DAG Factory ( > > https://github.com/astronomer/dag-factory), and I like the direction the > > AIP is taking. I agree with the pain points Igor raised earlier; it would > > be great to support non-Python DAGs natively in Airflow, rather than > > converting them as we currently do in DAG Factory. > > > > I've reviewed the AIP page and left some inline comments & questions. > Since > > the authors are already responding on the AIP page, happy to continue the > > detailed discussion there - sharing the summary here mainly for > visibility: > > > > 1. *Location of schemas and importers*: Where do we see the schemas and > > actual importers defined - and how do we plan to govern them? It would > be > > important to align, for instance, on how non-primitive operator arguments > > should be defined in YAML (e.g., Asset, V1Pod, arbitrary classes) and on > > the security aspects of loading arbitrary classes from YAML. > > > > 2. *Validation contract*: If users define unknown or unsupported keys and > > configurations, should we fail loudly with DagImportErrors referencing > > files/lines like the Python DAG import errors? It would be great if we > > didn't just skip them. > > > > 3.* Decision on which files to parse: *As raised by TP on > > < > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > > > > the > > Google > > < > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > > > > Doc > > < > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > > > > , I feel that we should have additional criteria to decide which files to > > treat as a DAG - and not rely only on the file extension. As an example, > > bundles normally have non-DAG YAML - dbt_project.yml, CI configs, K8s > > manifests. Would it make sense to have something similar to the Python > > content-level heuristic ("airflow + dag" string check)? > > > > 4. *Different importers for the same file extension*: Can/should we > support > > different importers for the same file extension? It would be great if the > > architecture could support overriding the importer per file. > > > > 5.* Cross-file dependency tracking:* Declarative DAGs are rarely > > single-file. They can share default definitions, for example: > > https://astronomer.github.io/dag-factory/1.1.0/configuration/defaults/. > > Could importers declare file dependencies in DagImportResult, so the > > processor can use them? > > > > 6. *Code tab UI*: On get_source_code(), could we have a parameter related > > to the file language, which could be used by the Code tab (it is > currently > > hard-coded with Python)? How are we planning to resolve ZipImporter and > > other transformed sources in the UI? > > > > 7. *Precedence*: What should be the precedence (per-bundle vs global vs > > default importers)? > > > > For the performance questions raised earlier in the thread, we could use > > DAG Factory workloads as a baseline, comparing the non-built-in-Airflow > > Python layer against a native YAML importer. > > > > Our team is very keen to contribute to the YAML contract definition. > > > > Kind regards, > > > > Tatiana > > > > > > On Tue, 7 Jul 2026 at 13:48, Dilnaz Amanzholova via dev < > > [email protected]> wrote: > > > > > Hi all, I am following up to request your feedback on AIP-85: DAG > > importer. > > > I attempted to present this at the last Airflow dev call, but the > agenda > > > was full, and today’s meeting was unfortunately cancelled. As it has > been > > > some time since this AIP was submitted for review, I would greatly > > > appreciate it if you could share your thoughts or any remaining > concerns > > > here. I understand there are many active discussions, but your input > > would > > > be invaluable in helping move this proposal forward. Thanks, Dilnaz > > > Amanzholova > > > > > > On Thu, Jun 18, 2026 at 2:08 PM Dilnaz Amanzholova <[email protected] > > > > > wrote: > > > > > > > Hey Kaxil, > > > > > > > > Thanks for the clarification. I have decided to attend, as the next > > > > meeting is not for another two weeks. I have added the AIP to the > > agenda. > > > > > > > > Best regards, > > > > Dilnaz Amanzholova > > > > > > > > > > > > On Wed, Jun 17, 2026 at 6:37 PM Kaxil Naik <[email protected]> > > wrote: > > > > > > > >> Hey not required at all. > > > >> > > > >> This is just in case you wanted to solicit feedback live on the call > > > with > > > >> other Airflow devs. Sometimes it is easier that way -- but not > > > mandatory. > > > >> > > > >> More details on the dev calls: > > > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Call > > if > > > >> you decide about presenting it next time. > > > >> > > > >> Regards, > > > >> Kaxil > > > >> > > > >> On Wed, 17 Jun 2026 at 17:32, Dilnaz Amanzholova < > [email protected]> > > > >> wrote: > > > >> > > > >>> Hi Kaxil, > > > >>> > > > >>> I was not aware there was a regular call. Is it required to present > > the > > > >>> AIP before voting? > > > >>> > > > >>> Kind regards, > > > >>> Dilnaz Amanzholova > > > >>> > > > >>> > > > >>> On Wed, Jun 17, 2026 at 6:26 PM Kaxil Naik <[email protected]> > > > wrote: > > > >>> > > > >>>> Hi @dilnaz -- Are you up for presenting it tomorrow or on the next > > dev > > > >>>> call? > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Tue, 16 Jun 2026 at 15:15, Dilnaz Amanzholova via dev < > > > >>>> [email protected]> wrote: > > > >>>> > > > >>>>> Hi all, > > > >>>>> > > > >>>>> I am following up on AIP-85: DAG importer. All previous concerns > > have > > > >>>>> been > > > >>>>> addressed, and the content has been updated on the AIP Confluence > > > page. > > > >>>>> > > > >>>>> Please review the latest updates at your earliest convenience. I > > > would > > > >>>>> appreciate your feedback and am available to answer any further > > > >>>>> questions > > > >>>>> you may have. > > > >>>>> > > > >>>>> Best regards, > > > >>>>> Dilnaz Amanzholova > > > >>>>> > > > >>>>> > > > >>>>> On Fri, Jun 5, 2026 at 1:23 PM Dilnaz Amanzholova < > > > [email protected]> > > > >>>>> wrote: > > > >>>>> > > > >>>>> > Hi all, > > > >>>>> > > > > >>>>> > I’m calling vote on AIP-85: DAG importer > > > >>>>> > https://cwiki.apache.org/confluence/x/_Q7OEg > > > >>>>> > > > > >>>>> > You can also review the design document here > > > >>>>> > < > > > >>>>> > > > > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 > > > >>>>> > > > > >>>>> > . > > > >>>>> > > > > >>>>> > Current discussion thread (after scoping down): > > > >>>>> > > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt > > > >>>>> > Original discussion thread (with wider "extendable DAG parsing > > > >>>>> controls" > > > >>>>> > title): > > > >>>>> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl > > > >>>>> > > > > >>>>> > The vote will run for 5 days, closing on Wednesday, 10th June > > 2026, > > > >>>>> at > > > >>>>> > 10:00 UTC. > > > >>>>> > > > > >>>>> > Everyone is encouraged to vote, but only PMC members and > > > Committers' > > > >>>>> votes > > > >>>>> > are considered binding. Please vote accordingly. > > > >>>>> > > > > >>>>> > [ ] +1 Approve > > > >>>>> > [ ] +0 no opinion > > > >>>>> > [ ] -1 disapprove with the reason > > > >>>>> > > > > >>>>> > Kind regards, > > > >>>>> > Dilnaz Amanzholova > > > >>>>> > > > > >>>>> > > > >>>> > > > > > > > > > -- > > Tatiana Al-Chueyr > > > -- Tatiana Al-Chueyr
