Hi Kaxil, TP, Sumit, Thanks for the feedback.
Agreed with Kaxil: for a factory/driver file the unit of construction is a config (e.g. a YAML). Malformed configs can trigger import errors if not handled gracefully. And thanks TP for calling out the "Per DAG import error" title, you're right, if the DAG object was never constructed there's no dag_id to scope the error to. Import errors are inherently source-scoped. I'd reframe this as "Reporting partially-parsed drivers to the dag-processor": after a factory file is parsed, report both the healthy DAGs and the config-level errors, where each error is fatal to its broken config but non-fatal to the file as a whole. I'd separate one thing that I think is a bug independent of AIP-85. Today, if a single DAG from a multi-DAG file fails during serialization or a dag_policy, all the healthy dags are marked stale. So healthy DAGs that already serialized get deactivated because a sibling failed. This hits any file that legitimately defines several DAGs (no factory needed), and AIP-85 doesn't address it since it happens after parsing. I'd like to fix this on its own: scope the stale/import-error update to the DAG(s) that actually failed. On AIP-85 more broadly: I agree it's the right home for source-granularity, but its scope is large, a full abstraction layer for pluggable importers. I would propose enriching the existing Python importer with an opt-in contract so a factory file can return healthy DAGs plus per-config errors. Our setup The reason source-granularity alone doesn't cover us: our driver files don't map 1 YAML → 1 DAG. They merge environment-specific YAMLs into a single DAG (Staging.yaml overrides Production.yaml so we never accidentally trigger prod DAGs in non-prod). So even if each YAML were its own parse unit, we'd still have one parse unit producing one DAG from many configs. More generally, any importer where one parse unit emits multiple DAGs reproduces the all-or-nothing problem. That has to be handled by the importer and conveyed back to the processor. DagWarning can't carry it either as it is non-fatal by design. Looking for more feedback. Happy to write up a small, focused proposal, if you think it's a good addition. Thank you. Prajwal On Wed, Aug 12, 2026 at 2:11 PM Tzu-ping Chung via dev < [email protected]> wrote: > Perhaps I’m misunderstanding “DAG-scoped” in the original idea? If there’s > no DAG to attach to, how can the message be DAG-scoped? “I know I can > generate a DAG, but I can’t generate the DAG” doesn’t make that much sense > to me. > > TP > > > On Aug 12, 2026, at 17:15, Kaxil Naik <[email protected]> wrote: > > > > Agreed on the definitional split, and I should have been clearer about > what I meant. > > > > I wasn't suggesting import errors become DAG-scoped. I agree that an > import error means Airflow couldn't make sense of the input and the whole > file should fail. My point was about what counts as "the file". > > > > Today a driver file (Example: example_dag_factory.py for Dag factory as > shown in tutorial: > https://astronomer.github.io/dag-factory/latest/getting-started/quick-start-airflow-standalone/#step-6-generate-the-dag-from-yaml > ) is the parse unit, so the file scope is wrong by construction: the thing > that failed is one YAML config, but the unit Airflow keys the error on is > the Python bridge that loaded 500 of them. That is the entire reason this > looks like it needs DAG scoping. Once a YAML file is itself a DAG file, > which is what AIP-85 is for, or atleast it provides hook points for it, the > existing file-scoped semantics land in the right place on their own. 3 > files get import errors, 497 files parse fine, and no new contract between > the processor and a driver is needed. > > > > AIP-85 doesn't address error scoping, so you're right that it isn't > covered there. But it names this exact workaround as motivation: because > the processor only recognizes .py, teams either accept one bridge file as a > bottleneck or generate 1-to-1 Python files per config in CI to shard it. > Prajwal's setup is the second one. Removing that layer removes most of the > reported problem as a side effect, which is why I asked. > > > > On DagWarning, I agree it is the right home for "this imported, but you > should fix it". It can't carry Prajwal's case though. It is keyed on > (dag_id, warning_type) with a foreign key to dag.dag_id, so it needs a DAG > that was successfully produced, and a config broken enough to produce no > DAG has nothing to attach to. warning_type is also validated against a > closed enum today. Worth noting that AIP-85 already proposes relaxing that > to a namespaced string plus a context dict for frontend translation, so the > two threads do overlap there. > > > > The part I would separate out as a real bug, independent of both: > _update_import_errors sets is_stale=True on every DagModel sharing the > relative_fileloc. The healthy DAGs from that file are written and > serialized first, then staled because a sibling failed. Even with one YAML > per file that will cause us problems whenever a file legitimately defines > several DAGs, or when the failure comes from serialization or a dag_policy > rather than the parse itself, which is what Prajwal actually reported. > > > > Regards, > > Kaxil > > > > On Wed, 12 Aug 2026 at 12:26, Tzu-ping Chung via dev < > [email protected] <mailto:[email protected]>> wrote: > >> I don’t think it’s covered by AIP-85. > >> > >> However, doesn’t we already have DagWarning for this? Import errors are > source-level errors that implies the dags cannot be parsed at all, and by > definition should fail the entire thing without dag scopes because Airflow > simply can’t make sense of the input. > >> > >> A DagWarning, on the other hand, means the dags can be imported, but > there are still things you probably should fix. This already exists for a > long time. > >> > >> TP > >> > >> > >> > On Aug 12, 2026, at 15:13, Kaxil Naik <[email protected] <mailto: > [email protected]>> wrote: > >> > > >> > Is this part of AIP-85 already? > >> > > >> > On Tue, 11 Aug 2026 at 15:00, Sumit Maheshwari < > [email protected] <mailto:[email protected]>> > >> > wrote: > >> > > >> >> Strong +1 on this feature, as almost all of the companies using > Airflow are > >> >> probably using one or more such DAG generator frameworks, and a > problem in > >> >> one of the configs causes all the DAGs to go into a stale state. > >> >> > >> >> I would like to write a Google Doc proposal or an AIP (whichever the > >> >>> community prefers) and work on it. > >> >> > >> >> > >> >> Will this change require changes in all the generator frameworks, or > will > >> >> it be backward compatible? If it can be implemented without forcing > >> >> existing DAG generator frameworks, then I think we should not > require an > >> >> AIP. > >> >> > >> >> On Mon, Aug 10, 2026 at 6:32 PM Prajwal Agarwal < > [email protected] <mailto:[email protected]>> > >> >> wrote: > >> >> > >> >>> Hi Airflow Community, > >> >>> *TL;DR:* There is no way to surface import errors alongside healthy > dags > >> >>> from a driver (like dag- <https://github.com/astronomer/dag-factory > >> >>>> factory > >> >>> <https://github.com/astronomer/dag-factory>), if some configs > passed > >> >> into > >> >>> the loader file are wrong. It is either “all” or “none”. If the > driver > >> >> file > >> >>> raises an error, it can deactivate all healthy DAGs belonging to > that > >> >>> driver. Driver owners handle this by gracefully managing the > exceptions > >> >> in > >> >>> their code. > >> >>> I would like to discuss potential solutions (or learn if one already > >> >> exists > >> >>> or is planned). > >> >>> > >> >>> *Details* > >> >>> When a single Python file generates many DAGs — as dag-factory and > >> >> similar > >> >>> "driver file" patterns do — Airflow's import-error handling is > >> >> file-scoped, > >> >>> not DAG-scoped. One bad config in a factory that builds hundreds of > DAGs > >> >>> currently forces an all-or-nothing outcome. > >> >>> If the factory lets the exception propagate, the whole file fails to > >> >> import > >> >>> and every DAG it would have produced disappears > (has_import_error=True, > >> >>> is_stale=True in dag-processor). > >> >>> If the factory swallows the bad config to keep the others alive, the > >> >>> failure is silent—no import error is surfaced to the end user, on > the > >> >>> Airflow UI. > >> >>> There is no way to say, "These 3 configurations are broken, here is > why, > >> >>> but the other 497 DAGs are healthy and should keep running." > >> >>> I'd like to discuss a contract that makes that possible. > >> >>> > >> >>> *Current Behavior:* > >> >>> Import errors are keyed by file, one string per file. > >> >>> Staleness is applied to every DAG sharing the file's fileloc. > >> >>> > >> >>> *Proposed Solution (Details are intentionally missing)* > >> >>> A contract between dag-processor and driver file to return healthy > dags > >> >> as > >> >>> well as import errors (per file or per dag, given it is not > necessary > >> >> that > >> >>> dag_id is available) > >> >>> > >> >>> *Motivation* > >> >>> We use Airflow to power 100s of dags from a single driver > (dag-factory in > >> >>> our case) where our customers write YAML files. We do have some > build > >> >> time > >> >>> validations, but serialization errors can happen due to dag_policies > >> >>> execution and failure as well. We would like to provide better > >> >>> observability over these failures to our end users. > >> >>> > >> >>> *Ask* > >> >>> I would like to know the community's opinion on supporting such a > >> >> feature. > >> >>> If there is a use case, I would like to write a Google Doc proposal > or an > >> >>> AIP (whichever the community prefers) and work on it. > >> >>> > >> >>> Related Discussion thread on Github: > >> >>> https://github.com/apache/airflow/discussions/70119 > >> >>> > >> >>> Looking forward to get some thoughts on this. > >> >>> > >> >>> Thank you & Regards > >> >>> Prajwal > >> >>> > >> >> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] <mailto: > [email protected]> > >> For additional commands, e-mail: [email protected] <mailto: > [email protected]> > >> > >
