Atleast the motivation to me based on the below (and at the start of the original thread) seems to be an an error in one dag via a generator could impact others. And that is because currently Airflow only needs dag definition to be in a python file, hence my suggestion in the previous email that this should be solved in a better way once we already accept non-python dags natively -- eliminating the need of a driver python file that imports other non-python files.
> if some configs passed into the loader file are wrong. It is either “all” or “none”. If the driver file raises an error, it can deactivate all healthy DAGs belonging to that driver. Driver owners handle this by gracefully managing the exceptions in their code. On Wed, 12 Aug 2026 at 14:11, Tzu-ping Chung via dev <[email protected]> wrote: > Perhaps I’m misunderstanding “DAG-scoped” in the original idea? If there’s > no DAG to attach to, how can the message be DAG-scoped? “I know I can > generate a DAG, but I can’t generate the DAG” doesn’t make that much sense > to me. > > TP > > > On Aug 12, 2026, at 17:15, Kaxil Naik <[email protected]> wrote: > > > > Agreed on the definitional split, and I should have been clearer about > what I meant. > > > > I wasn't suggesting import errors become DAG-scoped. I agree that an > import error means Airflow couldn't make sense of the input and the whole > file should fail. My point was about what counts as "the file". > > > > Today a driver file (Example: example_dag_factory.py for Dag factory as > shown in tutorial: > https://astronomer.github.io/dag-factory/latest/getting-started/quick-start-airflow-standalone/#step-6-generate-the-dag-from-yaml > ) is the parse unit, so the file scope is wrong by construction: the thing > that failed is one YAML config, but the unit Airflow keys the error on is > the Python bridge that loaded 500 of them. That is the entire reason this > looks like it needs DAG scoping. Once a YAML file is itself a DAG file, > which is what AIP-85 is for, or atleast it provides hook points for it, the > existing file-scoped semantics land in the right place on their own. 3 > files get import errors, 497 files parse fine, and no new contract between > the processor and a driver is needed. > > > > AIP-85 doesn't address error scoping, so you're right that it isn't > covered there. But it names this exact workaround as motivation: because > the processor only recognizes .py, teams either accept one bridge file as a > bottleneck or generate 1-to-1 Python files per config in CI to shard it. > Prajwal's setup is the second one. Removing that layer removes most of the > reported problem as a side effect, which is why I asked. > > > > On DagWarning, I agree it is the right home for "this imported, but you > should fix it". It can't carry Prajwal's case though. It is keyed on > (dag_id, warning_type) with a foreign key to dag.dag_id, so it needs a DAG > that was successfully produced, and a config broken enough to produce no > DAG has nothing to attach to. warning_type is also validated against a > closed enum today. Worth noting that AIP-85 already proposes relaxing that > to a namespaced string plus a context dict for frontend translation, so the > two threads do overlap there. > > > > The part I would separate out as a real bug, independent of both: > _update_import_errors sets is_stale=True on every DagModel sharing the > relative_fileloc. The healthy DAGs from that file are written and > serialized first, then staled because a sibling failed. Even with one YAML > per file that will cause us problems whenever a file legitimately defines > several DAGs, or when the failure comes from serialization or a dag_policy > rather than the parse itself, which is what Prajwal actually reported. > > > > Regards, > > Kaxil > > > > On Wed, 12 Aug 2026 at 12:26, Tzu-ping Chung via dev < > [email protected] <mailto:[email protected]>> wrote: > >> I don’t think it’s covered by AIP-85. > >> > >> However, doesn’t we already have DagWarning for this? Import errors are > source-level errors that implies the dags cannot be parsed at all, and by > definition should fail the entire thing without dag scopes because Airflow > simply can’t make sense of the input. > >> > >> A DagWarning, on the other hand, means the dags can be imported, but > there are still things you probably should fix. This already exists for a > long time. > >> > >> TP > >> > >> > >> > On Aug 12, 2026, at 15:13, Kaxil Naik <[email protected] <mailto: > [email protected]>> wrote: > >> > > >> > Is this part of AIP-85 already? > >> > > >> > On Tue, 11 Aug 2026 at 15:00, Sumit Maheshwari < > [email protected] <mailto:[email protected]>> > >> > wrote: > >> > > >> >> Strong +1 on this feature, as almost all of the companies using > Airflow are > >> >> probably using one or more such DAG generator frameworks, and a > problem in > >> >> one of the configs causes all the DAGs to go into a stale state. > >> >> > >> >> I would like to write a Google Doc proposal or an AIP (whichever the > >> >>> community prefers) and work on it. > >> >> > >> >> > >> >> Will this change require changes in all the generator frameworks, or > will > >> >> it be backward compatible? If it can be implemented without forcing > >> >> existing DAG generator frameworks, then I think we should not > require an > >> >> AIP. > >> >> > >> >> On Mon, Aug 10, 2026 at 6:32 PM Prajwal Agarwal < > [email protected] <mailto:[email protected]>> > >> >> wrote: > >> >> > >> >>> Hi Airflow Community, > >> >>> *TL;DR:* There is no way to surface import errors alongside healthy > dags > >> >>> from a driver (like dag- <https://github.com/astronomer/dag-factory > >> >>>> factory > >> >>> <https://github.com/astronomer/dag-factory>), if some configs > passed > >> >> into > >> >>> the loader file are wrong. It is either “all” or “none”. If the > driver > >> >> file > >> >>> raises an error, it can deactivate all healthy DAGs belonging to > that > >> >>> driver. Driver owners handle this by gracefully managing the > exceptions > >> >> in > >> >>> their code. > >> >>> I would like to discuss potential solutions (or learn if one already > >> >> exists > >> >>> or is planned). > >> >>> > >> >>> *Details* > >> >>> When a single Python file generates many DAGs — as dag-factory and > >> >> similar > >> >>> "driver file" patterns do — Airflow's import-error handling is > >> >> file-scoped, > >> >>> not DAG-scoped. One bad config in a factory that builds hundreds of > DAGs > >> >>> currently forces an all-or-nothing outcome. > >> >>> If the factory lets the exception propagate, the whole file fails to > >> >> import > >> >>> and every DAG it would have produced disappears > (has_import_error=True, > >> >>> is_stale=True in dag-processor). > >> >>> If the factory swallows the bad config to keep the others alive, the > >> >>> failure is silent—no import error is surfaced to the end user, on > the > >> >>> Airflow UI. > >> >>> There is no way to say, "These 3 configurations are broken, here is > why, > >> >>> but the other 497 DAGs are healthy and should keep running." > >> >>> I'd like to discuss a contract that makes that possible. > >> >>> > >> >>> *Current Behavior:* > >> >>> Import errors are keyed by file, one string per file. > >> >>> Staleness is applied to every DAG sharing the file's fileloc. > >> >>> > >> >>> *Proposed Solution (Details are intentionally missing)* > >> >>> A contract between dag-processor and driver file to return healthy > dags > >> >> as > >> >>> well as import errors (per file or per dag, given it is not > necessary > >> >> that > >> >>> dag_id is available) > >> >>> > >> >>> *Motivation* > >> >>> We use Airflow to power 100s of dags from a single driver > (dag-factory in > >> >>> our case) where our customers write YAML files. We do have some > build > >> >> time > >> >>> validations, but serialization errors can happen due to dag_policies > >> >>> execution and failure as well. We would like to provide better > >> >>> observability over these failures to our end users. > >> >>> > >> >>> *Ask* > >> >>> I would like to know the community's opinion on supporting such a > >> >> feature. > >> >>> If there is a use case, I would like to write a Google Doc proposal > or an > >> >>> AIP (whichever the community prefers) and work on it. > >> >>> > >> >>> Related Discussion thread on Github: > >> >>> https://github.com/apache/airflow/discussions/70119 > >> >>> > >> >>> Looking forward to get some thoughts on this. > >> >>> > >> >>> Thank you & Regards > >> >>> Prajwal > >> >>> > >> >> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] <mailto: > [email protected]> > >> For additional commands, e-mail: [email protected] <mailto: > [email protected]> > >> > >
