I see, so the point is not about the error itself per se, but how the error propagates to other dags under the same scope. That makes sense, thanks for the clarification.
> On Aug 12, 2026, at 18:33, Kaxil Naik <[email protected]> wrote: > > Atleast the motivation to me based on the below (and at the start of the > original thread) seems to be an an error in one dag via a generator could > impact others. And that is because currently Airflow only needs dag > definition to be in a python file, hence my suggestion in the previous email > that this should be solved in a better way once we already accept non-python > dags natively -- eliminating the need of a driver python file that imports > other non-python files. > > > if some configs passed into the loader file are wrong. It is either “all” > > or “none”. If the driver file raises an error, it can deactivate all > > healthy DAGs belonging to that driver. Driver owners handle this by > > gracefully managing the exceptions in their code. > > On Wed, 12 Aug 2026 at 14:11, Tzu-ping Chung via dev <[email protected] > <mailto:[email protected]>> wrote: >> Perhaps I’m misunderstanding “DAG-scoped” in the original idea? If there’s >> no DAG to attach to, how can the message be DAG-scoped? “I know I can >> generate a DAG, but I can’t generate the DAG” doesn’t make that much sense >> to me. >> >> TP >> >> > On Aug 12, 2026, at 17:15, Kaxil Naik <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Agreed on the definitional split, and I should have been clearer about >> > what I meant. >> > >> > I wasn't suggesting import errors become DAG-scoped. I agree that an >> > import error means Airflow couldn't make sense of the input and the whole >> > file should fail. My point was about what counts as "the file". >> > >> > Today a driver file (Example: example_dag_factory.py for Dag factory as >> > shown in tutorial: >> > https://astronomer.github.io/dag-factory/latest/getting-started/quick-start-airflow-standalone/#step-6-generate-the-dag-from-yaml >> > ) is the parse unit, so the file scope is wrong by construction: the >> > thing that failed is one YAML config, but the unit Airflow keys the error >> > on is the Python bridge that loaded 500 of them. That is the entire reason >> > this looks like it needs DAG scoping. Once a YAML file is itself a DAG >> > file, which is what AIP-85 is for, or atleast it provides hook points for >> > it, the existing file-scoped semantics land in the right place on their >> > own. 3 files get import errors, 497 files parse fine, and no new contract >> > between the processor and a driver is needed. >> > >> > AIP-85 doesn't address error scoping, so you're right that it isn't >> > covered there. But it names this exact workaround as motivation: because >> > the processor only recognizes .py, teams either accept one bridge file as >> > a bottleneck or generate 1-to-1 Python files per config in CI to shard it. >> > Prajwal's setup is the second one. Removing that layer removes most of the >> > reported problem as a side effect, which is why I asked. >> > >> > On DagWarning, I agree it is the right home for "this imported, but you >> > should fix it". It can't carry Prajwal's case though. It is keyed on >> > (dag_id, warning_type) with a foreign key to dag.dag_id, so it needs a DAG >> > that was successfully produced, and a config broken enough to produce no >> > DAG has nothing to attach to. warning_type is also validated against a >> > closed enum today. Worth noting that AIP-85 already proposes relaxing that >> > to a namespaced string plus a context dict for frontend translation, so >> > the two threads do overlap there. >> > >> > The part I would separate out as a real bug, independent of both: >> > _update_import_errors sets is_stale=True on every DagModel sharing the >> > relative_fileloc. The healthy DAGs from that file are written and >> > serialized first, then staled because a sibling failed. Even with one YAML >> > per file that will cause us problems whenever a file legitimately defines >> > several DAGs, or when the failure comes from serialization or a dag_policy >> > rather than the parse itself, which is what Prajwal actually reported. >> > >> > Regards, >> > Kaxil >> > >> > On Wed, 12 Aug 2026 at 12:26, Tzu-ping Chung via dev >> > <[email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >> >> I don’t think it’s covered by AIP-85. >> >> >> >> However, doesn’t we already have DagWarning for this? Import errors are >> >> source-level errors that implies the dags cannot be parsed at all, and by >> >> definition should fail the entire thing without dag scopes because >> >> Airflow simply can’t make sense of the input. >> >> >> >> A DagWarning, on the other hand, means the dags can be imported, but >> >> there are still things you probably should fix. This already exists for a >> >> long time. >> >> >> >> TP >> >> >> >> >> >> > On Aug 12, 2026, at 15:13, Kaxil Naik <[email protected] >> >> > <mailto:[email protected]> <mailto:[email protected] >> >> > <mailto:[email protected]>>> wrote: >> >> > >> >> > Is this part of AIP-85 already? >> >> > >> >> > On Tue, 11 Aug 2026 at 15:00, Sumit Maheshwari <[email protected] >> >> > <mailto:[email protected]> <mailto:[email protected] >> >> > <mailto:[email protected]>>> >> >> > wrote: >> >> > >> >> >> Strong +1 on this feature, as almost all of the companies using >> >> >> Airflow are >> >> >> probably using one or more such DAG generator frameworks, and a >> >> >> problem in >> >> >> one of the configs causes all the DAGs to go into a stale state. >> >> >> >> >> >> I would like to write a Google Doc proposal or an AIP (whichever the >> >> >>> community prefers) and work on it. >> >> >> >> >> >> >> >> >> Will this change require changes in all the generator frameworks, or >> >> >> will >> >> >> it be backward compatible? If it can be implemented without forcing >> >> >> existing DAG generator frameworks, then I think we should not require >> >> >> an >> >> >> AIP. >> >> >> >> >> >> On Mon, Aug 10, 2026 at 6:32 PM Prajwal Agarwal >> >> >> <[email protected] <mailto:[email protected]> >> >> >> <mailto:[email protected] <mailto:[email protected]>>> >> >> >> wrote: >> >> >> >> >> >>> Hi Airflow Community, >> >> >>> *TL;DR:* There is no way to surface import errors alongside healthy >> >> >>> dags >> >> >>> from a driver (like dag- <https://github.com/astronomer/dag-factory >> >> >>>> factory >> >> >>> <https://github.com/astronomer/dag-factory>), if some configs passed >> >> >> into >> >> >>> the loader file are wrong. It is either “all” or “none”. If the driver >> >> >> file >> >> >>> raises an error, it can deactivate all healthy DAGs belonging to that >> >> >>> driver. Driver owners handle this by gracefully managing the >> >> >>> exceptions >> >> >> in >> >> >>> their code. >> >> >>> I would like to discuss potential solutions (or learn if one already >> >> >> exists >> >> >>> or is planned). >> >> >>> >> >> >>> *Details* >> >> >>> When a single Python file generates many DAGs — as dag-factory and >> >> >> similar >> >> >>> "driver file" patterns do — Airflow's import-error handling is >> >> >> file-scoped, >> >> >>> not DAG-scoped. One bad config in a factory that builds hundreds of >> >> >>> DAGs >> >> >>> currently forces an all-or-nothing outcome. >> >> >>> If the factory lets the exception propagate, the whole file fails to >> >> >> import >> >> >>> and every DAG it would have produced disappears >> >> >>> (has_import_error=True, >> >> >>> is_stale=True in dag-processor). >> >> >>> If the factory swallows the bad config to keep the others alive, the >> >> >>> failure is silent—no import error is surfaced to the end user, on the >> >> >>> Airflow UI. >> >> >>> There is no way to say, "These 3 configurations are broken, here is >> >> >>> why, >> >> >>> but the other 497 DAGs are healthy and should keep running." >> >> >>> I'd like to discuss a contract that makes that possible. >> >> >>> >> >> >>> *Current Behavior:* >> >> >>> Import errors are keyed by file, one string per file. >> >> >>> Staleness is applied to every DAG sharing the file's fileloc. >> >> >>> >> >> >>> *Proposed Solution (Details are intentionally missing)* >> >> >>> A contract between dag-processor and driver file to return healthy >> >> >>> dags >> >> >> as >> >> >>> well as import errors (per file or per dag, given it is not necessary >> >> >> that >> >> >>> dag_id is available) >> >> >>> >> >> >>> *Motivation* >> >> >>> We use Airflow to power 100s of dags from a single driver >> >> >>> (dag-factory in >> >> >>> our case) where our customers write YAML files. We do have some build >> >> >> time >> >> >>> validations, but serialization errors can happen due to dag_policies >> >> >>> execution and failure as well. We would like to provide better >> >> >>> observability over these failures to our end users. >> >> >>> >> >> >>> *Ask* >> >> >>> I would like to know the community's opinion on supporting such a >> >> >> feature. >> >> >>> If there is a use case, I would like to write a Google Doc proposal >> >> >>> or an >> >> >>> AIP (whichever the community prefers) and work on it. >> >> >>> >> >> >>> Related Discussion thread on Github: >> >> >>> https://github.com/apache/airflow/discussions/70119 >> >> >>> >> >> >>> Looking forward to get some thoughts on this. >> >> >>> >> >> >>> Thank you & Regards >> >> >>> Prajwal >> >> >>> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> >> <mailto:[email protected]> >> >> <mailto:[email protected] >> >> <mailto:[email protected]>> >> >> For additional commands, e-mail: [email protected] >> >> <mailto:[email protected]> <mailto:[email protected] >> >> <mailto:[email protected]>> >> >> >>
