Perhaps I’m misunderstanding “DAG-scoped” in the original idea? If there’s no DAG to attach to, how can the message be DAG-scoped? “I know I can generate a DAG, but I can’t generate the DAG” doesn’t make that much sense to me.
TP > On Aug 12, 2026, at 17:15, Kaxil Naik <[email protected]> wrote: > > Agreed on the definitional split, and I should have been clearer about what I > meant. > > I wasn't suggesting import errors become DAG-scoped. I agree that an import > error means Airflow couldn't make sense of the input and the whole file > should fail. My point was about what counts as "the file". > > Today a driver file (Example: example_dag_factory.py for Dag factory as shown > in tutorial: > https://astronomer.github.io/dag-factory/latest/getting-started/quick-start-airflow-standalone/#step-6-generate-the-dag-from-yaml > ) is the parse unit, so the file scope is wrong by construction: the thing > that failed is one YAML config, but the unit Airflow keys the error on is the > Python bridge that loaded 500 of them. That is the entire reason this looks > like it needs DAG scoping. Once a YAML file is itself a DAG file, which is > what AIP-85 is for, or atleast it provides hook points for it, the existing > file-scoped semantics land in the right place on their own. 3 files get > import errors, 497 files parse fine, and no new contract between the > processor and a driver is needed. > > AIP-85 doesn't address error scoping, so you're right that it isn't covered > there. But it names this exact workaround as motivation: because the > processor only recognizes .py, teams either accept one bridge file as a > bottleneck or generate 1-to-1 Python files per config in CI to shard it. > Prajwal's setup is the second one. Removing that layer removes most of the > reported problem as a side effect, which is why I asked. > > On DagWarning, I agree it is the right home for "this imported, but you > should fix it". It can't carry Prajwal's case though. It is keyed on (dag_id, > warning_type) with a foreign key to dag.dag_id, so it needs a DAG that was > successfully produced, and a config broken enough to produce no DAG has > nothing to attach to. warning_type is also validated against a closed enum > today. Worth noting that AIP-85 already proposes relaxing that to a > namespaced string plus a context dict for frontend translation, so the two > threads do overlap there. > > The part I would separate out as a real bug, independent of both: > _update_import_errors sets is_stale=True on every DagModel sharing the > relative_fileloc. The healthy DAGs from that file are written and serialized > first, then staled because a sibling failed. Even with one YAML per file that > will cause us problems whenever a file legitimately defines several DAGs, or > when the failure comes from serialization or a dag_policy rather than the > parse itself, which is what Prajwal actually reported. > > Regards, > Kaxil > > On Wed, 12 Aug 2026 at 12:26, Tzu-ping Chung via dev <[email protected] > <mailto:[email protected]>> wrote: >> I don’t think it’s covered by AIP-85. >> >> However, doesn’t we already have DagWarning for this? Import errors are >> source-level errors that implies the dags cannot be parsed at all, and by >> definition should fail the entire thing without dag scopes because Airflow >> simply can’t make sense of the input. >> >> A DagWarning, on the other hand, means the dags can be imported, but there >> are still things you probably should fix. This already exists for a long >> time. >> >> TP >> >> >> > On Aug 12, 2026, at 15:13, Kaxil Naik <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Is this part of AIP-85 already? >> > >> > On Tue, 11 Aug 2026 at 15:00, Sumit Maheshwari <[email protected] >> > <mailto:[email protected]>> >> > wrote: >> > >> >> Strong +1 on this feature, as almost all of the companies using Airflow >> >> are >> >> probably using one or more such DAG generator frameworks, and a problem in >> >> one of the configs causes all the DAGs to go into a stale state. >> >> >> >> I would like to write a Google Doc proposal or an AIP (whichever the >> >>> community prefers) and work on it. >> >> >> >> >> >> Will this change require changes in all the generator frameworks, or will >> >> it be backward compatible? If it can be implemented without forcing >> >> existing DAG generator frameworks, then I think we should not require an >> >> AIP. >> >> >> >> On Mon, Aug 10, 2026 at 6:32 PM Prajwal Agarwal <[email protected] >> >> <mailto:[email protected]>> >> >> wrote: >> >> >> >>> Hi Airflow Community, >> >>> *TL;DR:* There is no way to surface import errors alongside healthy dags >> >>> from a driver (like dag- <https://github.com/astronomer/dag-factory >> >>>> factory >> >>> <https://github.com/astronomer/dag-factory>), if some configs passed >> >> into >> >>> the loader file are wrong. It is either “all” or “none”. If the driver >> >> file >> >>> raises an error, it can deactivate all healthy DAGs belonging to that >> >>> driver. Driver owners handle this by gracefully managing the exceptions >> >> in >> >>> their code. >> >>> I would like to discuss potential solutions (or learn if one already >> >> exists >> >>> or is planned). >> >>> >> >>> *Details* >> >>> When a single Python file generates many DAGs — as dag-factory and >> >> similar >> >>> "driver file" patterns do — Airflow's import-error handling is >> >> file-scoped, >> >>> not DAG-scoped. One bad config in a factory that builds hundreds of DAGs >> >>> currently forces an all-or-nothing outcome. >> >>> If the factory lets the exception propagate, the whole file fails to >> >> import >> >>> and every DAG it would have produced disappears (has_import_error=True, >> >>> is_stale=True in dag-processor). >> >>> If the factory swallows the bad config to keep the others alive, the >> >>> failure is silent—no import error is surfaced to the end user, on the >> >>> Airflow UI. >> >>> There is no way to say, "These 3 configurations are broken, here is why, >> >>> but the other 497 DAGs are healthy and should keep running." >> >>> I'd like to discuss a contract that makes that possible. >> >>> >> >>> *Current Behavior:* >> >>> Import errors are keyed by file, one string per file. >> >>> Staleness is applied to every DAG sharing the file's fileloc. >> >>> >> >>> *Proposed Solution (Details are intentionally missing)* >> >>> A contract between dag-processor and driver file to return healthy dags >> >> as >> >>> well as import errors (per file or per dag, given it is not necessary >> >> that >> >>> dag_id is available) >> >>> >> >>> *Motivation* >> >>> We use Airflow to power 100s of dags from a single driver (dag-factory in >> >>> our case) where our customers write YAML files. We do have some build >> >> time >> >>> validations, but serialization errors can happen due to dag_policies >> >>> execution and failure as well. We would like to provide better >> >>> observability over these failures to our end users. >> >>> >> >>> *Ask* >> >>> I would like to know the community's opinion on supporting such a >> >> feature. >> >>> If there is a use case, I would like to write a Google Doc proposal or an >> >>> AIP (whichever the community prefers) and work on it. >> >>> >> >>> Related Discussion thread on Github: >> >>> https://github.com/apache/airflow/discussions/70119 >> >>> >> >>> Looking forward to get some thoughts on this. >> >>> >> >>> Thank you & Regards >> >>> Prajwal >> >>> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> For additional commands, e-mail: [email protected] >> <mailto:[email protected]> >>
