Is this part of AIP-85 already? On Tue, 11 Aug 2026 at 15:00, Sumit Maheshwari <[email protected]> wrote:
> Strong +1 on this feature, as almost all of the companies using Airflow are > probably using one or more such DAG generator frameworks, and a problem in > one of the configs causes all the DAGs to go into a stale state. > > I would like to write a Google Doc proposal or an AIP (whichever the > > community prefers) and work on it. > > > Will this change require changes in all the generator frameworks, or will > it be backward compatible? If it can be implemented without forcing > existing DAG generator frameworks, then I think we should not require an > AIP. > > On Mon, Aug 10, 2026 at 6:32 PM Prajwal Agarwal <[email protected]> > wrote: > > > Hi Airflow Community, > > *TL;DR:* There is no way to surface import errors alongside healthy dags > > from a driver (like dag- <https://github.com/astronomer/dag-factory > > >factory > > <https://github.com/astronomer/dag-factory>), if some configs passed > into > > the loader file are wrong. It is either “all” or “none”. If the driver > file > > raises an error, it can deactivate all healthy DAGs belonging to that > > driver. Driver owners handle this by gracefully managing the exceptions > in > > their code. > > I would like to discuss potential solutions (or learn if one already > exists > > or is planned). > > > > *Details* > > When a single Python file generates many DAGs — as dag-factory and > similar > > "driver file" patterns do — Airflow's import-error handling is > file-scoped, > > not DAG-scoped. One bad config in a factory that builds hundreds of DAGs > > currently forces an all-or-nothing outcome. > > If the factory lets the exception propagate, the whole file fails to > import > > and every DAG it would have produced disappears (has_import_error=True, > > is_stale=True in dag-processor). > > If the factory swallows the bad config to keep the others alive, the > > failure is silent—no import error is surfaced to the end user, on the > > Airflow UI. > > There is no way to say, "These 3 configurations are broken, here is why, > > but the other 497 DAGs are healthy and should keep running." > > I'd like to discuss a contract that makes that possible. > > > > *Current Behavior:* > > Import errors are keyed by file, one string per file. > > Staleness is applied to every DAG sharing the file's fileloc. > > > > *Proposed Solution (Details are intentionally missing)* > > A contract between dag-processor and driver file to return healthy dags > as > > well as import errors (per file or per dag, given it is not necessary > that > > dag_id is available) > > > > *Motivation* > > We use Airflow to power 100s of dags from a single driver (dag-factory in > > our case) where our customers write YAML files. We do have some build > time > > validations, but serialization errors can happen due to dag_policies > > execution and failure as well. We would like to provide better > > observability over these failures to our end users. > > > > *Ask* > > I would like to know the community's opinion on supporting such a > feature. > > If there is a use case, I would like to write a Google Doc proposal or an > > AIP (whichever the community prefers) and work on it. > > > > Related Discussion thread on Github: > > https://github.com/apache/airflow/discussions/70119 > > > > Looking forward to get some thoughts on this. > > > > Thank you & Regards > > Prajwal > > >
