GitHub user Prajwal7842 created a discussion: [Discussion] DAG-scoped import errors for factory/driver files (e.g. dag-factory)
### Summary Hi folks, starting this discussion to discuss “dag-scoped import errors” for driver files like dag-factory. **TL’DR:** There is no way to surface import errors as well as healthy dags from a driver (like dag-factory), if some configs passed into the generator are wrong. It is either “all” or “none”. I would like to discuss potential solutions (or would like to know if it already exists or is planned). ### Details When a single Python file generates many DAGs — as dag-factory and similar "driver file" patterns do — Airflow's import-error handling is file-scoped, not DAG-scoped. One bad config in a factory that builds hundreds of DAGs currently forces an all-or-nothing outcome. If the factory lets the exception propagate, the whole file fails to import and every DAG it would have produced disappears (has_import_error=True, is_stale=True in dag-processor). If the factory swallows the bad config to keep the others alive, the failure is silent — no import error is surfaced to the author at all. There is no way to say "these 3 configs are broken, here is why, but the other 497 DAGs are healthy and should keep running." I'd like to discuss a contract that makes that possible. ### Current Behavior: Import errors are keyed by file, one string per file. Staleness is applied to every DAG sharing the file's fileloc. ### Proposed Solution: A contract between dag-processor and driver file to return healthy dags as well as import errors (per file or per dag, given it is not necessary that dag_id is available) ### Questions Does something like this already exist / is it planned? Per-file concatenation preserves file level import errors. Is the solution acceptable ? GitHub link: https://github.com/apache/airflow/discussions/70119 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
