Hi everyone,
I would like to start a discussion on AIP-116: On-Demand DAG Parsing:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=430407790
Originally, AIP-116 originated alongside AIP-85 (DAG Importer) and has been
updated to align with the current Airflow 3 architecture and related
ongoing work (AIP-85, AIP-92, AIP-109, scheduler-routed callbacks, and
moving DAG parsing to executors).
Quick summary: Currently, Airflow relies exclusively on a continuous
background parsing loop, which leaves two key gaps:
1. CI/CD pipelines, IDE integrations, and AI agents have no
side-effect-free way to validate a DAG or bundle against a live Airflow
environment and receive immediate structured feedback without mutating
scheduler state.
2. Deployment pipelines have no synchronous barrier to publish a new bundle
revision, and immutable bundles waste compute by being continuously
re-parsed.
To deliver value incrementally, AIP-116 splits the proposal into two parts:
- *Part 1: Fast Feedback / Dry-Run* (no prerequisites): Adds `POST
/api/v2/dagBundles/{bundle_name}/parse` to run a read-only parse of a
bundle or relative path, returning DAG summaries, warnings, and import
errors with zero writes to the metadata DB.
- *Part 2: On-Demand Bundles & Publication*: Adds `POST
/api/v2/dagBundles/{bundle_name}/parseUpdate`, along with `parsing_mode =
"on_demand"` in `dag_bundle_config_list` to opt immutable bundles out of
continuous periodic re-parsing once several prerequisite works are in place.
Please take a look at the AIP page for more details and share your
thoughts, questions, or feedback in this thread (or in comments on
Confluence).
Thanks,
Dilnaz Amanzholova
