samraj2k opened a new pull request, #73217: URL: https://github.com/apache/airflow/pull/73217
Dag processors currently read the bundle list only at startup. A deployment must restart them to add, update, or remove a bundle. In a large deployment, these restarts are slow and different processors can use different lists during the rollout. This PR makes each Dag processor request the active bundle list at the existing `bundle_refresh_check_interval`. The provider returns the complete list of active bundles. A missing bundle is inactive. The contract does not add an `active` field. Each processor also reconciles this list with the metadata database on every check. This repeated reconciliation makes the database converge after processors observe a change at different times. For example: 1. Processor A still sees `[sales]`. 2. Processor B sees `[]` and marks `sales` inactive. 3. Processor A runs later and marks `sales` active again. Processor A then stops permanently. 4. Processor B runs its next check and marks `sales` inactive again. The final state follows the current provider output. This does not need a shared configuration revision, a new database table, or a new Dag column. When a bundle is removed, the Dag processor uses `handle_removed_files()` to remove its files from the local queue, stop active file processors, and clear file statistics. The existing stale-Dag scan then marks Dags from inactive bundles as stale. A bundle name identifies stable construction settings. If settings such as the bundle class, repository, branch, connection, or refresh interval change, the provider must use a new bundle name. The provider must continue to resolve the previous name while retained Dag runs can still need it. The default configuration provider can be used by filtered Dag processors that have different local bundle lists. In that case, absence from one processor's partial list does not deactivate a bundle. A dynamic provider supplies the complete active list, so a filtered processor can still reconcile every bundle while it parses only its assigned bundles. If the provider cannot read its source, it raises an exception. The Dag processor keeps the last valid list and tries again. If Airflow cannot construct a newly added bundle, it retries the addition on the next check. This PR depends on #73209. Related alternative: #71111. Validation: - Ruff formatting and lint checks passed. - Airflow core mypy passed. - Fast and manual prek checks passed. - The focused Breeze tests and the generated default-configuration check could not start because Docker Desktop requires an Uber organization sign-in on this machine. CI must run these checks. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-6); reviewed by @samraj2k before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
