Hi everyone! Following the discussion [1 <https://lists.apache.org/thread/cdhdg60t0q03lmphbjrhjl21xys04wpn>], I would like to call a lazy consensus on adding support for reloading [dag_processor] dag_bundle_config_list through Airflow's _CMD / _SECRET mechanism, as implemented in PR #74915 [2 <https://github.com/apache/airflow/pull/74915>]. This is proposed for Airflow 3.5.
In our use case, we need to manage more than 500 Dag bundles, with additions being automated. A deployment managed command can fetch the complete bundle list from an internal API or a file, without requiring a restart whenever a bundle is added or removed. The earlier DagBundleProvider abstraction has been dropped from this work. A public interface can be proposed separately through a future AIP. This PR introduces no new configuration option, bundle-management API, or database migration. The proposed behaviour is: - Reload is opt-in through _CMD or _SECRET. Literal configuration keeps its existing startup-only behaviour. The Dag processor checks for changes at bundle_refresh_check_interval. - Added bundles begin processing. Removed bundles stop processing, and unfiltered processors mark them inactive. The existing stale Dag scan then marks their Dags stale. A successful read containing [] removes all configured bundles. - Failed, invalid, or blank reads keep the last accepted list in the running process. The configured command or secrets backend must return the complete bundle list. Commands must report failures with a nonzero exit status instead of returning []. - Processors started with --bundle-name stop processing removed bundles locally, but never deactivate missing bundles globally. A processor that sees the complete list is needed for global deactivation. - Additions and removals are eventually consistent across Dag processors. Each processor checks for configuration changes independently. Unfiltered processors continue reconciling bundle state in the database even when their own accepted list has not changed. Processors may temporarily see different lists and overwrite each other's active-state updates. They converge once all surviving processors see the same complete, valid list and continue reconciling. For v1, changing an existing bundle's definition, including tracking_ref or classpath, rejects the entire read. Unrelated additions and removals in that read are also held back until the changed entry is reverted or published under a new bundle name. The accepted list is held only in memory. A fresh process accepts the current valid configuration. It cannot recover a previous list if the configured command or secrets backend fails or returns invalid configuration. The PR also adds the optional DagFileParseRequest.bundle_config field. This passes accepted importer settings and team ownership to parsing processes, so they do not execute the command or query the secrets backend again. Parser protocol compatibility handling is included. dag_bundle_config_list is marked sensitive: true to enable _CMD and _SECRET. This also hides its value in configuration views that hide sensitive options. The Deployment Manager continues to control the configured command or secrets backend and its credentials. Components that resolve bundles, including workers, need the access required to execute the configured command or query the secrets backend. Existing Dag runs should finish before removing their bundle if those runs need to complete, since tasks and callbacks can still need its definition. This lazy consensus will remain open for at least 72 hours, until Friday, 6 November 2026, 23:59 UTC. Please share any concerns or objections. If there are no objections by then, I will consider this scope accepted and ask a committer to proceed with the PR, subject to the usual review and CI checks. [1] https://lists.apache.org/thread/cdhdg60t0q03lmphbjrhjl21xys04wpn [2] https://github.com/apache/airflow/pull/74915 Thanks, Sameer Raj
