Concensus reached.
Shahar On Tue, Nov 3, 2026 at 7:06 PM Sameer Raj via dev <[email protected]> wrote: > Hi everyone! > > Following the discussion [1 > <https://lists.apache.org/thread/cdhdg60t0q03lmphbjrhjl21xys04wpn>], I > would like to call a lazy consensus on adding support for reloading > [dag_processor] dag_bundle_config_list through Airflow's _CMD / _SECRET > mechanism, as implemented in PR #74915 [2 > <https://github.com/apache/airflow/pull/74915>]. This is proposed for > Airflow 3.5. > > In our use case, we need to manage more than 500 Dag bundles, with > additions being automated. A deployment managed command can fetch the > complete bundle list from an internal API or a file, without requiring a > restart whenever a bundle is added or removed. > > The earlier DagBundleProvider abstraction has been dropped from this work. > A public interface can be proposed separately through a future AIP. This PR > introduces no new configuration option, bundle-management API, or database > migration. > > The proposed behaviour is: > > - Reload is opt-in through _CMD or _SECRET. Literal configuration keeps its > existing startup-only behaviour. The Dag processor checks for changes at > bundle_refresh_check_interval. > > - Added bundles begin processing. Removed bundles stop processing, and > unfiltered processors mark them inactive. The existing stale Dag scan then > marks their Dags stale. A successful read containing [] removes all > configured bundles. > > - Failed, invalid, or blank reads keep the last accepted list in the > running process. The configured command or secrets backend must return the > complete bundle list. Commands must report failures with a nonzero exit > status instead of returning []. > > - Processors started with --bundle-name stop processing removed bundles > locally, but never deactivate missing bundles globally. A processor that > sees the complete list is needed for global deactivation. > > - Additions and removals are eventually consistent across Dag processors. > Each processor checks for configuration changes independently. Unfiltered > processors continue reconciling bundle state in the database even when > their own accepted list has not changed. Processors may temporarily see > different lists and overwrite each other's active-state updates. They > converge once all surviving processors see the same complete, valid list > and continue reconciling. > > For v1, changing an existing bundle's definition, including tracking_ref or > classpath, rejects the entire read. Unrelated additions and removals in > that read are also held back until the changed entry is reverted or > published under a new bundle name. > > The accepted list is held only in memory. A fresh process accepts the > current valid configuration. It cannot recover a previous list if the > configured command or secrets backend fails or returns invalid > configuration. > > The PR also adds the optional DagFileParseRequest.bundle_config field. This > passes accepted importer settings and team ownership to parsing processes, > so they do not execute the command or query the secrets backend again. > Parser protocol compatibility handling is included. > > dag_bundle_config_list is marked sensitive: true to enable _CMD and > _SECRET. This also hides its value in configuration views that hide > sensitive options. The Deployment Manager continues to control the > configured command or secrets backend and its credentials. > > Components that resolve bundles, including workers, need the access > required to execute the configured command or query the secrets backend. > Existing Dag runs should finish before removing their bundle if those runs > need to complete, since tasks and callbacks can still need its definition. > > This lazy consensus will remain open for at least 72 hours, until Friday, 6 > November 2026, 23:59 UTC. > > Please share any concerns or objections. If there are no objections by > then, I will consider this scope accepted and ask a committer to proceed > with the PR, subject to the usual review and CI checks. > > [1] https://lists.apache.org/thread/cdhdg60t0q03lmphbjrhjl21xys04wpn > [2] https://github.com/apache/airflow/pull/74915 > > Thanks, > Sameer Raj >
