Hi everyone! Following up on this discussion, I would like to share the revised scope. The earlier proposal included a DagBundleProvider abstraction. I have dropped that abstraction from the current work. It can be proposed separately through a future AIP if needed. The new PR is limited to reloading the existing [dag_processor] dag_bundle_config_list through Airflow’s_CMD or_SECRET mechanism:
https://github.com/apache/airflow/pull/74915 Thanks to Shahar for helping me narrow down the scope and for reviewing the approach. In our use case, we need to manage more than 500 DAG bundles, with additions being automated. Restarting or redeploying Airflow processes whenever a bundle is added makes this difficult. We want to fetch the bundle list from an internal API. A deployment managed command can call that API and return the same JSON list already accepted by dag_bundle_config_list. The command could also read a file, or the list could come from a secrets backend. The DAG processor checks for changes at bundle_refresh_check_interval. Users do not need to change their configuration unless they want to enable reload. Users only need to configure _CMD or _SECRET if they want reload. The expected behavior is: - When a bundle is added, the Dag processor discovers it and starts processing its files. Existing bundle instances are reused. - When a bundle is removed, the processor stops processing it. An unfiltered processor (without --bundle-name) marks it inactive, and the existing stale DAG scan marks its dags stale. A successful read containing [] removes all bundles. - Failed commands, unavailable secrets, wrong configs, invalid new classpaths, and blank reads keep the last accepted list in that process. The source must return the complete list. Commands should return a non zero exit on failure, rather than returning []. - A fresh process has no previous list to fall back to. Invalid or unavailable configuration prevents startup. A valid [] starts with no bundles. - Processors started with --bundle-name stop processing removed bundles locally, but never deactivate missing bundles globally (retaining existing behaviour). A processor that sees the complete list is needed for global deactivation. - Multiple Dag processors may see changes at different times (due to variance in the refresh interval). Each unfiltered processor reconciles bundle in the database even when its own list has not changed, so it can correct a stale write from another processor. They eventually converge when surviving processors see the same list and continue reconciling. All components, i.e. the DAG processor and workers that resolve bundles need access to the configuration source. Existing Dag runs should finish before removing a bundle if those runs need to complete, since tasks and callbacks can still need its definition. For v1, changing an existing bundle’s definition is rejected, and the last accepted list is retained. This includes changing tracking_ref or classpath etc. Such a change requires a new bundle name or restarting the affected components. New commits on the already configured branch continue to be picked up through the existing bundle refresh behavior. Supporting changes to an existing bundle definition can be added later incase required! There is also an optional DagFileParseRequest.bundle_config field containing the accepted importer settings and team ownership. It allows the parsing process (to parse the DAG) to use the configuration accepted by its parent, without reading the command or secrets backend again and potentially seeing a different list. This does change the parser protocol but the compatibility handling is included in the PR The dag_bundle_config_list option is marked sensitive: true to enable _CMD and _SECRET. This also means the option is hidden in configuration views that hide sensitive values. The Deployment Manager continues to control the command or secrets backend and its credentials. No API for creating or changing bundle definitions is added. Related discussions by various users: - Runtime bundle management: https://github.com/apache/airflow/issues/61645 - Automatic bundle configuration loading: https://github.com/apache/airflow/discussions/59799 - Earlier file-based reload PR: https://github.com/apache/airflow/pull/71111 - Earlier dev-list discussion on runtime reload: https://lists.apache.org/thread/co0z6yxc1g6ctqqmt7rdszjhjsq0w2hx I would appreciate feedback on this revised scope, proposed flow and the state v1 limitation. Thanks, Sameer Raj On Wed, Sep 23, 2026 at 3:35 PM Sameer Raj <[email protected]> wrote: > Hi everyone! > I would like to discuss adding support for dynamically discovering new DAG > bundle configurations without requiring Airflow restarts. (following up on > comment: > https://github.com/apache/airflow/issues/61645#issuecomment-5017217627) > > There's also multiple PRs/discussion going over for the same feature: > - Runtime CRUD proposal: > https://github.com/apache/airflow/issues/61645 > - Automatic bundle configuration loading: > https://github.com/apache/airflow/discussions/59799 > - File-based hot-reload PR: > https://github.com/apache/airflow/pull/63928 > - Add dynamic DAG bundle configuration from file path > https://github.com/apache/airflow/pull/71111 > > *The Problem:* > > Today, Dag bundles are configured through [dag_processor] > dag_bundle_config_list. DagBundlesManager loads this configuration when > the process starts, and the Dag processor uses the same bundle list. > > This becomes operationally difficult for deployments that manage many > bundles and regularly add (or remove) them. Each configuration change > currently requires restarting or redeploying the affected Airflow processes. > > *Proposal* > > The proposal has two parts: > > First, introduce a DagBundleProvider abstraction into DagBundlesManager. > It would allow Airflow to discover the bundle configurations construct > bundles accordingly. Thus, the responsibility to fetch bundles is passed on > to DagBundleProvider by DagBundleManager. > > The configured class could obtain bundle information from a file, blob > storage, an internal service, or another deployment-specific source (and > this will decouple Airflow being aware of the source). Airflow would define > the class contract without adding direct support for every possible source. > > The existing dag_bundle_config_list would remain the default > implementation, so current deployments would keep the same behavior. > > Second, allow the Dag processor to periodically ask the configured class > for the latest bundle configurations from its existing processing loop. > > When a bundle is added, the Dag processor would discover it and begin > processing its files. When a bundle is removed, the Dag processor would > stop processing it and use the existing removed-file handling for its Dags. > > Each Dag processor performs this reconciliation periodically. Processors > may observe a change at slightly different times, but they will converge on > the configuration returned by the provider. > > *Why use a configurable class?* > > In this comment > <https://github.com/apache/airflow/issues/61645#issuecomment-5017217627>, > Jarek asked why a custom dynamic manager would be preferable to managing > Airflow configuration dynamically. > > On this, having this abstraction can allow users to implement their way by > which they plan to reload the configs, for example, fetch from external > system etc. Also changing the environment variable for bundle list wont > automatically reload the bundles, and the capability need to be built for > the same. > > *Security* > > The security implications of changing bundle definitions through the > Airflow API were raised in issue #61645 > <https://github.com/apache/airflow/issues/61645#issuecomment-4212433885>. > > This proposal keeps management of bundle definitions within the existing > Deployment Manager trust boundary. > > - The Deployment Manager selects and deploys the DagBundleProvider > implementation. > - The Deployment Manager controls access to its configuration source. > - Credentials and authorization for that source remain deployment > managed. > - The Airflow API does not expose operations for creating or changing > bundle definitions. > - No new Airflow user role is introduced for managing bundles. > > *Compatibility* > > The existing dag_bundle_config_list remains the default source and keeps > its current behavior. > > *Value Addition* > > Multiple users have raised PR/discuss thread on Airflow Github for the > same, and this feature will be very useful for the community who are > managing multiple bundles. Reference for the threads is shared above. > > *Current work* > > I have split the work into two PRs so each part can be reviewed separately: > > - Provider abstraction: > https://github.com/apache/airflow/pull/73209 > - Runtime configuration reload: > https://github.com/apache/airflow/pull/73217 > > I would appreciate feedback on this! > > Thanks, > Sameer Raj > > > > >
