Personally, I prefer the interface approach as it aligns better with enterprise-grade software. That said, I am also fine with proceeding via the _cmd and _secret route for the time being.
Thanks, Sumit On Fri, Oct 30, 2026 at 11:47 AM Shahar Epstein <[email protected]> wrote: > Hey Sameer and everyone, > > +1 to the revised scope in #74915. > > The need has come up several times: #61645, discussion #59799, #63928, and > #71111 all asked for this in different forms. I think it’s a good time to > add the feature. > > I think it can proceed without an AIP for the following reasons: > > 1. The revised scope adds no pluggable abstraction, config entry, API, or > DB migration. Reloading uses the existing dag_bundle_config_list format, > supplied through Airflow’s existing _cmd / _secret mechanism. The PR makes > that mechanism available for this key by marking it as sensitive. > 2. It’s opt-in. A value set through a plain environment variable or > airflow.cfg keeps today’s startup-only behavior. _CMD / _SECRET were not > available for this key before, so no existing deployment will start > reloading without an explicit change. > 3. The trust boundary does not change. Bundle definitions remain under the > Deployment Manager’s control; users cannot change them through Airflow. > > Neither of the two previous threads has received a reply yet. Sameer, I > suggest waiting few more days (let's say until Monday, Nov. 2nd) to give a > chance for others to comment on this thread, and following up with a [LAZY > CONSENSUS] thread on the revised scope if there are no responses by then, > or there's consensus. > > Disclosure: I helped Sameer narrow the scope and I’m reviewing the PR, so > please take that into account when weighing my comment. > > > Shahar > > On Sun, Oct 25, 2026 at 12:33 PM Sameer Raj via dev < > [email protected]> > wrote: > > > Hi everyone! > > > > Following up on this discussion, I would like to share the revised scope. > > The earlier proposal included a DagBundleProvider abstraction. I have > > dropped that abstraction from the current work. It can be proposed > > separately through a future AIP if needed. The new PR is limited to > > reloading the existing [dag_processor] dag_bundle_config_list through > > Airflow’s_CMD or_SECRET mechanism: > > > > https://github.com/apache/airflow/pull/74915 > > > > Thanks to Shahar for helping me narrow down the scope and for reviewing > the > > approach. > > > > In our use case, we need to manage more than 500 DAG bundles, with > > additions being automated. Restarting or redeploying Airflow processes > > whenever a bundle is added makes this difficult. We want to fetch the > > bundle list from an internal API. A deployment managed command can call > > that API and return the same JSON list already accepted by > > dag_bundle_config_list. The command could also read a file, or the list > > could come from a secrets backend. > > > > The DAG processor checks for changes at > > bundle_refresh_check_interval. Users do not need to change their > > configuration unless they want to enable reload. Users only need to > > configure _CMD or _SECRET if they want reload. > > > > The expected behavior is: > > > > - When a bundle is added, the Dag processor discovers it and starts > > processing its files. Existing bundle instances are reused. > > - When a bundle is removed, the processor stops processing it. An > > unfiltered processor (without --bundle-name) marks it inactive, and > the > > existing stale DAG scan marks its dags stale. A successful read > > containing > > [] removes all bundles. > > - Failed commands, unavailable secrets, wrong configs, invalid new > > classpaths, and blank reads keep the last accepted list in that > process. > > The source must return the complete list. Commands should return a non > > zero > > exit on failure, rather than returning []. > > - A fresh process has no previous list to fall back to. Invalid or > > unavailable configuration prevents startup. A valid [] starts with no > > bundles. > > - Processors started with --bundle-name stop processing removed > bundles > > locally, but never deactivate missing bundles globally (retaining > > existing > > behaviour). A processor that sees the complete list is needed for > global > > deactivation. > > - Multiple Dag processors may see changes at different times (due to > > variance in the refresh interval). Each unfiltered processor > reconciles > > bundle in the database even when its own list has not changed, so it > can > > correct a stale write from another processor. They eventually converge > > when > > surviving processors see the same list and continue reconciling. > > > > All components, i.e. the DAG processor and workers that resolve bundles > > need access to the configuration source. Existing Dag runs should finish > > before removing a bundle if those runs need to complete, since tasks and > > callbacks can still need its definition. > > > > For v1, changing an existing bundle’s definition is rejected, and the > last > > accepted list is retained. This includes changing tracking_ref or > classpath > > etc. Such a change requires a new bundle name or restarting the affected > > components. New commits on the already configured branch continue to be > > picked up through the existing bundle refresh behavior. Supporting > changes > > to an existing bundle definition can be added later incase required! > > > > There is also an optional DagFileParseRequest.bundle_config field > > containing the accepted importer settings and team ownership. It allows > the > > parsing process (to parse the DAG) to use the configuration accepted by > its > > parent, without reading the command or secrets backend again and > > potentially seeing a different list. This does change the parser protocol > > but the compatibility handling is included in the PR > > > > The dag_bundle_config_list option is marked sensitive: true to enable > _CMD > > and _SECRET. This also means the option is hidden in configuration views > > that hide sensitive values. The Deployment Manager continues to control > the > > command or secrets backend and its credentials. No API for creating or > > changing bundle definitions is added. > > > > Related discussions by various users: > > > > - Runtime bundle management: > > https://github.com/apache/airflow/issues/61645 > > - Automatic bundle configuration loading: > > https://github.com/apache/airflow/discussions/59799 > > - Earlier file-based reload PR: > > https://github.com/apache/airflow/pull/71111 > > - Earlier dev-list discussion on runtime reload: > > https://lists.apache.org/thread/co0z6yxc1g6ctqqmt7rdszjhjsq0w2hx > > > > I would appreciate feedback on this revised scope, proposed flow and the > > state v1 limitation. > > > > Thanks, > > Sameer Raj > > > > On Wed, Sep 23, 2026 at 3:35 PM Sameer Raj <[email protected]> wrote: > > > > > Hi everyone! > > > I would like to discuss adding support for dynamically discovering new > > DAG > > > bundle configurations without requiring Airflow restarts. (following up > > on > > > comment: > > > https://github.com/apache/airflow/issues/61645#issuecomment-5017217627 > ) > > > > > > There's also multiple PRs/discussion going over for the same feature: > > > - Runtime CRUD proposal: > > > https://github.com/apache/airflow/issues/61645 > > > - Automatic bundle configuration loading: > > > https://github.com/apache/airflow/discussions/59799 > > > - File-based hot-reload PR: > > > https://github.com/apache/airflow/pull/63928 > > > - Add dynamic DAG bundle configuration from file path > > > https://github.com/apache/airflow/pull/71111 > > > > > > *The Problem:* > > > > > > Today, Dag bundles are configured through [dag_processor] > > > dag_bundle_config_list. DagBundlesManager loads this configuration when > > > the process starts, and the Dag processor uses the same bundle list. > > > > > > This becomes operationally difficult for deployments that manage many > > > bundles and regularly add (or remove) them. Each configuration change > > > currently requires restarting or redeploying the affected Airflow > > processes. > > > > > > *Proposal* > > > > > > The proposal has two parts: > > > > > > First, introduce a DagBundleProvider abstraction into > DagBundlesManager. > > > It would allow Airflow to discover the bundle configurations construct > > > bundles accordingly. Thus, the responsibility to fetch bundles is > passed > > on > > > to DagBundleProvider by DagBundleManager. > > > > > > The configured class could obtain bundle information from a file, blob > > > storage, an internal service, or another deployment-specific source > (and > > > this will decouple Airflow being aware of the source). Airflow would > > define > > > the class contract without adding direct support for every possible > > source. > > > > > > The existing dag_bundle_config_list would remain the default > > > implementation, so current deployments would keep the same behavior. > > > > > > Second, allow the Dag processor to periodically ask the configured > class > > > for the latest bundle configurations from its existing processing loop. > > > > > > When a bundle is added, the Dag processor would discover it and begin > > > processing its files. When a bundle is removed, the Dag processor would > > > stop processing it and use the existing removed-file handling for its > > Dags. > > > > > > Each Dag processor performs this reconciliation periodically. > Processors > > > may observe a change at slightly different times, but they will > converge > > on > > > the configuration returned by the provider. > > > > > > *Why use a configurable class?* > > > > > > In this comment > > > < > https://github.com/apache/airflow/issues/61645#issuecomment-5017217627 > > >, > > > Jarek asked why a custom dynamic manager would be preferable to > managing > > > Airflow configuration dynamically. > > > > > > On this, having this abstraction can allow users to implement their way > > by > > > which they plan to reload the configs, for example, fetch from external > > > system etc. Also changing the environment variable for bundle list wont > > > automatically reload the bundles, and the capability need to be built > for > > > the same. > > > > > > *Security* > > > > > > The security implications of changing bundle definitions through the > > > Airflow API were raised in issue #61645 > > > < > https://github.com/apache/airflow/issues/61645#issuecomment-4212433885 > > >. > > > > > > This proposal keeps management of bundle definitions within the > existing > > > Deployment Manager trust boundary. > > > > > > - The Deployment Manager selects and deploys the DagBundleProvider > > > implementation. > > > - The Deployment Manager controls access to its configuration > source. > > > - Credentials and authorization for that source remain deployment > > > managed. > > > - The Airflow API does not expose operations for creating or > changing > > > bundle definitions. > > > - No new Airflow user role is introduced for managing bundles. > > > > > > *Compatibility* > > > > > > The existing dag_bundle_config_list remains the default source and > keeps > > > its current behavior. > > > > > > *Value Addition* > > > > > > Multiple users have raised PR/discuss thread on Airflow Github for the > > > same, and this feature will be very useful for the community who are > > > managing multiple bundles. Reference for the threads is shared above. > > > > > > *Current work* > > > > > > I have split the work into two PRs so each part can be reviewed > > separately: > > > > > > - Provider abstraction: > > > https://github.com/apache/airflow/pull/73209 > > > - Runtime configuration reload: > > > https://github.com/apache/airflow/pull/73217 > > > > > > I would appreciate feedback on this! > > > > > > Thanks, > > > Sameer Raj > > > > > > > > > > > > > > > > > >
