samraj2k opened a new pull request, #73209: URL: https://github.com/apache/airflow/pull/73209
Currently, Airflow reads Dag bundle configuration from `dag_bundle_config_list`. This works for static configuration, but it does not support configuration that can change while a Dag processor runs. Though Dag processor does not have capability yet to support runtime hot reload of bundles, but this PR is pre-requisite of adding hot reload functioanlity. Hot reload needs two separate changes: 1. A way to get bundle configuration from a source that can change. 2. A process that regularly reads that source and reconciles added and removed bundles. This PR implements the first change only. The plan in further PR is to add reload and reconciliation logic. For example, assume these bundles are active: ``` finance risk marketing ``` A deployment can later provide this list from a file, service, or another configuration system. DagBundlesManager must not need source-specific code for each option. A DagBundleProvider supplies: - The complete list of active bundle configurations. - A specific bundle for a retained Dag run. DagBundlesManager continues to manage validation, bundle state, teams, and database reconciliation. The existing dag_bundle_config_list remains the default configuration source, so this PR does not change current behavior. A later PR can periodically ask the provider for the current list: ``` Previous list: finance, risk, marketing Current list: finance, marketing, payments ``` The reconciliation process can then add payments and deactivate risk. That work does not require changes to the configuration source or its contract. This change adds `DagBundleProvider`. A provider returns the full list of active bundle configurations. The new `[dag_processor] dag_bundle_provider` option selects the provider class. `ConfigDagBundleProvider` is the default so we preserve the current `dag_bundle_config_list` behaviour. Validation: - 169 tests passed and 1 test was skipped. - The pre-commit checks passed. - The manual checks passed. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — OpenAI Codex (GPT-5.6) Generated-by: OpenAI Codex (GPT-5.6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
