jason810496 opened a new pull request, #62696: URL: https://github.com/apache/airflow/pull/62696
related: #60000 ## Why Resolve the blocker for #60000 (replacing all Core `conf` usage in providers). As pointed out in https://github.com/apache/airflow/pull/60074#issuecomment-3711902451, we have to make the Task SDK `conf` respect the provider default values. ## How Currently, there are two ways to specify default values for providers 1. `provider_config_fallback_defaults.cfg` (outdated with default value specified in `provider.yaml`, but it still works in Core) 2. provider metadata (the `get_provider_info.py` entry point via metadata import in `ProvidersManager`) The current status before this PR is 1. Only Core `conf` respects `provider_config_fallback_defaults.cfg` 2. **Neither** Core `conf` **nor** SDK `conf` respects the default values from provider metadata Since both Core and SDK `conf` require access to the provider config default values, we add new shared logic that respects default config from provider metadata. ## What Refactor both Core `conf` and Task SDK `conf` by consolidating all lookups of provider default values into a shared library, including: - Move the existing logic for reading default values from `provider_config_fallback_defaults.cfg` from Core `conf` to a shared module. - Add a `provider_configs` property in `ProvidersManagerTaskRuntime`. - Add `_get_option_from_provider_cfg_config_fallbacks` and `_get_option_from_provider_metadata_config_fallbacks` into the shared `_lookup_sequence`, and ensure that `ProvidersManager` / `ProvidersManagerTaskRuntime` are initialized lazily. ## Verification After the patch, all of the following `conf` usages will show the correct default values. For Core: ```python import contextlib from airflow.configuration import conf print(conf.get("core", "executor")) # defined in provider_config_fallback_defaults.cfg, already works before the fix print(conf.get("kubernetes_executor", "tcp_keep_idle")) # defined in Amazon provider.yaml, will raise an exception before the fix with contextlib.suppress(Exception): print(conf.get("aws_lambda_executor", "conn_id")) ``` For TaskSDK: ```python import contextlib from airflow.sdk import conf print(conf.get("core", "executor")) # define in provider_config_fallback_defaults.cfg, will raise exception before the fix with contextlib.suppress(Exception): print(conf.get("kubernetes_executor", "tcp_keep_idle")) # define in Amazon provider.yaml, will raise exception before the fix with contextlib.suppress(Exception): print(conf.get("aws_lambda_executor", "conn_id")) # provider.yaml ``` ## TODO / Follow-up - update unit tests - add prek script to enforce updating `provider.yaml` when adding new `[section/option]` in community providers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
