Taragolis commented on PR #23560: URL: https://github.com/apache/airflow/pull/23560#issuecomment-1164779057
> airflow provides support for grabbing certain "secret" config values from secrets backend. and this logic assumes there's one custom secrets backend. and this logic assumes there's one custom secrets backend. and given that, as well as the difficulties you've encountered with trying to implement deprecation of the old values, it seems it may not be worth the complexity tradeoff to go with the "arbitrary" approach, and instead go the more limited route I add some changes when switch to the list. Now it works in the same way that secrets backends work for Variables and Connections. Env and Metastore just not implements `get_config` method so it always return None and try to next backend in the list https://github.com/apache/airflow/blob/68e0741b8cd0554948be4db7946d8c7afaa5b4d0/airflow/configuration.py#L125-L140 >and for those users who really need to implement multiple external secrets backends, they can still do so by adding a custom secrets backend that hits multiple sources. It is not so long if compare to current. If we do not want to provide ability drop EnvVar and Metastore we could simply add in L1603 `backend_configs.extend(default_configs)` and remove from L1602 https://github.com/apache/airflow/blob/68e0741b8cd0554948be4db7946d8c7afaa5b4d0/airflow/configuration.py#L1601-L1605 And it only add to the list if user not specified it yet current implementation ```ini [secrets] backend = "airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend" backend_kwargs ={"connections_prefix":"/airflow/connections","variables_prefix":"/airflow/variables","profile_name":"default"} ``` proposal ```ini [secrets] # It could be just backends backends_config = [{"backend":"airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend","backend_kwargs":{"connections_prefix":"/airflow/connections","variables_prefix":"/airflow/variables","profile_name":"default"}}] ``` The only one difference if user now specify backend=airflow.secrets.metastore.MetastoreBackend than result now `[MetastoreBackend, EnvironmentVariablesBackend, MetastoreBackend]` with current proposal `[MetastoreBackend, EnvironmentVariablesBackend]` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
