amoghrajesh commented on code in PR #45931:
URL: https://github.com/apache/airflow/pull/45931#discussion_r2558826591
##########
airflow-core/docs/security/secrets/secrets-backend/index.rst:
##########
@@ -89,13 +100,21 @@ configure separate secrets backend for workers, you can do
that using:
[workers]
secrets_backend =
secrets_backend_kwargs =
-
+ backends_order =
Set ``secrets_backend`` to the fully qualified class name of the backend you
want to enable.
You can provide ``secrets_backend_kwargs`` with json and it will be passed as
kwargs to the ``__init__`` method of
your secrets backend for the workers.
+``backends_order`` comma-separated list of secret backends for workers. These
backends will be used in the order they are specified.
Review Comment:
```suggestion
``backends_order`` is a comma-separated list of secret backends for workers.
These backends will be used in the order they are specified.
```
##########
airflow-core/src/airflow/configuration.py:
##########
@@ -2348,28 +2334,101 @@ def get_custom_secret_backend(worker_mode: bool =
False) -> BaseSecretsBackend |
return secrets_backend_cls(**backend_kwargs)
+def get_importable_secret_backend(class_name: str | None) ->
BaseSecretsBackend | None:
+ """Get secret backend defined in the given class name."""
+ if class_name is not None:
+ secrets_backend_cls = import_string(class_name)
+ return secrets_backend_cls()
+ return None
+
+
+class Backends(Enum):
+ """Type of the secrets backend."""
+
+ ENVIRONMENT_VARIABLE = "environment_variable"
+ EXECUTION_API = "execution_api"
+ CUSTOM = "custom"
+ METASTORE = "metastore"
+
+
def initialize_secrets_backends(
- default_backends: list[str] = DEFAULT_SECRETS_SEARCH_PATH,
+ default_backends: list[str] | None = None,
) -> list[BaseSecretsBackend]:
"""
Initialize secrets backend.
* import secrets backend classes
* instantiate them and return them in a list
"""
- backend_list = []
worker_mode = False
- if default_backends != DEFAULT_SECRETS_SEARCH_PATH:
+ search_section = "secrets"
+ environment_variable_args: str | None = (
+ "airflow.secrets.environment_variables.EnvironmentVariablesBackend"
+ )
+ metastore_args: str | None = "airflow.secrets.metastore.MetastoreBackend"
+ execution_args: str | None = None
+ if default_backends is not None:
worker_mode = True
+ search_section = "workers"
+ environment_variable_args = (
+ environment_variable_args if environment_variable_args in
default_backends else None
+ )
+ metastore_args = metastore_args if metastore_args in default_backends
else None
+ execution_args = (
+
"airflow.sdk.execution_time.secrets.execution_api.ExecutionAPISecretsBackend"
+ if
"airflow.sdk.execution_time.secrets.execution_api.ExecutionAPISecretsBackend"
+ in default_backends
+ else None
+ )
Review Comment:
We are trying to avoid importing sdk in airflow core when possible
##########
airflow-core/docs/security/secrets/secrets-backend/index.rst:
##########
@@ -64,12 +66,21 @@ The ``[secrets]`` section has the following options:
[secrets]
backend =
backend_kwargs =
+ backends_order =
Set ``backend`` to the fully qualified class name of the backend you want to
enable.
You can provide ``backend_kwargs`` with json and it will be passed as kwargs
to the ``__init__`` method of
your secrets backend.
+``backends_order`` comma-separated list of secret backends. These backends
will be used in the order they are specified.
Review Comment:
```suggestion
``backends_order`` is a comma-separated list of secret backends. These
backends will be used in the order they are specified.
```
##########
airflow-core/src/airflow/configuration.py:
##########
@@ -2301,9 +2288,8 @@ def ensure_secrets_loaded(
"""
# Check if the secrets_backend_list contains only 2 default backends.
- # Check if we are loading the backends for worker too by checking if the
default_backends is equal
- # to DEFAULT_SECRETS_SEARCH_PATH.
- if len(secrets_backend_list) == 2 or default_backends !=
DEFAULT_SECRETS_SEARCH_PATH:
+ # Check if we are loading the backends for worker too by checking if the
default_backends is not None
+ if len(secrets_backend_list) == 2 or default_backends is not None:
return initialize_secrets_backends(default_backends=default_backends)
return secrets_backend_list
Review Comment:
With: https://github.com/apache/airflow/pull/57744 merged, there is a shared
config parser now. You will have to update this in sdk/configuration.py too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
