kaxil commented on issue #16952:
URL: https://github.com/apache/airflow/issues/16952#issuecomment-880281931
I think the priority is correct and will cause confusion if changed later
on. And with customer Secrets Backend, you can mix and match however you like.
We intentionally did this (same fox XCom Backend) so that companies can create
one for their own needs as ONE SIZE DOES NOT FEEL ALL.
That being said, something I had planned earlier, was to allow DAG Authors
to pick a single backend to choose the variable or connections from ( **Not
getting configurations from Secrets Backend though** ).
Example: `Variable.get(key="example_key",
secrets_backend="airflow.secrets.environment_variables.EnvironmentVariablesBackend")`
will only check Environment Variables to get Airflow Variables.
Only changes required:
```diff
diff --git a/airflow/models/variable.py b/airflow/models/variable.py
index 7d4726966..05b266bd3 100644
--- a/airflow/models/variable.py
+++ b/airflow/models/variable.py
@@ -124,6 +124,7 @@ class Variable(Base, LoggingMixin):
key: str,
default_var: Any = __NO_DEFAULT_SENTINEL,
deserialize_json: bool = False,
+ secrets_backend: Optional[str] = None,
) -> Any:
"""
Gets a value for an Airflow Variable Key
@@ -132,7 +133,7 @@ class Variable(Base, LoggingMixin):
:param default_var: Default value of the Variable if the Variable
doesn't exists
:param deserialize_json: Deserialize the value to a Python dict
"""
- var_val = Variable.get_variable_from_secrets(key=key)
+ var_val = Variable.get_variable_from_secrets(key, secrets_backend)
if var_val is None:
if default_var is not cls.__NO_DEFAULT_SENTINEL:
return default_var
@@ -193,14 +194,35 @@ class Variable(Base, LoggingMixin):
self._val = fernet.rotate(self._val.encode('utf-8')).decode()
@staticmethod
- def get_variable_from_secrets(key: str) -> Optional[str]:
+ def get_variable_from_secrets(
+ key: str,
+ secrets_backend: Optional[str] = None,
+ ) -> Optional[str]:
"""
Get Airflow Variable by iterating over all Secret Backends.
:param key: Variable Key
:return: Variable Value
"""
- for secrets_backend in ensure_secrets_loaded():
+ secrets_backends = ensure_secrets_loaded()
+ secrets_backends_classes = {
+ f"{backend.__class__.__module__}.{backend.__class__.__name__}":
backend
+ for backend in secrets_backends
+ }
+
+ if secrets_backend not in secrets_backends_classes:
+ raise KeyError(
+ f"Invalid secrets backend - '{secrets_backend}'. "
+ f"Should be one of {',
'.join(secrets_backends_classes.keys())}"
+ )
+
+ if secrets_backend:
+ var_val =
secrets_backends_classes[secrets_backends].get_variable(key=key)
+ if var_val is not None:
+ return var_val
+ return None
+
```
cc @fhoda
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
