fhoda commented on issue #14592:
URL: https://github.com/apache/airflow/issues/14592#issuecomment-856125981
As of now it seems the expected behavior is not what is happening and is
inconsistent across different secret backends.
I have tried to reproduce this issue with Airflow 2.0 (main branch) and am
not able to do so for any AWS secrets backends. I was only able to reproduce a
crashing webserver for GCP Secret Manager and not any other secrets backend.
I used the `airflow.providers.*` secrets packages for each. I noticed that
the original post on the issue uses the `contrib` package and Airflow `1.10.12`.
```
#export
AIRFLOW__SECRETS__BACKEND=airflow.providers.hashicorp.secrets.vault.VaultBackend
#export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_path":
"airflow/connections", "variables_path": "airflow/variables", "config_path":
"airflow/config", "url": "http://127.0.0.1:8200";, "token": "$VAULT_TOKEN"}'
export
AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix":
"airflow-connections", "variables_prefix": "airflow-variables",
"gcp_keyfile_dict": $GCP_SECRET_MANAGER_SA_KEY}'
#export
AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.azure_key_vault.AzureKeyVaultBackend
#export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix":
"airflow-connections", "variables_prefix": null, "vault_url":
"https://example-akv-resource-name.vault.azure.net/"}'
#export
AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
#export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix":
"airflow/connections", "variables_prefix": "airflow/variables", "profile_name":
"default"}'
#export
AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
#export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix":
"/airflow/connections", "variables_prefix": "/airflow/variables",
"profile_name": "default"}'
```
Here are my findings:
- AWS Secret Manager - No Crash
- AWS SSM - No Crash
- Vault - No Crash
- Azure Key Vault - No Crash
- **GCP Secret Manager - Crash**
I believe we should evaluate what the expected behavior should be as
compared to what is actually happening.
Also after discussing with @kaxil there may be a middle ground for fail over
implementation that could make sense here.
1. If configs are being retrieved through the secrets backend then a failure
makes sense.
2. If connections and/or variables are not able to be retrieved, then fail
over could be a strategy used by users to ensure DAG/task success and
predictable execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
