aalemanq opened a new issue #8643:
URL: https://github.com/apache/airflow/issues/8643
Hello, I spend so many weeks trying to configure airflow using secrets in
docker swarm.
I try to use this config in my compose:
environment:
- LOAD_EX=y
- FERNET_KEY_CMD=$$(cat /run/secrets/fernet_key) (If I put '$" swarm
tells me that I need to escape with another $)
- EXECUTOR=Celery
- AIRFLOW__CELERY__BROKER_URL_CMD=$$(cat /run/secrets/broker_url)
- AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=$$(cat
/run/secrets/sql_alchemy_conn)
- AIRFLOW__CELERY__RESULT_BACKEND_CMD=$$(cat
/run/secrets/result_backend)
I tried with another bash commands to do cat to this secret and all fails,
it never gets enviroments and use redis default.
I tried to follow oficial doc but...
https://airflow.readthedocs.io/en/stable/howto/set-config.html
`The _cmd config options can also be set using a corresponding environment
variable the same way the usual config options can. For example:
export AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=bash_command_to_run`
Really is so difficult touse enviroments pointing to file like another
software?
enviroments:
- AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=$(cat /run/blabla) ?¿?
A little debug:
Thanks for reply. I tried to apply env_file but same issue. When I use
enviroments like:
```
LOAD_EX=y
FERNET_KEY=XXXX
EXECUTOR=Celery
AIRFLOW__CELERY__BROKER_URL=pyamqp://airflow:airflow@rabbitmq:5672/airflow
AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:airflow@postgres/airflow
```
It's works
but if I apply secrets using this in env_file, is not work :( :
```
AIRFLOW__CORE__FERNET_KEY_CMD=$(cat /run/secrets/fernet_key)
AIRFLOW__CELERY__BROKER_URL_CMD=$(cat /run/secrets/broker_url)
AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=$(cat /run/secrets/sql_alchemy_conn)
AIRFLOW__CELERY__RESULT_BACKEND_CMD=$(cat /run/secrets/result_backend)
```
Airflow never get broker_url and rabbitmq conection is replaced by redis...
I check if everything is right and I don't see any error about
secrets/chains:
Some debug:
I deploy stack and enter in worker airflow container to check if secrets
exists:
```
airflow@03eb98bd469d:~$ cat /run/secrets/sql_alchemy_conn
postgresql+psycopg2://airflow:airflow@postgres/airflow
airflow@03eb98bd469d:~$ cat /run/secrets/broker_url
pyamqp://airflow:airflow@rabbitmq:5672/airflow
airflow@03eb98bd469d:~$ cat /run/secrets/result_backend
db+postgresql://airflow:airflow@postgres/airflow
```
I execute by hand this enviroments:
```
airflow@03eb98bd469d:~$ LOAD_EX=y
airflow@03eb98bd469d:~$ AIRFLOW__CORE__FERNET_KEY_CMD=$(cat
/run/secrets/fernet_key)
airflow@03eb98bd469d:~$ EXECUTOR=Celery
airflow@03eb98bd469d:~$ AIRFLOW__CELERY__BROKER_URL_CMD=$(cat
/run/secrets/broker_url)
airflow@03eb98bd469d:~$ AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=$(cat
/run/secrets/sql_alchemy_conn)
airflow@03eb98bd469d:~$ AIRFLOW__CELERY__RESULT_BACKEND_CMD=$(cat
/run/secrets/result_backend)
```
And echo is right:
```
airflow@03eb98bd469d:~$ echo $AIRFLOW__CORE__FERNET_KEY_CMD
46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
airflow@03eb98bd469d:~$ echo $AIRFLOW__CELERY__BROKER_URL_CMD
pyamqp://airflow:airflow@rabbitmq:5672/airflow
airflow@03eb98bd469d:~$ echo $AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD
postgresql+psycopg2://airflow:airflow@postgres/airflow
airflow@03eb98bd469d:~$ echo $AIRFLOW__CELERY__RESULT_BACKEND_CMD
db+postgresql://airflow:airflow@postgres/airflow
```
I don't know what I can do, secrets testes, enviroments testesd, no aiflow
running with enviroments and secrets but without secrets it works as exepted...
Regards
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
