GitHub user andrii-korotkov-verkada created a discussion: What's the best way to integrate Airflow with encrypted secrets stored in manifests?
Hello. I hope your day is going well. I'm trying to use ArgoCD to manage Airflow, which makes it tougher to work with secrets, as I can't just check them into the repo due to values being only base64 encoded. I've seen there's AWS Secrets Manager backend, which is a fine option, but has a downside of some manual overhead when updating values. For example, `pgbouncer.ini` is a whole config which has some sensitive data (e.g. password in db connection). A solid alternative would be to store the secrets in a source manifests code in encrypted form (e.g. with AWS KMS). This would allow to automate a process bringing up things in a new account/region for example. However, Airflow doesn't seem to support this out of the box. I can set env variables in `airflowLocalSettings` (which maps to a Python file run on init), e.g. by creating _ENCRYPTED versions of env variables, decrypting them on startup and setting regular versions of variables. However, it won't work out of the box for things like `pgbouncer.ini`, which is mounted as a file in the deployment manifest. What's the best way forward? Shall I do some code changes to Airflow to natively support encrypted data with some new backend? Or is there some way to configure all secrets without doing this? Thanks. GitHub link: https://github.com/apache/airflow/discussions/45190 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
