potiuk commented on issue #6801: [AIRFLOW-6260] Drive _cmd config options by env var URL: https://github.com/apache/airflow/pull/6801#issuecomment-566581583 @NBardelo: I agree in in the container world that is how you keep secrets. And I understand your need. But we seem to try to add a new generic feature that we might have to support in all future versions of Airflow. If people start using it so we should be careful because that feature can be used in totally unexpected ways I can imagine (for example) someone implementing a round-robin script that assigns different `sql_alchemy_conn` to different instances in order to distribute traffic across several databases (been there, done that). While this is pretty powerful, it's also super dangerous for maintenance and support of a product. I am pretty sceptical when in a product we say "and here we can run any script with arbitrary complexity to achieve whatever user wants". That does not seem right because people will abuse it and product will have to support it indefinitely. I don't like the "over-generic" approach here - seems we are killing a fly with a cannon gun (as we say in Polish). But again - this is more to start discussion than to completely abandon that approach. Maybe I am wrong and others have other opinion. I would love to hear what other think (@ashb , @kaxil, @feluelle, @Fokko, @dimberman ?). Maybe that qualifies for devlist discussion and should be moved there? I totally agree that we should never keep many secrets in the same config file together with other settings, but my question is - why do you want to keep secrets in the .cfg file (or in your case in ENV variable) at all? Are there settings that you want to be kept secret that you cannot set otherwise? Which ones? Currently the default way for airflow to keep the connection secrets is to store them in connection database (which for the security reason can be encrypted with symmetric Fernet Key and the secrets are not visible via the UI) - not to keep them in .cfg file nor in environment variables (the environment variables are just a convenience to allow mostly testing and debugging). This is also security-by-obscurity in a way (but protects from revealing the password in case someone gets just access to the DB). Maybe we should have similar approach for anything that is secret and comes from the .cfg file currently instead? But maybe we could use some of the "vault" apis instead- whether via secrets in Kubernetes, or KMS, ApacheVault etc - we had discussion about it in the devlist I recall. Maybe that's a better approach?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
