potiuk edited a comment on issue #14592: URL: https://github.com/apache/airflow/issues/14592#issuecomment-856216348
Agree we have consistency issue here - Interestingly, the AWS secret manager crashed originally for @subashcanapathy and @john-jac but did not crash for you @fhoda. Not sure what the reason is for that (maybe the 1.10 vs 2.* behavioral difference)? I really like the idea of different behavior for different type of access. I think it answers my concerns perfectly and what it really boils down to is "who" is the "client" - whether it is "airflow" or the "DAG/task writer". I think the main difference of configuration vs. variables and connections is that Airflow has default values for most of the configurations and when they are not found, they will fall-back to the default values - which might alter behavior of airflow. So lack of secrets backend when it is configured and configuration is retrieved is very dangerous. And since it is accessed under-the hood by Airflow, without the "dag" or "task" using it, it's airflow that is the "client" and it's airflow that should handle it (and crashing is the only reasonable behavior IMHO). Simply "dag writer" is not in a control to make any decision here. This is (as you rightfully noticed), far less of a concern for connections and variables - "clients" for those are "dag writers". Whoever uses them should be prepared for what happens when the secret backend is missing. Either the "writers" will prepare fallback values for those in the DB or they will have to handle "missing" value somehow (and this is up to the 'user' what to do in this case). But they are in full control, there is no need to crash Airflow (yet! - until configuration is not accessed by Airflow itself). Reopening it as it might actually be an actionable item to do :) @subashcanapathy , @john-jac - would that be a reasonable approach for you as well ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
