potiuk commented on issue #14592:
URL: https://github.com/apache/airflow/issues/14592#issuecomment-852516068
If I understand correctly - this is only when you cannot reach secret
backend when it is configured ? Not about just a value missing in the backend?
If my assumption is correct, then I have no doubts whatsoever that his is
desired, nice and "good" behavior for airflow to crash hard in this case.
It's far worse to continue running, when you rely on the backend with
variables to be available and they cannot be read where you expec them (and the
assumption is that when you configure the backend, you want it to be
reachable).
This might lead to a number of problems - bad processing of data, wrong
databases updated, wrong calculations, you name it. Totally unpredictable,
because we have no idea what the retrieved value will be used for, and we have
no idea whether a "fallback default' is good or not.
It's NEVER a good idea to continue when you have unpredictable and
unexpected configuration. Airflow as a system is not able to assess if the
"default" value is "OK" to continue if the "expected" backend does not provide
the right values (either because it is missing or wrongly configured). Actually
it's even worse - Airflow is not even able to determine which kind of problem
it is - is it wrongly configured? or just temporarily missing-in-action ? It
will look exactly the same from Airlfow's point of view and this means that by
not-crashing we accept that sometimes Airflow will run with configuration A and
sometimes (when the secret backend is not reachable) with configuration B. This
is no-go.
You either expect the secret backend to be available or not - there is no
middle ground.
Crashing in this case is the only reasonable approach.
And I really like the output @lidalei copied. It is very straightworward
and easy to find what the problem is. Again - we have no idea whether we can
continue with default value or not. The only sensible approach is to crash hard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
