potiuk commented on issue #9713: URL: https://github.com/apache/airflow/issues/9713#issuecomment-675436524
My proposals: 1) I think we can implement tenacity's retrieval mechanism (https://pypi.org/project/tenacity/) with exponential back-off. We use it in a few places already and we could easily set a default sane behaviour and make sure in the interface we define a "Temporary" problem that needs retry. This needs to be done at the "backend" level (so that retrieving multiple variables from the backend are also treated this way). 2) I think also - I do not think having a "local" default" as fallback is a good idea in general. The way it is implemented now was implemented to make it easy to introduce by users of the previous variable mechanisms, but I think we should do it differently long term. I think we should have a way to define whether certain variable is "expected" to be retrieved from the secret backend ("secret=true") either at the DAG level (parameter) or possibly at the definition of the variable in the metastore (so reverse the check - first we check in metastore if a given variable has "expect_secet") flag set and then check tt in the secret.). I do not see any case where something that is expected to be secured in the Secret Backend has a reasonable local fallback value. @kaxil WDYT? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
