I second Elad's view here.
I would also propose an alternative fix: let's come up with a way to tell
Airflow to not continuously reparse a file!
A strawman example:
```
from airflow import ReparseMode, DAG
AIRFLOW_REPARSE = ReparseMode.ON_FILE_CHANGED
with DAG(...):
....
```
We could change things so that if that top level value is seen Airflow stops
parsing that file unless a change is detected.
We could also add a configuration option to set a default mode for all dag
files that don't have an explicit mode set.
Not only cos this put less load/cost on the secrets backend, it also puts less
load on the Dag parser itself! Win win.
-ash
On 23 March 2023 22:56:11 GMT, Elad Kalif <elad...@apache.org> wrote:
>To me Airflow is not "open and play". It is not a closed system that guides
>you how to develop. Workflow as code requires more skills.
>There are stuff to learn before authoring dags.
>This discussion is about Variables but I might ask similar question about
>users who query API as part of top level code. This is also very bad and
>not covered by current discussion.
>
>In my prespective users who hit this issue are likely also to hit many
>other issues. Its a good sign that they should try to learn Airflow better
>before continue development. I don't think a solution involves in hinding
>the issue is a good one.
>
>For now I'm -1 for making complecated changes to mitigate this. However I
>am infavor of finding a way to alert users that we detected a possible bad
>practice and we advise to recheck the code (that also requires some
>thought.. I don't know if bumping many warnings in the main UI is the right
>way...)
>
>בתאריך יום ו׳, 24 במרץ 2023, 00:25, מאת Vandon, Raphael
><vand...@amazon.com.invalid>:
>
>> Hello,
>> I’d like to submit to discussion the idea of having a cache on Airflow
>> Variables.
>> The aim is to reduce DAG parsing time and Secret Manager API bill for
>> users who make a liberal use of Variable.get in their DAG files.
>> The drawback, of course, is that caches introduce a delay in how fast
>> changes are propagated.
>>
>> For improved readability, I kindly invite you to read and comment in the
>> github discussion : https://github.com/apache/airflow/discussions/30265
>>
>> Cheers
>>
>>
