eladkal commented on PR #30259:
URL: https://github.com/apache/airflow/pull/30259#issuecomment-1520875547
> Before a task can execute, does it not both have to parse the DAG and
retrieve the connection details? And if a secrets backend is specified, don't
those requests lead to Secrets Manager calls each time? If so, then if the
Airflow worker could also cache connections and variables then it could lead to
a dramatic reduction in cost.
Maybe I'm missing something here but you are talking on a whole another
issue.
You are not referring to dag parsing at all. You are refering to a user who
excel in writing DAGs, follow all best practicies. This user authored a DAG
with 50 tasks that use the same connection. This connection is retrieved when
the task are running (Not in parsing time that happens every 30 seconds!) and
for that case you are suggesting to bring the connection only once rather than
50 times. I am perfectly fine with this! We should do this if we can!
But this is not the issue we discuss in this PR.
We discuss a DAG that looks like:
```
dag = DAG(...)
Variable.get("my_var")
MyOperator(...)
```
This will invoke retrieve of the variable every 30 seconds (with all the
related costs) and this is what @vandonr-amz is looking to optimize
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
