jens-scheffler-bosch commented on issue #33835:
URL: https://github.com/apache/airflow/issues/33835#issuecomment-1702388281
I see also DAG import errors because of timeouts, side effects might be also
related to bad performance or top level code. I doubt that Variables are "not
available" if all is properly set-up and performance is good.
As written in your logs attached, can you try making a round of optimization
of DAGs and ensure they are properly loaded? The one example I saw in the logs
is realted to:
`["-d","{{ ds }}", "-r",
Variable.get("cbc_adobe_analytics_ingester_report_v2"), "-mt",
"{{ti.max_tries}}", "-tn", "{{ti.try_number}}"]`
Can you convert the variable also to a Jinja expression like:
`["-d","{{ ds }}", "-r", "{{ variable.cbc_adobe_analytics_ingester_report_v2
}}", "-mt", "{{ti.max_tries}}", "-tn", "{{ti.try_number}}"]`
...then it is not accessed during parse-time but at task scheduling time.
Might be that poor DB performance and accessing many variables creates a
problem loading them in your environment.
```
Please take a look at these docs to improve your DAG import time:
*
https://airflow.apache.org/docs/apache-airflow/2.6.3/best-practices.html#top-level-python-code
*
https://airflow.apache.org/docs/apache-airflow/2.6.3/best-practices.html#reducing-dag-complexity
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
