GitHub user DTelpuhovsky added a comment to the discussion: Dagbag import is taking longer time than an older version, which makes some tasks get timeout exceed errors
Sorry for jumping in. @potiuk I have the similar issue with dynamic dag generation for 1k+ dags from json config files. The performance problems lifecycle looks like this: - increase amount of json config files --> fixed by splitting to several "dag generators" - parsing problems for scheduling --> fixed by adjustments of scheduler params and adding dag processors - parsing problems on execution with version 2.4.3 --> fixed by experimental feature suggested here https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html - performance problems on execution appeared again after an upgrade to 2.7.3 and persist till 2.10.3 Do you have any idea why does this code stop returning dag_id approx from Airflow version 2.4.3? ``` from airflow.utils.dag_parsing_context import get_parsing_context current_dag_id = get_parsing_context().dag_id for thing in list_of_things: dag_id = f"generated_dag_{thing}" if current_dag_id is not None and current_dag_id != dag_id: continue # skip generation of non-selected DAG ``` GitHub link: https://github.com/apache/airflow/discussions/44402#discussioncomment-11917207 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
