jedcunningham commented on code in PR #25121: URL: https://github.com/apache/airflow/pull/25121#discussion_r924594900
########## docs/apache-airflow/howto/dynamic-dag-generation.rst: ########## @@ -140,3 +140,78 @@ Each of them can run separately with related configuration .. warning:: Using this practice, pay attention to "late binding" behaviour in Python loops. See `that GitHub discussion <https://github.com/apache/airflow/discussions/21278#discussioncomment-2103559>`_ for more details + + +Optimizing DAG parsing delays during execution +---------------------------------------------- + +Sometimes when you generate a lot of Dynamic DAGs from a single DAG file, it might cause unnecessary delays +when the DAG file is parsed during task execution. The impact is a delay before a task starts. + +Why is this happening? You might not be aware but when your task is executed just before execution, Review Comment: ```suggestion Why is this happening? You might not be aware but just before your task is executed, ``` ########## docs/apache-airflow/howto/dynamic-dag-generation.rst: ########## @@ -140,3 +140,78 @@ Each of them can run separately with related configuration .. warning:: Using this practice, pay attention to "late binding" behaviour in Python loops. See `that GitHub discussion <https://github.com/apache/airflow/discussions/21278#discussioncomment-2103559>`_ for more details + + +Optimizing DAG parsing delays during execution +---------------------------------------------- + +Sometimes when you generate a lot of Dynamic DAGs from a single DAG file, it might cause unnecessary delays +when the DAG file is parsed during task execution. The impact is a delay before a task starts. + +Why is this happening? You might not be aware but when your task is executed just before execution, +Airflow parses the Python file the DAG comes from. + +The Airflow Scheduler (or DAG Processor) requires loading of a complete DAG file to process all metadata. +However, task execution requires only a single DAG object to execute a task. Knowing this, we can +skip the generation of unnecessary DAG objects when task is executed, shortening the parsing time. Review Comment: ```suggestion skip the generation of unnecessary DAG objects when a task is executed, shortening the parsing time. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
