This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 224c3c2a8f968a34f17fba9d3c9d7f9b7871c17b
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Aug 22 00:28:16 2025 +0100
[v3-0-test] `get_parsing_context` is no longer experimental (#54793)
(#54802)
We've had this for a while now, time to make it official. I've also
reworded the docs section a bit as well.
(cherry picked from commit fdef01c7d25d28560100484d449e1ca1393c53f0)
Co-authored-by: Jed Cunningham
<[email protected]>
---
airflow-core/docs/howto/dynamic-dag-generation.rst | 26 +++++++++-------------
1 file changed, 10 insertions(+), 16 deletions(-)
diff --git a/airflow-core/docs/howto/dynamic-dag-generation.rst
b/airflow-core/docs/howto/dynamic-dag-generation.rst
index 734e89f5d80..dcd07df8139 100644
--- a/airflow-core/docs/howto/dynamic-dag-generation.rst
+++ b/airflow-core/docs/howto/dynamic-dag-generation.rst
@@ -165,42 +165,36 @@ Optimizing DAG parsing delays during execution
.. versionadded:: 2.4
-|experimental|
-
Sometimes when you generate a lot of Dynamic dags from a single DAG file, it
might cause unnecessary delays
when the DAG file is parsed during task execution. The impact is a delay
before a task starts.
Why is this happening? You might not be aware but just before your task is
executed,
Airflow parses the Python file the DAG comes from.
-The Airflow Scheduler (or rather DAG File Processor) requires loading of a
complete DAG file to process
+The Airflow DAG File Processor requires loading of a complete DAG file to
process
all metadata. However, task execution requires only a single DAG object to
execute a task. Knowing this,
we can skip the generation of unnecessary DAG objects when a task is executed,
shortening the parsing time.
This optimization is most effective when the number of generated dags is high.
-There is an experimental approach that you can take to optimize this
behaviour. Note that it is not always
-possible to use (for example when generation of subsequent dags depends on the
previous dags) or when
-there are some side-effects of your dags generation. Also the code snippet
below is pretty complex and while
-we tested it and it works in most circumstances, there might be cases where
detection of the currently
-parsed DAG will fail and it will revert to creating all the dags or fail. Use
this solution with care and
-test it thoroughly.
+Note that it is not always possible to use (for example when generation of
subsequent dags depends on the previous dags) or when
+there are some side-effects of your dags generation. Use this solution with
care and test it thoroughly.
A nice example of performance improvements you can gain is shown in the
`Airflow's Magic Loop
<https://medium.com/apache-airflow/airflows-magic-loop-ec424b05b629>`_ blog post
that describes how parsing during task execution was reduced from 120 seconds
to 200 ms. (The example was
-written before Airflow 2.4 so it uses undocumented behaviour of Airflow.)
+written before Airflow 2.4 so it uses undocumented behaviour of Airflow, not
:py:meth:`~airflow.utils.dag_parsing_context.get_parsing_context` shown below.)
-In Airflow 2.4 instead you can use
:py:meth:`~airflow.utils.dag_parsing_context.get_parsing_context` method
-to retrieve the current context in documented and predictable way.
+In Airflow 2.4+ instead you can use
:py:meth:`~airflow.utils.dag_parsing_context.get_parsing_context` method
+to retrieve the current context in a documented and predictable way.
Upon iterating over the collection of things to generate dags for, you can use
the context to determine
whether you need to generate all DAG objects (when parsing in the DAG File
processor), or to generate only
a single DAG object (when executing the task).
-The :py:meth:`~airflow.utils.dag_parsing_context.get_parsing_context` return
the current parsing
-context. The context is of
:py:class:`~airflow.utils.dag_parsing_context.AirflowParsingContext` and
-in case only single DAG/task is needed, it contains ``dag_id`` and ``task_id``
fields set.
-In case "full" parsing is needed (for example in DAG File Processor),
``dag_id`` and ``task_id``
+:py:meth:`~airflow.utils.dag_parsing_context.get_parsing_context` returns the
current parsing
+context. The context is an
:py:class:`~airflow.utils.dag_parsing_context.AirflowParsingContext` and
+when only a single DAG/task is needed, it contains ``dag_id`` and ``task_id``
fields set.
+When "full" parsing is needed (for example in DAG File Processor), ``dag_id``
and ``task_id``
of the context are set to ``None``.
