potiuk commented on a change in pull request #18356:
URL: https://github.com/apache/airflow/pull/18356#discussion_r712915300
##########
File path: docs/apache-airflow/best-practices.rst
##########
@@ -241,11 +243,51 @@ each parameter by following the links):
* :ref:`config:scheduler__parsing_processes`
* :ref:`config:scheduler__file_parsing_sort_mode`
+.. _best_practices/reducing_dag_complexity:
+
+Reducing DAG complexity
+^^^^^^^^^^^^^^^^^^^^^^^
+
+While Airflow is good in handling a lot of DAGs with a lot of task and
dependencies between them, when you
+have many complex DAGs, their complexity might impact performance of
scheduling. One of the ways to keep
+your Airflow instance performant and well utilized, you should strive to
simplify and optimize your DAGs
+whenever possible - you have to remember that DAG parsing process and creation
is just executing
+Python code and it's up to you to make it as performant as possible. There are
no magic recipes for making
+your DAG "less complex" - since this is a Python code, it's the DAG writer who
controls the complexity of
+their code.
+
+There are no "metrics" for DAG complexity, especially, there are no metrics
that can tell you
+whether your DAG is "simple enough". However - as with any Python code you can
definitely tell that
+your code is "simpler" or "faster" when you optimize it, the same can be said
about DAG code. If you
+want to optimize your DAGs there are the following actions you can take:
+
+* Make your DAG load faster. This is a single improvement advice that might be
implemented in various ways
+ but this is the one that has biggest impact on scheduler's performance.
Whenever you have a chance to make
+ your DAG load faster - go for it, if your goal is to improve performance.
See below
+ :ref:`best_practices/dag_loader_test` on how to asses your DAG loading time.
Review comment:
Excellent one. That's often overlooked by people but can have tremendous
impact. Will add..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
