wolfier commented on a change in pull request #15183:
URL: https://github.com/apache/airflow/pull/15183#discussion_r624532523
##########
File path: docs/apache-airflow/faq.rst
##########
@@ -159,72 +215,194 @@ simple dictionary.
other_dag_id = f'bar_{i}'
globals()[other_dag_id] = create_dag(other_dag_id)
-What are all the ``airflow tasks run`` commands in my process list?
--------------------------------------------------------------------
+Even though Airflow supports multiple DAG definition per python file,
dynamically generated or else, it is not
+recommended as Airflow would like better isolation between DAGs from a fault
and deployment perspective and multiple
+DAGs in the same file goes against that.
Review comment:
The line came verbatim from the old [Common
Pitfall](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62694614)
page.
> We do support more than one DAG definition per python file, but it is not
recommended as we would like better isolation between DAGs from a fault and
deployment perspective and multiple DAGs in the same file goes against that.
For now, make sure that the dag object is in the global namespace : you can use
the globals dict as in globals()[dag_id] = DAG(...)
I think the original author wanted to affirm that Airflow does allow a user
to define multiple DAGs in a file whether they are statically defining those
DAGs or in a loop of some sorts and using the globals symbol table to store the
definitions. However, having multiple DAG definition goes against Airflow's
mindset/ideal of isolation between DAGs from a fault and deployment perspective.
To sum it up, yes, you can create dynamic DAGs but Airflow does not
recommend it.
Unless we do recommend it (?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
