hussein-awala commented on code in PR #34898: URL: https://github.com/apache/airflow/pull/34898#discussion_r1357371674
########## docs/apache-airflow/core-concepts/dags.rst: ########## @@ -514,6 +514,12 @@ For example, here is a DAG that uses a ``for`` loop to define some tasks: In general, we advise you to try and keep the *topology* (the layout) of your DAG tasks relatively stable; dynamic DAGs are usually better used for dynamically loading configuration options or changing operator options. +.. warning:: + + When utilizing Dynamic DAGs, please ensure you monitor key metrics like DagBagSize, to ensure your DAG script is not producing unexpected results. Performing iterations or looping, may cause unexpected results if your DAG + code does not properly handle all cases. Dynamic DAG creation may also adversely affect your environments overall health and performance. + + Review Comment: I agree with Elad on the location of the document, but I'm also not sure about the warning itself. The Airflow DAG file is a normal python script, so if the user expects a certain number of dags to be created, they can add a check in the script itself (input validation, size limit from the list, even sending a notification or failing the dags creation). If we have multiple dynamic DAGs files, monitoring the DagBagSize metric could generate a lot of false alerts (the sum of multiple small changes could be big). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
