url54 commented on code in PR #34898: URL: https://github.com/apache/airflow/pull/34898#discussion_r1357438953
########## docs/apache-airflow/core-concepts/dags.rst: ########## @@ -514,6 +514,12 @@ For example, here is a DAG that uses a ``for`` loop to define some tasks: In general, we advise you to try and keep the *topology* (the layout) of your DAG tasks relatively stable; dynamic DAGs are usually better used for dynamically loading configuration options or changing operator options. +.. warning:: + + When utilizing Dynamic DAGs, please ensure you monitor key metrics like DagBagSize, to ensure your DAG script is not producing unexpected results. Performing iterations or looping, may cause unexpected results if your DAG + code does not properly handle all cases. Dynamic DAG creation may also adversely affect your environments overall health and performance. + + Review Comment: > I am not so comfortable with this phrasing and the place of this warning. > > 1. It raises more questions than it answers. If want to be explicit about what you warn you must also provide the full details and the answers. We should avoid from being vague. For example: what does it mean to monitor the metric? what are good value and what are bad? Obviously the answer is - it depends but we should provide better context if we are to open this subject > 2. the core-concepts is the very start of the docs. Airflow docs are written as a book. Read from start to end and you will learn everything you need to know. The core-concept is more of introduction and this note is for more advanced users. > > We have a whole section for dynamic dags: https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html and specifically there is a section about optimization. Maybe we should elaborate on this topic there and have it as a paragraph with full details rather than a small warning? > > WDYT? I think your points are correct. The item could find a better home in the more advanced sections like you stated. I can definitely scrap this and work on providing a better chunk of information with details on how to monitor those metrics and what to look for. I truly appreciate your comments! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
