Hello Airflow team, Please advise if you can. In our environment, we have noticed that dynamic tasks place quite of stress on scheduler, webserver and increase MySQL DB connections. We are run about 1000 Dynamic Tasks every 30 min and parsing time increases from 5 to 65 sec with Runtime from 2sec to 350+ . This happens at execution time then it drops to normal while still executing tasks. Webserver hangs for few minutes.
Airflow 1.10.1. MySQL DB Example: Dynamic Tasks: Number of DAGs: 44 Total task number: 950 DagBag parsing time: 65.879642000000001 Static Tasks: Number of DAGs: 73 Total task number: 1351 DagBag parsing time: 1.731088 Is this something you aware of? Any advises on Dynamic tasks optimization/best practices? Thank you in advance, Eugene
