jscheffl commented on issue #49508: URL: https://github.com/apache/airflow/issues/49508#issuecomment-2836466232
We also had such issue (had a talk about this in https://airflowsummit.org/sessions/2024/how-we-tuned-our-airflow-to-make-1-2-million-dag-runs-per-day/) and I _think_ my colleguage @AutomationDev85 supplied a fix to Airflow 2.6 or 2.7... we our problem in this area were gone. But one important remark: We ensured that the amount of parallel DAG runs is limited. We triggered new DAG runs in state "queued" and not "running", then the scheduler saves efforts in scheduling "not ready" runs. And was only adding "running" state at a point when sufficient DAGs have been worked-off. Can you try this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
