c-thiel commented on issue #35675: URL: https://github.com/apache/airflow/issues/35675#issuecomment-1842586190
We had the same problem and only noticed it from Airflow 2.7.3, but it was there before. Maybe this helps for debugging: On the left of the red line we had Airflow 2.6.X On the right side is the update to 2.7.3. Each line is one scheduler. <img width="1522" alt="image" src="https://github.com/apache/airflow/assets/12560027/c6cd22e4-cbfd-445f-b0dd-5f19bbae0982";> Before the update, scheduler open slots continuously went down to even below zero sometimes, but only for some executors. After the update this is much more distributed between all the executors. **Our Workaround** What helped us is limit the parallelism via pools, and just set `AIRFLOW__CORE__PARALLELISM` to an arbitrary high number. Also regular deployment restarts helps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
