joaopamaral opened a new issue, #51212: URL: https://github.com/apache/airflow/issues/51212
### Apache Airflow version 2.11.0 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? We observed a significant increase in scheduler loop duration after attempting to upgrade from Airflow 2.10.4 to 2.10.5. Due to this issue, we reverted the upgrade. However, after attempting to migrate from 2.10.4 to 2.11.0, the problem persists. - v2.10.4  - v2.11.0 After upgrading to version 2.11.0 we noticed that the scheduler loop increased to ~18s (2x the old values) running the same dag.  ### What you think should happen instead? I’ve reviewed the code changes between versions but haven’t been able to identify any specific modification that could explain this increase in scheduler loop duration. Nothing in the diffs stands out as an obvious cause for the performance degradation. ### How to reproduce Using Airflow v2.10.4 • Create a DAG with a large number of tasks (>5,000). • Trigger the DAG and monitor the scheduler_scheduler_loop_duration metric. Using Airflow v2.11.0 • Repeat the same setup: using a DAG with >5,000 tasks. • Trigger the DAG and monitor the same scheduler_scheduler_loop_duration metric. Expected vs. Observed Behavior • Expected: The scheduler loop duration should remain consistent or show minimal variation between versions. • Observed: In v2.10.4, the loop duration typically remains around 9 seconds. In v2.11.0, we observed a significant increase in this duration (2x). ### Operating System Debian GNU/Linux 12 (bookworm) ### Versions of Apache Airflow Providers - 2.10.4  - 2.11.0  ### Deployment Other Docker-based deployment ### Deployment details Docker (v28.0.4) deployment ### Anything else? Observed after upgrade attempt. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
