shreyashdwivedi36 opened a new issue, #45849: URL: https://github.com/apache/airflow/issues/45849
### Apache Airflow version 2.10.4 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? The scheduler stops processing tasks due to a deadlock in the resource allocation mechanism. Logs show repeated warnings about locking mechanisms and no progress in DAG processing. ### What you think should happen instead? The scheduler should efficiently handle high concurrency workloads without encountering deadlocks. Tasks should progress as expected, and the system should avoid resource contention issues. ### How to reproduce 1. Deploy Apache Airflow 2.7.0 with a high-concurrency configuration. 2. Submit a complex DAG with multiple tasks requiring concurrent execution. 3. Observe the scheduler logs for deadlock warnings and stalled task processing. ### Operating System Ubuntu 20.04 ### Versions of Apache Airflow Providers Run the following command in your environment where Apache Airflow is installed: `pip freeze | grep apache-airflow-providers ` ### Deployment Docker-Compose ### Deployment details 1. Airflow Deployment Type: Deployed using Docker Compose with the official Airflow Docker image. 2. Environment Configuration: Includes PostgreSQL as the metadata database and Redis as the Celery backend. 3. Customization: Custom configuration for logging and email alerts added to airflow.cfg. 4. Version of Deployment Tools: - Docker: 20.10.14 - Docker Compose: 2.12.2 ### Anything else? Frequency of the Problem: The issue occurs consistently under high concurrency scenarios. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
