zyong-moveworks opened a new issue #18131: URL: https://github.com/apache/airflow/issues/18131
### Apache Airflow version 2.1.3 (latest released) ### Operating System ubuntu ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==2.1.0 apache-airflow-providers-apache-hive==2.0.1 apache-airflow-providers-celery==2.0.0 apache-airflow-providers-ftp==2.0.0 apache-airflow-providers-http==2.0.0 apache-airflow-providers-imap==2.0.0 apache-airflow-providers-postgres==2.0.0 apache-airflow-providers-slack==4.0.0 apache-airflow-providers-snowflake==2.1.0 apache-airflow-providers-sqlite==2.0.0 apache-airflow-providers-ssh==2.1.0 ### Deployment Docker-Compose ### Deployment details using celery executor ### What happened Current scheduler_job.py tries to fetch at most max_tis number of task instances. Then there's the logic of checking concurrency limit. Consider this situation: If we have 2 DAGs dag_a and dag_b that each has 200 tasks and dag_a has concurrency limit of 1, and let's assume max_tis = 64. there's a chance where scheduler will fetch 64 tasks from dag_a, then later on all 64 task instances will get discarded because there's the concurrency limit. In this case scheduler is stuck and only 1 task instance will be running in airflow, instead of having 63 other tasks from dag_b running. This only happens for large DAG and low concurrency limit. I'm not sure what's the best way of solving this since now scheduler supports multi instance. Maybe lowering weight of the ti every time a ti does not meet requirements. Hope this all makes sense. Thanks! I'm willing to fix it but I don't believe I have enough context to not break airflow, especially not break multi instance scheduler. ### What you expected to happen In the above case. I expect 64 tasks running, instead of scheduler getting stuck with only 1 task running. ### How to reproduce As mentioned in the above section with those two dags. ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
