SakshamSinghal20 opened a new pull request, #61769: URL: https://github.com/apache/airflow/pull/61769
issue #61700 ### Description Deferred tasks were previously excluded from [max_active_tis_per_dag](cci:1://file:///c:/Users/Saksham%20Singhal/OneDrive/Desktop/Airflowww/airflow/airflow-core/tests/unit/models/test_taskinstance.py:313:4-325:37) and [max_active_tis_per_dagrun](cci:1://file:///c:/Users/Saksham%20Singhal/OneDrive/Desktop/Airflowww/airflow/airflow-core/tests/unit/models/test_taskinstance.py:327:4-339:37) concurrency checks. This allowed an unlimited number of tasks to enter the `DEFERRED` state, bypassing configured limits. This PR enforces these limits for `DEFERRED` task instances while ensuring that [max_active_tasks](cci:1://file:///c:/Users/Saksham%20Singhal/OneDrive/Desktop/Airflowww/airflow/airflow-core/tests/unit/jobs/test_scheduler_job.py:1444:4-1491:79) (which limits worker slot usage) remains unaffected. **Changes:** * **[airflow/ti_deps/dependencies_states.py](cci:7://file:///c:/Users/Saksham%20Singhal/OneDrive/Desktop/Airflowww/airflow/airflow-core/src/airflow/ti_deps/dependencies_states.py:0:0-0:0)**: Introduced `TASK_CONCURRENCY_EXECUTION_STATES` to include `DEFERRED`, `RUNNING`, and `QUEUED`. * **[airflow/jobs/scheduler_job_runner.py](cci:7://file:///c:/Users/Saksham%20Singhal/OneDrive/Desktop/Airflowww/airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py:0:0-0:0)**: Updated [ConcurrencyMap](cci:2://file:///c:/Users/Saksham%20Singhal/OneDrive/Desktop/Airflowww/airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py:167:0-207:76) to count deferred TIs for task-level limits, but explicitly exclude them from `dag_run_active_tasks_map`. * **[airflow/models/taskinstance.py](cci:7://file:///c:/Users/Saksham%20Singhal/OneDrive/Desktop/Airflowww/airflow/airflow-core/src/airflow/models/taskinstance.py:0:0-0:0)**: Updated [get_num_running_task_instances](cci:1://file:///c:/Users/Saksham%20Singhal/OneDrive/Desktop/Airflowww/airflow/airflow-core/src/airflow/models/taskinstance.py:1738:4-1755:68) to count deferred TIs for pre-execution dependency checks. **Tests:** Added unit tests covering: * Deferred tasks blocking new scheduling attempts. * Mixed state scenarios (Running + Deferred). * Regression checks ensuring [max_active_tasks](cci:1://file:///c:/Users/Saksham%20Singhal/OneDrive/Desktop/Airflowww/airflow/airflow-core/tests/unit/jobs/test_scheduler_job.py:1444:4-1491:79) is not impacted. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes (used for guidance and file locating) --- * closes: #61700 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
