SamWheating opened a new pull request #20816: URL: https://github.com/apache/airflow/pull/20816
While investigating another issue, I found what appears to be a bug introduced in https://github.com/apache/airflow/pull/20178 in the `_executable_task_instances_to_queued` function of the scheduler job. Here is the affected piece of code: https://github.com/apache/airflow/blob/c9023fad4287213e4d3d77f4c66799c762bff7ba/airflow/jobs/scheduler_job.py#L335-L351 So we're iterating through a list of `(pool_name: string, task_instances: List[TaskInstance])` pairs, but then checking the value of `task_instance.pool_slots`, which is not explicitly set anywhere. This means that we actually end up using the value set implicitly by this loop earlier in the function: https://github.com/apache/airflow/blob/c9023fad4287213e4d3d77f4c66799c762bff7ba/airflow/jobs/scheduler_job.py#L318-L319 And because of the `continue`, if this task happens to require more slots than the specified pool can provide, then all of the scheduled tasks in this pool will be skipped over. Anyways, I've refactored this check such that it checks every TI in `task_instances` individually and simply removes any non-executable TIs from the list of candidates tasks to be queued. I've updated the test accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
