avolant commented on PR #70307: URL: https://github.com/apache/airflow/pull/70307#issuecomment-5070914275
Here is my _humain_'s formulation of the problem: In our production environment, we had a problem where some tasks stayed in `SCHEDULED` state for longer than expected. we tracked down the root cause of the problem to [this line](https://github.com/apache/airflow/blob/d81660a8d705a77b71d13000ffc77680e438c6e3/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L974); ```py is_done = executable_tis or len(task_instances_to_examine) < max_tis ``` To our understanding, `is_done` condition was met (meaning, that `is_done` was set to `True`), but only one `executable_tis` was transitionned from `SCHEDULED` to `QUEUED`: We had 32 as `max_tis_per_query`, but 31 of the _studied_ tasks couldn't run (meaning, couldn't transition from `SCHEDULED` to `QUEUED`), because they had reached the internal DAG `max_tis_per_task`. The trouble is that each scheduler iterations were taking the same 31 tasks, and scheduling only task per loop which was highly inefficient, because the above `is_done` condition was always met. This PR attempt to solve this by making sure the scheduler loop continues to check the next batch of _schedulable_ tasks: ```py if len(executable_tis) >= max_tis or not found_new_filters: ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
