avolant opened a new pull request, #70307: URL: https://github.com/apache/airflow/pull/70307
## Why The scheduler candidate query applies `LIMIT max_tis`, then performs several concurrency and capacity checks in memory. If a page contains both a blocked high-priority task and a runnable task, the previous loop stopped as soon as it selected any executable task, leaving the batch partially filled. Repeating that pattern can starve lower-priority runnable tasks. For example, with `max_tis=2` and candidates ordered as: ```text A: blocked by task concurrency B: runnable C: runnable ``` The first query returns `[A, B]`. The scheduler rejects `A`, accepts `B`, and previously stopped with only one of two available positions filled. `C` is never examined during that scheduling pass. ## What changed The scheduler now continues querying while the batch remains under `max_tis` and the previous iteration discovered a new starvation filter. After the first query above: 1. The task-concurrency filter learned from `A` excludes it from the next query. 2. The selected TI ID for `B` is explicitly excluded because selected TIs remain `SCHEDULED` in the database until the loop finishes. 3. The next query uses `LIMIT 1`, which is the remaining batch capacity, and selects `C`. 4. The completed batch is `[B, C]`. This intentionally does not use SQL `OFFSET`. Every iteration adds newly learned starvation filters, which changes and shrinks the ordered result set. Applying an offset to that changed result set could skip valid candidates. Re-querying with the learned filters and selected-TI exclusion provides stable iteration over this dynamic candidate set. Executor slot availability is also initialized outside the query loop. Accepted tasks can now span multiple query iterations, so executor capacity must be consumed across the entire loop. Reinitializing it per page would allow later pages to reuse capacity already consumed by earlier pages. ## Validation - Added a database-backed regression test without monkeypatching that fails before the fix and fills the batch after the fix. - Full SQLite scheduler module: 389 passed, 3 expected skips. - PostgreSQL regression and executor-capacity coverage: passed. - Ruff, mypy, pre-commit, and manual hooks: passed. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes -- Codex (GPT-5) Generated-by: Codex (GPT-5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-5) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
