akasakariko opened a new pull request, #76227: URL: https://github.com/apache/airflow/pull/76227
With multiple schedulers, `SKIP LOCKED` can bypass the earlier queued Dag runs of a backfill and start a later ordinal even when `max_active_runs=1` Restrict serial backfills to their lowest eligible queued `sort_ordinal` before applying candidate row locks, preserving forward and reverse ordering while retaining the existing Backfill lock and concurrency checks The regression test holds candidate locks in an independent session, checks that another scheduler cannot overtake them, and verifies ordered progress after releasing the locks closes: #76040 Validation - Before the fix, both forward and reverse regression cases fail on PostgreSQL 17 - After the fix, 138 relevant tests pass on PostgreSQL 17 and MySQL 8.4 - SQLite compatibility checks pass with 134 passed and 4 row-lock tests skipped - The combined scheduler, DagRun and Backfill modules have 1021 passed, 2 skipped, 1 xfailed and 4 failures in `test_documented_task_loop_examples_complete` - Repeating that combined run with the original production code reproduces the same 4 failures plus the 2 new regression failures - Running the complete DagRun module independently passes with 326 passed and 1 xfailed, including the 4 examples above - Regular and manual prek checks pass, including Ruff, mypy and bandit - Candidate selection remains one SQL query, checked with `assert_queries_count` - The broader database and non-database suites and documentation build listed by `breeze verify` are still in progress after building the ARM64 CI image locally, and are not counted as passing PostgreSQL query timing with Python 3.12 and 1000 queued runs per scenario, using medians of five alternating groups of 50 calls | Scenario | Before | After | Change | | --- | ---: | ---: | ---: | | Ordinary Dag runs | 3.26 ms | 3.45 ms | +6% | | Backfill with `max_active_runs=1` | 3.87 ms | 3.57 ms | -8% | | Backfill with `max_active_runs=3` | 3.73 ms | 3.96 ms | +6% | Control-query medians varied by less than 8% between paired runs The additional filtering costs about 0.2 ms per candidate-selection query in the ordinary and parallel-backfill scenarios --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-6) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
