rjgoyln opened a new pull request, #72595: URL: https://github.com/apache/airflow/pull/72595
## Summary The deferred-task timeout sweep issued one blind, unbounded `UPDATE` over every task instance past its trigger timeout. On MySQL it waits on rows the triggerer is writing while the triggerer waits on rows the sweep already holds, so InnoDB breaks the cycle by killing one of them. The retry the sweep already had does not save it. The deadlock recurs on every tick, and once the retries are exhausted the exception ends the scheduler loop — which is what #56670 reports on 2.10.5, a version that already carries that retry. closes: #56670 ## Change - Claim the timed-out rows with `SELECT ... FOR UPDATE SKIP LOCKED` before writing them, so the sweep never waits on a row another writer holds. - Bound the claim to 500 task instances per tick, PK-ordered; the rest are failed on later ticks. - Roll back a failed attempt before the retry runs again, as `adopt_or_reset_orphaned_tasks` does. A sweep that never waits cannot be one side of a lock cycle, so the deadlock has nowhere to form. The rollback still matters where `with_row_locks` degrades to a no-op — SQLite, MariaDB, `use_row_level_locking = False` — because a cycle remains possible there, and it is the remedy #56670 asked for. ## Tests A harness driving the real sweep against triggerer-shaped write traffic on MySQL 8 — 450 deferred task instances, four threads re-deferring ten rows per transaction, 30s per run: | | InnoDB 1213 deadlocks | |---|---| | before | 5 / 7 / 6 | | after | 0 / 0 / 0 | `SHOW ENGINE INNODB STATUS` on a failing run names the sweep's `UPDATE ... WHERE state = 'deferred' AND trigger_timeout < ...` as one side of the cycle — the statement from the traceback in #41428 — and a triggerer's per-row update as the other. `test_retry_on_db_error_when_update_timeout_triggers` now commits its fixture rows rather than only flushing them, since rows that were merely flushed do not survive the rollback. ## Behavior change One tick now fails at most 500 timed-out task instances instead of all of them; a larger backlog drains over subsequent ticks, 15s apart by default. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
