rjgoyln opened a new pull request, #75962: URL: https://github.com/apache/airflow/pull/75962
Continues #72595, closed by the open PR limit. ## Summary The deferred-task timeout sweep fails every timed-out task instance in one unbounded `UPDATE`. On MySQL that statement can wait on rows the triggerer is writing while the triggerer waits on rows the sweep holds, and InnoDB kills one side. The sweep's retry does not prevent this: under the same triggerer traffic the cycle can form again, and once the retries run out the exception ends the scheduler loop, which is the crash #56670 reports. closes: #56670 ## Change - Claim the timed-out rows with `FOR UPDATE SKIP LOCKED` and update only those. - Claim at most 500 task instances per tick. - Roll back a failed attempt before retrying, as `adopt_or_reset_orphaned_tasks` does. With `SKIP LOCKED` the sweep no longer waits on rows the triggerer holds, and the `UPDATE` touches only rows its own transaction already claimed. Where `with_row_locks` is a no-op (SQLite, MariaDB, `use_row_level_locking = False`), the rollback is what lets the retry start from a clean transaction. ## Tests MySQL 8, 450 deferred task instances, four threads re-deferring ten rows per transaction, three 30-second runs: | | InnoDB 1213 deadlocks | |---|---| | before | 5 / 7 / 6 | | after | 0 / 0 / 0 | On a failing run, `SHOW ENGINE INNODB STATUS` names the sweep's `UPDATE` and a triggerer's per-row update as the two sides. ## Behavior change A tick fails at most 500 timed-out task instances; a larger backlog drains over later ticks. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
