viragtripathi opened a new pull request, #69556:
URL: https://github.com/apache/airflow/pull/69556
Under SERIALIZABLE isolation PostgreSQL can abort a transaction with
SQLSTATE 40001 (serialization_failure), and deadlocks surface as 40P01. In the
scheduler's critical section either can be raised while queuing task instances
or at COMMIT time. Today that exception escapes _do_scheduling and crashes the
scheduler job even though the transaction
is safe to retry, while the sibling case of a busy advisory lock (55P03) is
already handled gracefully.
This handles serialization failures the same way: roll back, increment a new
scheduler.critical_section_conflict metric, and let the next scheduler loop
retry. Two details worth reviewer attention:
- The critical section buffers workloads into the executors' in-memory
queued_tasks dict
before the commit, so the handler discards exactly the workloads buffered
in the failed
round. Dispatching them after rollback would trip the execution API's
requirement that a
task instance be in QUEUED state before it can start.
- guard.commit() moves inside the handled scope, because serialization
failures frequently
surface at commit time rather than at statement time.
Behavior change for PostgreSQL deployments running SERIALIZABLE: a
serialization failure or deadlock in the critical section no longer crashes the
scheduler; it is counted and retried on the next loop, mirroring the existing
55P03 handling. MySQL deadlock codes (errno 1213) are out of scope here and
could be a follow-up if there is interest.
Verified in an HA deployment (two schedulers, 240 concurrent task instances,
on a PostgreSQL-compatible database running SERIALIZABLE): serialization
conflicts were observed and survived, and both schedulers completed with zero
restarts. Unit tests cover the new helper for psycopg2 and psycopg3 error
shapes, the 40001 and 40P01 handler paths,
a 55P03 regression case, and the buffered-workload purge.
Discussed on the devlist:
https://lists.apache.org/thread/t6jo4th3sn23jmr34m6gcxzw4k8mo4pc
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes - Claude Code
Generated-by: Claude Code following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
