boqiang opened a new pull request, #74761: URL: https://github.com/apache/airflow/pull/74761
Database errors while loading a serialized Dag or creating a scheduled DagRun are caught by per-Dag handlers. Continuing with the failed SQLAlchemy Session can turn the original retryable error into `PendingRollbackError` and interrupt scheduling. Propagate `DBAPIError` from both handlers to the existing `_create_dagruns_for_dags` transaction retry boundary. That boundary rolls back before querying candidates again and reacquiring their row locks. Keep ordinary per-Dag error handling and `CommitProhibitorGuard` unchanged. Persistent database errors still propagate after the configured retry limit; this change does not provide per-Dag isolation for persistent failures. Add regression coverage for transient errors and retry exhaustion at both interception points. Add a MySQL-only test using two real InnoDB transactions to cause error 1213 during pending Dag metadata flush (with autoflush both enabled and disabled), after one DagRun has already been created in the batch. Check that the batch re-queries locked candidates, produces exactly one run per scheduled Dag, and preserves the concurrent transaction's metadata updates. Related: #59120 Validation on Python 3.11 through Breeze (Linux amd64 CI image on Apple Silicon): - MySQL 8.4: all 6 new regression cases passed, including native InnoDB error 1213 with autoflush enabled and disabled. - SQLite and PostgreSQL 14: 4 passed and 2 MySQL-only cases skipped on each backend. - Reverting only the production change on the same upstream base: all 6 MySQL cases failed with the expected `PendingRollbackError` following the original constraint error or deadlock. - Full selective DB checks: API 4291 passed/16 skipped/4 baseline failures; Other 967 passed/8 skipped/1 baseline failure. Non-DB checks: 5343 passed/11 skipped/2 xfailed/2 baseline failures. - Full MySQL scheduler test file: 566 passed, 2 skipped. PostgreSQL Core group: 3560 passed, 32 skipped, 1 xfailed. PostgreSQL CLI group: 673 passed. PostgreSQL Serialization: 62 passed. PostgreSQL Always: 1918 passed, 6 skipped. - Ruff, airflow-core mypy, and manual static checks passed. - Broader selective checks found five plugin-count failures that also reproduce with both PR files restored to the upstream base. The two non-DB LangSDK file-descriptor failures also reproduce with both PR files restored to that base (extra descriptors 3–6, including `/usr/bin/dash`). Documentation build (`breeze build-docs apache-airflow`) passed. These seven baseline failures keep the broader checks from being fully green; no unrelated changes or test skips are included. These tests exercise the transaction boundary directly; they do not establish multi-scheduler process acceptance. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
