steveahnahn opened a new pull request, #69767: URL: https://github.com/apache/airflow/pull/69767
Fixes #68263. Supersedes the stale draft #68267 (closed by maintainer triage with an invitation to open a fresh PR); credit to its author for the initial direction. ### Problem - With multiple Dag processors (HA), concurrent parsing of the same dag races on the `dag_tag` insert; the loser gets an `IntegrityError`. - `run_with_db_retries` retries every `DBAPIError`, but the retry loop in `update_dag_parsing_results_in_db` only rolled back on `OperationalError`. - The retry then ran on the poisoned session and crashed with `PendingRollbackError` (not retryable), masking the original error and failing the whole parse batch. Matches the reporter's production traceback exactly. - One level down, `_serialize_dag_capturing_errors` re-raised only `OperationalError`, so a retryable DB error from per-dag serialization was misrecorded as a dag import error instead of reaching the retry loop (also flagged by the reviewer on #68267). ### Change - Align both except clauses with the `run_with_db_retries` policy `(DBAPIError, StaleDataError)`: roll back before the retry, and let retryable per-dag DB errors reach the retry loop. - `OperationalError` is a `DBAPIError` subclass, so existing behavior is preserved; non-retryable exceptions propagate unchanged. ### Tests - `test_sync_to_db_rolls_back_before_retry_on_integrity_error`: `IntegrityError` on attempt 1 rolls the session back and attempt 2 succeeds. - `test_serialize_dag_capturing_errors_reraises_retryable_db_errors`: retryable DB errors are re-raised, not recorded as import errors. - Both fail on the code before this change (recorded); full `test_collection.py` passes (61 passed, 2 pre-existing FAB skips). `mypy-airflow-core` clean. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes, Claude Code (Fable 5) Generated-by: Claude Code (Fable 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
