vatsrahul1001 opened a new pull request, #75739:
URL: https://github.com/apache/airflow/pull/75739
### What
Remove the forced `airflow dags reserialize` from the e2e test session
startup (`spin_up_airflow_environment`).
### Why
The `Additional PROD image tests` e2e lanes have been failing on `v3-4-test`
with the environment never coming up:
```
Spinning up airflow environment using
ghcr.io/apache/airflow/.../prod/python3.11
Failed to start docker compose
airflow-dag-processor-1 | sqlalchemy.exc.PendingRollbackError: ...
Original exception was:
(psycopg.errors.UniqueViolation) duplicate key value violates unique
constraint "dag_pkey"
[SQL: INSERT INTO dag (dag_id=...)]
```
Root cause: after the stack is healthy the harness ran `airflow dags
reserialize` **inside the live `airflow-dag-processor` container while that
processor is still parsing the same Dags**. The two processes both do a
check-then-insert on the same `dag_id`, so one hits a `dag_pkey`
`UniqueViolation`. The dag-processor's DB-retry wrapper only rolls back on
`OperationalError`, so the `IntegrityError` leaves the session in a
pending-rollback state; the retry then dies with `PendingRollbackError` and the
processor crashes — taking the compose env (and the `reserialize` exec, exit
137) down with it, so pytest errors out at session start.
The forced reserialize is redundant: the `airflow-dag-processor` service
already serializes the Dags on its own, and every test path already polls for a
Dag before using it (`wait_for_dag` / `wait_for_dag_ids`, and `trigger_dag` →
`un_pause_dag` → `wait_for_dag`). Dropping the forced reserialize removes the
second concurrent writer and the crash, without losing any Dag-visibility
guarantee.
### Note
This fixes the e2e **harness** (the thing that triggers the race). The
underlying dag-processor robustness gap — crashing on a concurrent `dag_id`
insert instead of rolling back and retrying — is a separate, pre-existing issue
(the narrow `except OperationalError` rollback in
`update_dag_parsing_results_in_db`, from #44898) and can be hardened separately.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes - Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
