manucorujo commented on issue #76529:
URL: https://github.com/apache/airflow/issues/76529#issuecomment-6747472279
Another way to reach this on the released 3.4.0: a race between the executor
event and the worker's own state update. The scheduler exited once with a
`ValueError` from `TaskInstance.archive()`.
Timeline for one task instance (`ti_id=01a1c13b-5a3e-...`, try 2,
KubernetesExecutor, retries left):
- 16:57:59.690: the executor sees the pod fail (`Pod phase: Failed`).
- 16:58:07.465: the scheduler processes the event: `reported that the task
instance ... [running] ... finished with state failed, but the task instance's
state attribute is running`. It calls `handle_failure()`.
- 16:58:30.248: the worker's `PATCH /execution/task-instances/{id}/state`
commits on the API server and archives the same row for the retry
(`up_for_retry`, `host_name` = that task pod). The API server was slow at that
time: `PATCH /state` took up to 116 s, so this request finished about 30 s
after the pod had stopped.
- 16:58:30.352: the scheduler exits:
```
File ".../airflow/jobs/scheduler_job_runner.py", line 1813, in
process_executor_events
ti.handle_failure(error=msg, session=session)
File ".../airflow/models/taskinstance.py", line 2826, in
fetch_handle_failure_context
ti = ti.prepare_db_for_next_try(session)
File ".../airflow/models/taskinstance.py", line 1786, in
prepare_db_for_next_try
self.archive(reason="retry", session=session)
File ".../airflow/models/taskinstance.py", line 1695, in archive
raise ValueError("An archived task instance cannot be archived again")
```
Our reading: `archive()` takes `SELECT ... FOR UPDATE` on the row, waits for
the API server transaction that archives the same row, and then finds
`working_set` false. Before 3.4.0 the same double `handle_failure` path did not
raise.
#76544 would stop the scheduler from exiting. It may also be worth a check
in `process_executor_events` that skips `handle_failure()` when the row is no
longer in the working set, so that the retry is not prepared twice.
It happened once. The scheduler did not crash again after the restart.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @manucorujo before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
