arkadiuszbach commented on PR #63355:
URL: https://github.com/apache/airflow/pull/63355#issuecomment-4671852719
I was analyzing #65708 and, if I'm not mistaken, this also can be triggered
by the client-side timeout configured via `execution_api_timeout` (default
**5s**).
### Scenario
1. The worker reports the terminal state with a non-idempotent `PATCH
/task-instances/{id}/state` (e.g. `succeed()`).
2. Under pressure — high CPU on the API server or DB, or row-level lock
(I've seen some issue related to TI row locks recently) — that request takes
longer than `execution_api_timeout`, so the **client** times out.
3. But the Execution API endpoint is a sync `def`, so it runs in the anyio
threadpool and **isn't cancelled when the client disconnects**. The server
keeps going, commits the transaction, and sets the TI to `success`.
4. The client only saw a timeout (`httpx.RequestError`), which
`_should_retry_api_request` treats as retryable, so tenacity **re-sends the
same PATCH**.
5. The retry now finds `previous_state == success != running` → `409
invalid_state` (`previous_state: success`). `succeed()` doesn't catch the 409,
so it propagates as a `ServerResponseError` and the already-successful task
gets marked **failed**.
### Why this matches the report
This lines up with what was reported in #65708: the user mentions retries
and the tenacity logs showing the request being re-sent, e.g.
```
Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st
time calling it.
Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd
time calling it.
```
i.e. the same call is retried (1st → 2nd attempt), which is exactly the
duplicate `PATCH .../state` that hits the `409` on the second attempt.
> _Note: this comment was generated by Claude Opus 4.8._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
