HotSushi opened a new issue, #4428:
URL: https://github.com/apache/iceberg-python/issues/4428

   ### Feature Request / Improvement
   
   Follow-up to [#4290](https://github.com/apache/iceberg-python/issues/4290) 
and [#4310](https://github.com/apache/iceberg-python/pull/4310) cc: @YuangGao 
@Fokko 
   
   [#4310](https://github.com/apache/iceberg-python/pull/4310) sends 
`Idempotency-Key` on mutations and retries keyed `POST`s under `rest.client.*`, 
but it doesn't enforce the lifetime the server advertises in `GET /v1/config`. 
The PR describes this as intentional because urllib3 retries are bounded. They 
aren't bounded by the key lifetime, though, and a retry outside that window can 
apply a mutation twice.
   
   The [REST catalog 
spec](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml)
 says:
   
   > Clients SHOULD NOT reuse an Idempotency-Key after this window elapses.
   
   Java enforces this in 
[`ExponentialHttpRequestRetryStrategy`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/ExponentialHttpRequestRetryStrategy.java)
 ([apache/iceberg#18070](https://github.com/apache/iceberg/pull/18070), 
building on 
[apache/iceberg#17947](https://github.com/apache/iceberg/pull/17947)).
   
   ### Why it matters
   
   An overloaded server responds with `503` and `Retry-After`. 
[urllib3](https://github.com/urllib3/urllib3/blob/main/src/urllib3/util/retry.py)
 honors `Retry-After` by default (`respect_retry_after_header=True`) and sleeps 
for the full value. `backoff_max` caps backoff only, not `Retry-After`.
   
   Example with `idempotency-key-lifetime: PT5M`:
   
   1. The client sends a commit. The server applies it, but the response is 
lost (timeout, proxy 502, dropped connection).
   2. The retry gets `503` with `Retry-After: 600`.
   3. urllib3 sleeps 600s and retries with the same key.
   4. The server has evicted the key, treats the request as new, and applies 
the mutation again.
   
   The result is a duplicate commit and corrupted table state, which is exactly 
what the key is supposed to prevent.
   
   Backoff alone can also exceed short windows. For example, `max-retries=5`, 
`backoff-factor=10` and a 30s socket timeout add up to about 450s of retrying.
   
   ### Current behavior
   
   - In 
[`pyiceberg/catalog/rest/__init__.py`](https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/rest/__init__.py),
 `ConfigResponse.idempotency_key_lifetime` is read as a string and only checked 
for presence.
   - `_create_connection_adapter(..., retry_post=True)` adds `POST` to 
`allowed_methods`. Nothing parses the lifetime or compares it with elapsed time.
   - `PT0S` or a negative duration still turns on keyed retries. In Java, a 
non-positive lifetime means keyed requests are never retried.
   - A malformed value is accepted silently.
   
   ### Requested behavior
   
   1. Parse `idempotency-key-lifetime` as an ISO-8601 duration, e.g. with 
`pydantic.TypeAdapter(timedelta)`.
      - Malformed: log a warning and disable the bound. The key is still sent, 
matching Java's 
[`HTTPClient.parseKeyLifetime`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java).
      - Zero or negative: never retry keyed requests.
   2. Record when the first attempt of a keyed request started.
   3. Before each retry of a keyed request, decline the retry if `elapsed + 
next_sleep >= lifetime`. `next_sleep` is the `Retry-After` value when present, 
otherwise the backoff time including jitter. When a retry is declined, return 
the last response so `_handle_non_200_response` raises the typed error, as it 
does today when retries run out.
   4. Leave non-keyed requests and idempotent methods (`GET`, `HEAD`, `PUT`, 
`DELETE`) without a key as they are.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to