HotSushi opened a new issue, #4428: URL: https://github.com/apache/iceberg-python/issues/4428
### Feature Request / Improvement Follow-up to [#4290](https://github.com/apache/iceberg-python/issues/4290) and [#4310](https://github.com/apache/iceberg-python/pull/4310) cc: @YuangGao @Fokko [#4310](https://github.com/apache/iceberg-python/pull/4310) sends `Idempotency-Key` on mutations and retries keyed `POST`s under `rest.client.*`, but it doesn't enforce the lifetime the server advertises in `GET /v1/config`. The PR describes this as intentional because urllib3 retries are bounded. They aren't bounded by the key lifetime, though, and a retry outside that window can apply a mutation twice. The [REST catalog spec](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml) says: > Clients SHOULD NOT reuse an Idempotency-Key after this window elapses. Java enforces this in [`ExponentialHttpRequestRetryStrategy`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/ExponentialHttpRequestRetryStrategy.java) ([apache/iceberg#18070](https://github.com/apache/iceberg/pull/18070), building on [apache/iceberg#17947](https://github.com/apache/iceberg/pull/17947)). ### Why it matters An overloaded server responds with `503` and `Retry-After`. [urllib3](https://github.com/urllib3/urllib3/blob/main/src/urllib3/util/retry.py) honors `Retry-After` by default (`respect_retry_after_header=True`) and sleeps for the full value. `backoff_max` caps backoff only, not `Retry-After`. Example with `idempotency-key-lifetime: PT5M`: 1. The client sends a commit. The server applies it, but the response is lost (timeout, proxy 502, dropped connection). 2. The retry gets `503` with `Retry-After: 600`. 3. urllib3 sleeps 600s and retries with the same key. 4. The server has evicted the key, treats the request as new, and applies the mutation again. The result is a duplicate commit and corrupted table state, which is exactly what the key is supposed to prevent. Backoff alone can also exceed short windows. For example, `max-retries=5`, `backoff-factor=10` and a 30s socket timeout add up to about 450s of retrying. ### Current behavior - In [`pyiceberg/catalog/rest/__init__.py`](https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/rest/__init__.py), `ConfigResponse.idempotency_key_lifetime` is read as a string and only checked for presence. - `_create_connection_adapter(..., retry_post=True)` adds `POST` to `allowed_methods`. Nothing parses the lifetime or compares it with elapsed time. - `PT0S` or a negative duration still turns on keyed retries. In Java, a non-positive lifetime means keyed requests are never retried. - A malformed value is accepted silently. ### Requested behavior 1. Parse `idempotency-key-lifetime` as an ISO-8601 duration, e.g. with `pydantic.TypeAdapter(timedelta)`. - Malformed: log a warning and disable the bound. The key is still sent, matching Java's [`HTTPClient.parseKeyLifetime`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java). - Zero or negative: never retry keyed requests. 2. Record when the first attempt of a keyed request started. 3. Before each retry of a keyed request, decline the retry if `elapsed + next_sleep >= lifetime`. `next_sleep` is the `Retry-After` value when present, otherwise the backoff time including jitter. When a retry is declined, return the last response so `_handle_non_200_response` raises the typed error, as it does today when retries run out. 4. Leave non-keyed requests and idempotent methods (`GET`, `HEAD`, `PUT`, `DELETE`) without a key as they are. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
