linzch3 opened a new pull request, #4429:
URL: https://github.com/apache/iceberg-python/pull/4429

   Closes #4428
   
   # Rationale for this change
   
   #4310 sends `Idempotency-Key` on mutations and lets `rest.client.*` retry 
keyed `POST`s, but it never looks at the `idempotency-key-lifetime` the server 
returns from `/v1/config` beyond checking it's there. urllib3 sleeps for the 
full `Retry-After` (and backoff can add up too), so a retry can go out after 
the key has expired, and the server may then apply the same mutation twice. The 
spec says clients SHOULD NOT reuse a key after that window.
   
   This follows what the Java client does (`HTTPClient.parseKeyLifetime` + 
`ExponentialHttpRequestRetryStrategy`, apache/iceberg#18070):
   
   - `idempotency-key-lifetime` is parsed as an ISO-8601 duration 
(`TypeAdapter(timedelta)`).
     - malformed: log a warning and don't bound retries; the key is still sent 
(same as Java)
     - zero or negative: keyed requests are never retried
   - `_RetryTimeoutHTTPAdapter.send` records a deadline (`now + lifetime`, 
monotonic clock) in a `ContextVar` when the request carries an 
`Idempotency-Key`.
   - A small `Retry` subclass checks it in `increment`: if `now + next_sleep >= 
deadline` the retry is declined. `next_sleep` is what `Retry.sleep` would wait: 
`Retry-After` when urllib3 would honor it, otherwise the backoff plus the max 
jitter.
   - A declined retry raises `MaxRetryError`, i.e. it behaves exactly like 
running out of retries: with `raise_on_status=False` urllib3 hands back the 
last response, so `_handle_non_200_response` still raises the typed error (e.g. 
`ServiceUnavailableError`); a connection error is re-raised as before.
   
   Requests without a key (all `GET`/`HEAD`, and everything when the server 
doesn't advertise a lifetime) are untouched.
   
   ## Are these changes tested?
   
   Yes, in `tests/catalog/test_rest.py`, using the existing loopback server 
(now with an optional `Retry-After` header):
   - `Retry-After: 60` with `PT30S` lifetime: no retry, 
`ServiceUnavailableError`, 1 call
   - backoff alone exceeding the lifetime (`PT5S`, factor 10): stops before the 
second retry
   - `Retry-After: 0` within `PT30M`: still retried and succeeds
   - `PT0S` and `-PT5M`: keyed POST never retried, key still sent
   - `PT0S` doesn't affect unkeyed GETs, they still retry
   - malformed lifetime: warning logged, keyed POST still retried with the same 
key
   - `_parse_idempotency_key_lifetime` unit cases
   
   `make lint` passes. `pytest tests/catalog -m unit` passes except 4 kerberos 
tests in `test_hive.py` that fail the same way on `main` here (kerberos extra 
not installed locally).
   
   ## Are there any user-facing changes?
   
   Only for catalogs whose server advertises `idempotency-key-lifetime` and 
that set `rest.client.max-retries`: a keyed mutation is no longer retried once 
the next attempt would land outside the key's lifetime; the error that would 
have been raised after exhausting retries is raised earlier instead.
   
   ## Limitations
   
   - The deadline starts when the adapter sends the first attempt, not when the 
caller started the operation (e.g. the commit retry loop in `table/update` 
creates a new key per attempt, so that's fine).
   - The SigV4 adapter mounted on the catalog URI has its own plain integer 
retries and doesn't retry `POST` on status codes, so it isn't changed here.
   - The lifetime is checked before sleeping, so a slow next attempt itself 
(e.g. a long socket timeout) can still finish after the window. Java has the 
same behavior.
   
   #4310 cc @YuangGao. Happy to adjust if u'd prefer a different shape (e.g. 
doing this in the adapter instead of a `Retry` subclass). thx!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to