Hi Huaxin, Could you clarify the specific use cases we intend to support regarding retry checking? Here are a couple of possibilities I had in mind:
- *Network infrastructure retries* – where the exact same request is retried. - *Client-side retries* – where the client regenerates the request using the same program logic, resulting in identical content. If there are no security or other concerns, I’d suggest keeping the specification simple and avoiding mechanisms that surface client-side implementation errors. The cleanest approach might be to ignore the request content and rely solely on a user-provided key. Alternatively, we could include an optional error code in the response, which implementations may use to signal conflicts. The actual conflict detection logic can be left to the implementations—we don’t need to define it in the specification. If we go this route, we should also offer a way to disable these checks, since there will inevitably be cases where semantically identical requests are incorrectly flagged as conflicting. Thanks, Peter huaxin gao <huaxin.ga...@gmail.com> ezt írta (időpont: 2025. szept. 19., P, 1:38): > Thanks Steven for the +1 and for raising the fingerprint question! Great > points! > > What we need to protect against: > > > - Same logical request, different bytes across retries (pretty vs > compact JSON, map key order, ...). > - Accidental key reuse with a changed payload. > > Options and tradeoffs: > > > - Exact byte checksum (e.g., SHA‑256 over raw body) > - Pro: trivial, fast > - Con: too strict; benign diffs cause false mismatches > > > - Canonical JSON over full request, then hash (proposed) > - Pro: stable across whitespace/key order; simple to implement for > typed payloads > - Con: slightly more work than raw checksum; > > > - Checksum of selected fields / field-by-field match > - Pro: can be faster for huge payloads; can ignore noisy fields > - Con: could misses legitimate differences > > > - Request digest/signature > - Pro: very strong > - Con: heavyweight > > Maybe we could make this configurable: > > > - canonical-json-sha256 (default) > - raw-bytes-sha256 (strict) > - trust-client-key (no fingerprint check) > > On the IETF draft status: > > I have also noted the draft’s expiry. We will align with its semantics for > now and can adjust if a new version lands. > > Thanks, > > Huaxin > > On Thu, Sep 18, 2025 at 4:01 PM Steven Wu <stevenz...@gmail.com> wrote: > >> +1 for the feature that can make retry safe for 500s and improve the >> client fault-tolerance of transient server failures. >> >> Peter and Dimitri raised a good question on the fingerprint. The IETF >> draft doesn't actually define the fingerprint algo. We can also go with >> simple checksum of the entire request payload, which would be cheap to >> compute. Do we anticipate any anticipated scenarios where clients may >> rewrite the payload in different forms of serialized bytes during retries? >> >> * Checksum of the entire request payload. >> * Checksum of selected element(s) in the request payload. >> * Field value match for each field in the request payload. >> * Field value match for selected element(s) in the request payload. >> * Request digest/signature >> >> >> BTW, the IETF draft seems to have expired without approval >> >> https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/ >> >> On Thu, Sep 18, 2025 at 3:46 PM huaxin gao <huaxin.ga...@gmail.com> >> wrote: >> >>> Thanks Peter and Dmitri for the thoughtful feedback! I really appreciate >>> you taking a close look at my proposal. I agree that "semantic equality" is >>> tricky, that's why the scope here is intentionally narrow. >>> >>> Just to clarify scope: I’m not trying to solve general semantic >>> equivalence. For these specific, typed request payloads, I serialize to a >>> deterministic JSON and hash it. That normalizes benign diffs (map order, >>> whitespace) without trying to infer meaning. The goal is a stable >>> fingerprint so that if a key is accidentally reused with a changed payload, >>> we surface that instead of silently diverging. >>> >>> To make this feel less brittle, I’ll add tests for the practical cases >>> (ordering/whitespace, nested maps, a clear null‑vs‑missing rule, numeric >>> formatting), plus end‑to‑end tests in the in‑memory REST fixture with >>> failure injection (in‑flight dup, finalize failure -> reconcile, etc.). >>> Happy to walk through these if helpful. >>> >>> I’m also open to adding a config switch for “trust‑client‑key only” if >>> that’s preferred in some environments. My intent is to stay aligned with >>> the IETF Idempotency‑Key guidance (first request wins; conflicting reuse is >>> rejected, and reusing a key with a different request payload is rejected >>> via an idempotency fingerprint) while keeping things as simple as possible >>> and protecting us from accidental key misuse. Would love to align on the >>> lightest approach that meets those goals. >>> >>> Thanks, >>> >>> Huaxin >>> >>> On Thu, Sep 18, 2025 at 6:17 AM Dmitri Bourlatchkov <di...@apache.org> >>> wrote: >>> >>>> Hi All, >>>> >>>> I agree that checking request contents is almost redundant in this case. >>>> >>>> If the randomness quality of Idempotency-Key value is good, collisions >>>> are very unlikely on the server side. Given that, any content checks the >>>> server performs are essentially validating that clients correctly reuse the >>>> generated Idempotency-Key value. (this is mostly the same as my comment on >>>> the related Polaris discussion). >>>> >>>> I'd like to propose making the content check optional so that servers >>>> may or may not implement it according to their design principles and >>>> constraints and emphasizing that clients should use unique keys (e.g. >>>> UUIDs)... basically going with option 2 from Peter's email. >>>> >>>> I believe this is in line with the SHOULD word used for this case in >>>> the IETF draft [1] (section 2.7). >>>> >>>> [1] >>>> https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header-06 >>>> >>>> Thanks, >>>> Dmitri. >>>> >>>> On Thu, Sep 18, 2025 at 7:56 AM Péter Váry <peter.vary.apa...@gmail.com> >>>> wrote: >>>> >>>>> Thanks Huaxin for the proposal, and sorry for the late review - I had >>>>> a bit of a busy week. >>>>> I have one main question, which I have also added as a comment to the >>>>> doc: >>>>> - Why do we try to compare the request contents when the >>>>> Idempotency-Key is the same for the requests? The comparison algorithm is >>>>> a >>>>> bit complicated, and seems brittle to me. Consistent field ordering, maps, >>>>> and maybe even inconsistency in upper case/lower case letters might mean >>>>> technically the same request. >>>>> >>>>> In my previous roles (admittedly more than 10 years ago) I was >>>>> extensively working on APIs like this, and we have never really succeeded >>>>> in creating a good enough "are these 2 requests are really the same >>>>> semantically" checks. >>>>> >>>>> I would simplify these requirements, unless there are serious >>>>> arguments for the existence of these checks: >>>>> >>>>> 1. Either check for exact matches - without any magic - this could >>>>> be used for detecting issues where the duplication happens on the >>>>> network >>>>> side, or >>>>> 2. Rely entirely on the clients to provide the correct >>>>> Idempotency-Key. >>>>> >>>>> I would prefer the 2nd. >>>>> Otherwise I agree with the contents of the proposal. It is nicely >>>>> done! (edited) >>>>> >>>>> Yufei Gu <flyrain...@gmail.com> ezt írta (időpont: 2025. szept. 18., >>>>> Cs, 2:54): >>>>> >>>>>> Thanks for the proposal. It's a nice feature to make retry more >>>>>> reliable and efficient. Left some comments. >>>>>> >>>>>> Yufei >>>>>> >>>>>> >>>>>> On Mon, Sep 15, 2025 at 3:53 PM Kevin Liu <kevinjq...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> Thanks for writing up the proposal! Makes sense to add idempotency >>>>>>> to mutation requests. >>>>>>> >>>>>>> It would be helpful to add this feature to both the catalog test >>>>>>> framework and the iceberg-rest-fixture >>>>>>> <https://github.com/apache/iceberg/blob/754679ddccdf81a97dc65d40f1a2a6fb9f6ee9b0/open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java#L112>. >>>>>>> The latter is used by the subprojects for testing and would come in >>>>>>> handy >>>>>>> when we want to test out the client implementation. >>>>>>> >>>>>>> For other reviewers, the Stripe documentation on idempotency was a >>>>>>> helpful read, https://docs.stripe.com/api/idempotent_requests. >>>>>>> >>>>>>> >>>>>>> Best, >>>>>>> Kevin Liu >>>>>>> >>>>>>> On Mon, Sep 15, 2025 at 11:38 AM Szehon Ho <szehon.apa...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Sounds like fairly standard practice and makes sense to me in the >>>>>>>> first read. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Szehon >>>>>>>> >>>>>>>> On Mon, Sep 15, 2025 at 10:09 AM Russell Spitzer < >>>>>>>> russellspit...@apache.org> wrote: >>>>>>>> >>>>>>>>> I think based on the feedback on the proposal and in recent syncs >>>>>>>>> we should probably move forward with the actual Spec Change PR so we >>>>>>>>> can >>>>>>>>> see what this looks like and move on to a discussion of how the >>>>>>>>> Catalog >>>>>>>>> test framework should test this. >>>>>>>>> >>>>>>>>> On 2025/08/22 18:26:23 huaxin gao wrote: >>>>>>>>> > Hi all, >>>>>>>>> > >>>>>>>>> > I’d like to propose a change to Iceberg’s REST API to make >>>>>>>>> mutation >>>>>>>>> > requests safely retryable. >>>>>>>>> > >>>>>>>>> > *The Problem* >>>>>>>>> > If a POST mutation (e.g., updateTable) succeeds in the catalog >>>>>>>>> but the >>>>>>>>> > client doesn’t receive the response (timeout, connection closed, >>>>>>>>> etc.), a >>>>>>>>> > second attempt can hit 409 Conflict. The client interprets the >>>>>>>>> 409 as a >>>>>>>>> > failed commit and deletes the associated metadata files, causing >>>>>>>>> > catalog/storage inconsistency. >>>>>>>>> > >>>>>>>>> > *The Proposed Solution* >>>>>>>>> > Introduces an optional Idempotency-Key HTTP header on REST >>>>>>>>> mutation >>>>>>>>> > endpoints and has the Iceberg client pass it through. >>>>>>>>> > >>>>>>>>> > *Semantics *(first processed request wins): >>>>>>>>> > >>>>>>>>> > - >>>>>>>>> > >>>>>>>>> > Same key + same canonical payload -> return the original >>>>>>>>> result (no >>>>>>>>> > re-execution). >>>>>>>>> > - >>>>>>>>> > >>>>>>>>> > Same key + different payload -> 422 (Unprocessable Content). >>>>>>>>> > >>>>>>>>> > *Capability discovery:* catalogs can advertise support and >>>>>>>>> retention so >>>>>>>>> > clients know when a retry is safe, e.g. >>>>>>>>> > >>>>>>>>> > { >>>>>>>>> > "idempotency-tokens-respected": true, >>>>>>>>> > "idempotency-token-lifetime": "30m" } >>>>>>>>> > >>>>>>>>> > *Scope in Iceberg:* update the OpenAPI to include the header, >>>>>>>>> and add >>>>>>>>> > client pass-through + honoring capability discovery. No server >>>>>>>>> > implementation is mandated—catalogs (e.g., Polaris) can implement >>>>>>>>> > storage/TTL/replay as they choose. >>>>>>>>> > >>>>>>>>> > *Standards alignment:* uses the industry-standard header name >>>>>>>>> and matches >>>>>>>>> > the IETF HTTPAPI Idempotency-Key draft >>>>>>>>> > < >>>>>>>>> https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header >>>>>>>>> > >>>>>>>>> > semantics. >>>>>>>>> > >>>>>>>>> > *Compatibility:* fully backward compatible. Servers that don’t >>>>>>>>> support it >>>>>>>>> > can ignore the header; clients can detect support via capability >>>>>>>>> discovery. >>>>>>>>> > >>>>>>>>> > Here is the proposal >>>>>>>>> > < >>>>>>>>> https://docs.google.com/document/d/1WyiIk08JRe8AjWh63txIP4i2xcIUHYQWFrF_1CCS3uw/edit?tab=t.0 >>>>>>>>> >. >>>>>>>>> > Looking forward to your thoughts. >>>>>>>>> > >>>>>>>>> > Thanks, >>>>>>>>> > >>>>>>>>> > Huaxin >>>>>>>>> > >>>>>>>>> >>>>>>>>
