Thanks, Dmitri for the quick reply! IETF draft: the latest on the datatracker is -06, I’m not seeing a -07 yet. I’ll align to -06 for now and switch to the latest when it’s available.
On scope: I agree the spec text should stay client-visible and neutral; I’ll phrase it as “same key + same payload identity -> same HTTP status and body within the advertised lifetime,” and leave internals open. On Polaris details: I’ll add an “Alternative Implementations” section and include your tasks-based approach with a link to your thread. Thanks, Huaxin On Tue, Sep 16, 2025 at 2:51 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Huaxin, > > Thanks for mentioning the related IETF draft. Based on the links from your > Iceberg doc, I was able to find only revision 6 of this draft [2]. > > This revision is already expired, which is a bit of a surprise to me :) Do > you know of a newer revision? > > In any case, I believe it would be preferable to use the IETF draft as the > basis from the specification of client/server interactions in the REST API > and only put in the Iceberg/Polaris-specific docs the information where we > extend the IEFT proposal. > > That said, I guess this discussion thread becomes a discussion of possible > ways to implement idempotency keys in Polaris. > > I proposed an implementation based on "reliable Polaris tasks" [1]. > > If you have an alternative proposal in mind, it might be best to put it in > your Polaris document as a separate section (after the "spec" parts) for > further discussion. > > I'd appreciate it if you added my tasks-based impl. proposal as alternative > impl. section in that doc too, if you do not mind :) > > [1] https://lists.apache.org/thread/gg0kn89vmblmjgllxn7jkn8ky2k28f5l > [2] > > https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header-06 > > Cheers, > Dmitri. > > On Tue, Sep 16, 2025 at 4:43 PM Huaxin Gao <[email protected]> wrote: > > > Hi all, thanks for the thoughtful feedback (JB, Dmitri, Yufei). Sorry for > > the late reply. I missed the emails because I wasn’t subscribed to the > dev > > list. > > > > The companion Iceberg REST Catalog proposal is here (discussion + spec > > text): > > > > > https://docs.google.com/document/d/1WyiIk08JRe8AjWh63txIP4i2xcIUHYQWFrF_1CCS3uw/edit?tab=t.0#heading=h.jfecktgonj1i > > > > The discussion thread is here: > > https://lists.apache.org/[email protected] > > > > The intent of the Polaris proposal is to implement those Iceberg > > client-visible semantics. > > > > I see the conceptual link to tasks/operations. For this proposal, I’d > like > > to keep the scope small and client-visible. Concretely: > > > > Same key + same payload -> return the original result (no re-run) > > > > Same key + different payload -> 422 > > > > If the first request is still running -> don’t start another (409 or > > wait-then-replay) > > > > Only replay finalized results (2xx or terminal 4xx) > > > > Advertise support via GET /v1/config (idempotency-key-supported, > > idempotency-key-lifetime) > > > > This aligns with the IETF Idempotency-Key draft. It’s intentionally > > lightweight, so I’d prefer to keep the scope simple for now rather than > tie > > it to a tasks framework. > > > > Thanks, > > Huaxin > > > > On 2025/09/16 19:12:31 Yufei Gu wrote: > > > Hi Dimitri, > > > > > > Thanks for sharing your feedback. I see the analogy you’re making with > > > tasks/operations, but I don’t think it holds in this case. > > > > > > Idempotency and tasks solve very different problems: > > > > > > - Idempotency is about safely retrying a synchronous request without > > > causing side effects or inconsistencies. See more details in the > IETF > > > doc[1]. > > > - Tasks are about orchestrating and tracking asynchronous execution. > > > > > > Merging the two conceptually risks overcomplicating the problem. The > > client > > > doesn’t need or want to manage a long-lived task here, only to ensure > > that > > > a retry of the same payload doesn’t break consistency. > > > > > > I don’t think we should couple the idempotency proposal to tasks. Doing > > so > > > creates unnecessary dependencies and delays for a feature that is > > valuable > > > in its own right. Tasks may be useful for other scenarios, but > > idempotency > > > should remain a lightweight mechanism, independent of whether the > server > > > executes work synchronously or asynchronously. > > > > > > So while there might be some surface-level similarities, I believe > > treating > > > idempotency as a standalone concern is the more pragmatic path forward. > > > > > > [1] > > > > > > https://www.ietf.org/archive/id/draft-ietf-httpapi-idempotency-key-header-06.txt > > > > > > Yufei > > > > > > > > > On Tue, Sep 16, 2025 at 11:08 AM Dmitri Bourlatchkov <[email protected] > > > > > wrote: > > > > > > > Hi Yufei, > > > > > > > > As I understand, the proposal is to allow re-submitted requests to > > succeed > > > > when the previous request with the same key (ID) completed on the > > server > > > > side while the client was in the process of re-trying. > > > > > > > > I agree that this is a valuable feature. > > > > > > > > However, my point about connecting it to tasks / operations is > > completely > > > > at the conceptual level. > > > > > > > > If one considers the server as a black box, the client effectively > > submits > > > > a request with some key and payload. Then the client submits another > > > > request with the same key and payload and expects the second request > to > > > > return the result of the first request (success or failure). This is > > > > conceptually equivalent to the server executing requests as > > asynchronous > > > > tasks (the key being the task's ID). > > > > > > > > Therefore, before we consider any implementation for the > "idempotency" > > > > feature, I believe it would be wise to consider possible synergies > > with the > > > > tasks proposal [1]. > > > > > > > > I personally think these synergies are in fact quite large and > > powerful, > > > > especially considering Polaris Servers in a distributed environment. > > > > > > > > To put it another way, I believe implementing the "idempotency" > > proposal on > > > > top of the "tasks" proposal is going to be quite easy (if not > trivial). > > > > > > > > [1] https://lists.apache.org/thread/gg0kn89vmblmjgllxn7jkn8ky2k28f5l > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Sat, Sep 13, 2025 at 11:44 AM Yufei Gu <[email protected]> > > wrote: > > > > > > > > > Hi Dimitri, > > > > > > > > > > The idempotency key is mainly to reduce the chance of > > inconsistencies of > > > > > table commit. Here is a typical scenario: A commit succeeds but the > > > > client > > > > > misses the response (due to timeout, restart, etc.), client retries > > and > > > > > hits 409 Conflict, then mistakenly treats this as a failed commit > and > > > > > cleans up related files, leading to inconsistency. Without an > > idempotency > > > > > key, the best a client could do is to treat most of the non-200 > > response > > > > as > > > > > unknown status, and reload the table to verify, which is also > > > > inefficient. > > > > > > > > > > It could enable more use cases, as you mentioned, getting status of > > an > > > > > async task, but I would not complicate it at this point. That > > probably > > > > > deserves a separate proposal, since task status goes beyond simple > > HTTP > > > > > codes and would require clients to interpret them correctly. For > now, > > > > > http_status is sufficient, as it’s already well-supported by all > > existing > > > > > clients. > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Fri, Sep 12, 2025 at 7:58 PM Dmitri Bourlatchkov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Huaxin. > > > > > > > > > > > > Thanks again for starting this proposal. > > > > > > > > > > > > After a quick initial read of the docs, this feels like a > proposal > > to > > > > > treat > > > > > > each REST change request as an identifiable "task" or "operation" > > where > > > > > the > > > > > > ID is allocated by the client and the execution status is tracked > > by > > > > the > > > > > > server. > > > > > > > > > > > > Essentially, with the proposed feature implemented, it would be > > > > possible > > > > > to > > > > > > provide a "get status" API for each "idempotency key". > > > > > > > > > > > > Does that make sense? > > > > > > > > > > > > Do you think the proposal could be reformulated in terms of > > > > > > "tasks/operations"? From my POV it might be clearer and easier to > > > > > > understand. > > > > > > > > > > > > In that regard, as far as Polaris is concerned, I think it > > connected > > > > well > > > > > > and could build on top of the tasks proposal [1]. > > > > > > > > > > > > [1] > > https://lists.apache.org/thread/gg0kn89vmblmjgllxn7jkn8ky2k28f5l > > > > > > > > > > > > Thanks, > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On Thu, Sep 11, 2025 at 2:37 PM huaxin gao < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I’d like to propose adding *optional idempotent retries* to > > Polaris > > > > > REST > > > > > > > mutation endpoints via an Idempotency-Key header. > > > > > > > > > > > > > > *The Problem* > > > > > > > When a mutation (e.g., POST /v1/tables/{name}/commit) succeeds > > but > > > > the > > > > > > > client doesn’t receive the response (timeout, restart, etc.), a > > retry > > > > > can > > > > > > > hit 409 Conflict. Some clients then treat this as a failed > > commit and > > > > > > clean > > > > > > > up files that actually belong to the committed snapshot, > causing > > > > > > > catalog/storage inconsistency. > > > > > > > > > > > > > > *The Proposed Solution* > > > > > > > Accept Idempotency-Key on mutation routes. The server binds the > > key > > > > to > > > > > a > > > > > > > canonical payload hash and enforces: > > > > > > > > > > > > > > - > > > > > > > > > > > > > > *Same key + same payload* -> return the original result (no > > > > > > > re-execution). > > > > > > > - > > > > > > > > > > > > > > *Same key + different payload* -> 422 Unprocessable Content. > > > > > > > - > > > > > > > > > > > > > > *In-flight duplicate* -> 409 Conflict. > > > > > > > Transient 5xx are never cached. > > > > > > > > > > > > > > *Discovery & Compatibility* > > > > > > > When serving Iceberg REST, Polaris can advertise support and > > > > retention > > > > > > via > > > > > > > GET > > > > > > > /v1/config, e.g.: > > > > > > > > > > > > > > { "properties": { "idempotency-key-supported": "true", > > > > > > > "idempotency-key-lifetime": "PT30M" } } > > > > > > > > > > > > > > Fully backward compatible: servers may ignore the header; > clients > > > > > enable > > > > > > > retries only when discovery indicates support. > > > > > > > > > > > > > > This is the *server-side counterpart* to the Iceberg REST > Catalog > > > > > > > Idempotency proposal > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1WyiIk08JRe8AjWh63txIP4i2xcIUHYQWFrF_1CCS3uw/edit?tab=t.0#heading=h.jfecktgonj1i > > > > > > > > > > > > > > > I sent to the Iceberg community. > > > > > > > Here is the Polaris proposal > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1L5A8Cspugsk1dW4Ij4dy5wB5FKa8KnaXT0LHBJdgq9w/edit?usp=sharing > > > > > > > >. > > > > > > > Feedback welcome! > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Huaxin > > > > > > > > > > > > > > > > > > > > > > > > > > > >
