Hi Huaxin, The general direction of PR 4912 looks good to me.
I'll review the exact changes in GH. Cheers, Dmitri. On Tue, Jun 30, 2026 at 1:16 PM huaxin gao <[email protected]> wrote: > Hi Dmitri, all, > > Following up on the early thread and Dmitri’s review on the POC (PR #4912 > <https://github.com/apache/polaris/pull/4912>): the POC explores a > different path: stamp the key into the entity’s internalProperties in the > same transaction as the create, so the key and the table commit atomically. > > That closes the record-after-success gap for operations that leave a > surviving entity (e.g. createTable). Trade-off: keys ride on the entity hot > path (size / write amplification on busy tables), vs. a separate store > where retention and size are independent. > > POC scope today is createTable only, behind polaris.idempotency.enabled. > I’ve added microbenchmarks on the read/write cost. Posting here since this > is a meaningful shift from the separate-store plan, would appreciate input > on whether we should pursue this. > > Thanks, > > Huaxin > > On Tue, Jun 2, 2026 at 3:00 PM huaxin gao <[email protected]> wrote: > > > Hi Dmitri, > > > > Sounds reasonable, and yes, that matches the plan. > > > > The idempotency persistence will be its own set of Java interfaces (a > > standalone IdempotencyStore SPI), separate from the metastore persistence > > interfaces, and wired and configured independently. So in principle you > > could pair a NoSQL metastore with a JDBC idempotency store. > > > > I also agree with the failure-mode framing. A retry can be confirmed as > > "previously processed" only if both the metastore change and the > > idempotency record committed. If the metastore commits but the > > idempotency record is lost, that retry won't be idempotent. That's the > > same record-after-success gap we discussed under Model B, and as you note > > it exists even with a shared persistence layer, so decoupling doesn't > > make it worse. I think it's a reasonable assumption, and I'll document > it. > > > > I'll get the PR up. Thanks again for the thorough discussion! > > > > Huaxin > > > > On Tue, Jun 2, 2026 at 12:17 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > >> Hi Huaxin, > >> > >> That sounds like a good plan to me. Looking forward to a PR. > >> > >> I think we still have time to adjust the impl. in the PR if other points > >> are raised during review in GH or by email. > >> > >> With this approach my main ask regarding Persistence impl. is (repeating > >> my > >> previous comments) to isolate the Idempotency persistence java > interfaces > >> from "meta store" persistence interfaces. Ideally one should be able to > >> combine a NoSQL MetaStore with a JDBC Idempotency persistence (in > theory). > >> > >> This will naturally create a situation when a request retry can be > >> confirmed as "previuosly processed" only if the original MetaStore > change > >> and the related Idempotency record insert both committed successfully. I > >> believe this is a reasonable assumption. > >> > >> If we commit the MetaStore change but lose the Idempotency record, the > >> client will not be able to re-try idempotently. However, this failure > mode > >> is still possible even if the MetaStore and Idempotency share the same > >> Persistence layer. > >> > >> Does this sound reasonable to you? > >> > >> Thanks, > >> Dmitri. > >> > >> On Tue, Jun 2, 2026 at 3:05 PM huaxin gao <[email protected]> > wrote: > >> > >> > Hi Dmitri, > >> > > >> > Thanks! I think we're converging. > >> > > >> > On 422: you're right. It isn't specified in the IRC spec. I'd added > it > >> > as a safeguard, not because the spec requires it, so I'm fine making > it > >> > best-effort. > >> > > >> > On performance: that's a fair point. For a fast-changing table, a late > >> > retry would usually fail on stale update requirements anyway, so a > long > >> > retention window doesn't add much practical value there. So I agree we > >> > can cap retention and size in practice without losing much. > >> > > >> > Since you're fine with a specialized persistence impl for idempotency > >> > keys, I'd propose we go that route. It keeps keys off the entity hot > >> path > >> > and lets us tune retention independently of entity size, so we get the > >> > simplicity without the write-amplification and cache cost on busy > >> tables. > >> > > >> > I'll proceed with the separate idempotency persistence in the PR > unless > >> > others object. Thanks for working through this with me! > >> > > >> > Huaxin > >> > > >> > On Tue, Jun 2, 2026 at 11:12 AM Dmitri Bourlatchkov <[email protected] > > > >> > wrote: > >> > > >> > > Hi Huaxin, > >> > > > >> > > Re: Question 1 (422 responses). > >> > > > >> > > I do not actually see any mention of 422 responses WRT > >> Idempotency-Key in > >> > > the IRC spec. Did I miss it? > >> > > > >> > > So, if that behaviour is not strictly specified, Polaris can choose > to > >> > > produce those response it in any way that makes sense for the > >> > > Polaris implementation, I think. > >> > > > >> > > Re: Performance. > >> > > > >> > > Yes, it is a valid concern. My previous suggestion for this approach > >> was > >> > > not a strict requirement, but a point for discussion. Let's see if > >> other > >> > > reviewers comment on this aspect too. > >> > > > >> > > Thinking about the quick-changing table use case, I believe the more > >> > > frequent are the updates to a table the less retention period is > >> required > >> > > by the client. If a client takes too long to re-try and other > clients > >> > make > >> > > many in-between updates, that client is unlikely to commit > >> successfully > >> > > anyway due to table metadata having evolved beyond its expectations. > >> > > > >> > > All in all, in practice it should be possible to cap the size of > >> > > idempotency data, I think. This may be a bit at odds with the > current > >> IRC > >> > > spec language regarding idempotency key retention, but it might > still > >> > work > >> > > effectively. > >> > > > >> > > Also, UUID v7 has a time component. The server should be able to > >> > recognize > >> > > keys created outside the range of "recent" entries and flag those > >> cases > >> > > (e.g. in log) for the Polaris Admin user to note and take corrective > >> > > actions. > >> > > > >> > > As for me, I'm fine with a specialized persistence impl. for > >> idempotency > >> > > keys too. > >> > > > >> > > Cheers, > >> > > Dmitri. > >> > > > >> > > On Tue, Jun 2, 2026 at 1:14 PM huaxin gao <[email protected]> > >> > wrote: > >> > > > >> > > > Hi Dmitri, > >> > > > > >> > > > You're right: DELETE is a weak case for the idempotency key, so > I'll > >> > > > drop that objection. > >> > > > > >> > > > Two reasons I agree. First, a client can handle DELETE failures on > >> its > >> > > > own: reload the table, see that it's gone, and stop. Second, your > >> > > > identity point is the deeper one. The DELETE API uses the table > >> name, > >> > not > >> > > > a physical table id. If a name is dropped and recreated, it points > >> to a > >> > > > different table over time, so the key can't give a clean guarantee > >> for > >> > > > DELETE anyway. > >> > > > > >> > > > So I'm fine scoping idempotency to operations that leave a > surviving > >> > > > entity: create, commit/update, register, rename. For those, > storing > >> the > >> > > > key in the entity properties works, and the delete-storage problem > >> goes > >> > > > away. > >> > > > > >> > > > That leaves two things I'd want to settle before we pick > >> > entity-property > >> > > > storage over a separate store: > >> > > > > >> > > > 1. Key reuse (422). With keys in entity properties there's no > global > >> > > > (realm, key) index. If a client reuses one key for a different > >> > > > resource, we can't detect it and return 422. Are we okay > treating > >> > that > >> > > > as best-effort? > >> > > > > >> > > > 2. Performance on the hot path. I want to second Yufei's concern. > >> The > >> > > > number of keys per entity is roughly (write rate × retention > >> > window), > >> > > > and the cost concentrates on the busiest tables. Since entity > >> > > > properties are serialized as one blob and rewritten on each > >> update, > >> > > > every commit rewrites all the stored keys, not just the new > one, > >> so > >> > a > >> > > > hot table pays growing write amplification on its commit path, > >> plus > >> > > > larger loads and a heavier cache footprint, even for plain > table > >> > > > loads. We can bound this with a tight retention cap, but that > >> > directly > >> > > > shrinks the idempotency window, which is the part clients > >> actually > >> > > > rely on. A separate store keeps this off the hot path and lets > >> > > > retention be tuned independently of entity size. > >> > > > > >> > > > So my main questions are: are we okay with best-effort 422, and > how > >> do > >> > we > >> > > > want to handle the hot-path cost? If both have good answers, I > agree > >> > > > entity-property storage is the simpler choice. WDYT? > >> > > > > >> > > > Thanks, > >> > > > Huaxin > >> > > > > >> > > > On Tue, Jun 2, 2026 at 6:59 AM Dmitri Bourlatchkov < > >> [email protected]> > >> > > > wrote: > >> > > > > >> > > > > Hi Huaxin, > >> > > > > > >> > > > > Good point about handling DELETE idempotently! > >> > > > > > >> > > > > However, I wonder whether it is a critical use case? > >> > > > > > >> > > > > Do you expect DELETE to benefit a lot from the Idempotency Key? > >> > > > > > >> > > > > I'd think it should be fairly straightforward for the client to > >> > reload > >> > > > the > >> > > > > table to be deleted in case of failures, discover that it is > gone, > >> > and > >> > > > not > >> > > > > retry. WDYT? > >> > > > > > >> > > > > There's still the question of whether the client is deleting the > >> > table > >> > > it > >> > > > > actually intends to delete. Another client could delete the > >> current > >> > > table > >> > > > > and create a new table under the same name while the first > client > >> is > >> > > > > "deliberating". The IRC API does not provide for unique table > >> > > > > identification in DELETE operations, as far as I know. The > >> operation > >> > is > >> > > > > invoked simply on the name, which can map to different physical > >> > tables > >> > > at > >> > > > > different times. Adding Idempotency Keys does not help in this > >> > > context, I > >> > > > > think. > >> > > > > > >> > > > > Thanks, > >> > > > > Dmitri. > >> > > > > > >> > > > > On Mon, Jun 1, 2026 at 9:59 PM huaxin gao < > [email protected] > >> > > >> > > > wrote: > >> > > > > > >> > > > > > Hi Dmitri, > >> > > > > > > >> > > > > > I like the idea — the atomic key write closes the in-flight > gap, > >> > and > >> > > it > >> > > > > > avoids the Iceberg metadata and spec issues. Agreed too that > >> losing > >> > > > keys > >> > > > > > on already-deleted entities is harmless. > >> > > > > > > >> > > > > > But I think the harder case is delete operations themselves. > For > >> > drop > >> > > > > > table/view/namespace, the operation removes the entity, so > >> there is > >> > > no > >> > > > > > surviving entity to hold the key. A retry of a successful drop > >> > should > >> > > > > > return an equivalent success, but with entity-property storage > >> the > >> > > key > >> > > > > > has nowhere to live — so the retry would just see "not found" > >> and > >> > > > behave > >> > > > > > differently. Where would a drop's key live in this model? > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Huaxin > >> > > > > > > >> > > > > > On Mon, Jun 1, 2026 at 6:13 PM Yufei Gu <[email protected] > > > >> > > wrote: > >> > > > > > > >> > > > > > > One concern I have with storing idempotency records as > entity > >> > > > > properties > >> > > > > > is > >> > > > > > > the potential performance impact. Over time, an entity could > >> > have a > >> > > > > large > >> > > > > > > number of idempotency key/value pairs. That would increase > the > >> > > > entity's > >> > > > > > > size, which may affect load, update, serialization, and > >> caching > >> > > costs > >> > > > > for > >> > > > > > > normal catalog operations, even when idempotency is not > >> involved. > >> > > Use > >> > > > > > cases > >> > > > > > > such as table loading and entity in-memory caching could be > >> > > affected. > >> > > > > > > Before moving in that direction, I think it would be useful > to > >> > > better > >> > > > > > > understand and measure the performance implications. If the > >> > entity > >> > > > size > >> > > > > > > growth turns out to be negligible in practice, the approach > >> may > >> > > still > >> > > > > be > >> > > > > > > attractive because of its transactional simplicity. > >> > > > > > > > >> > > > > > > Yufei > >> > > > > > > > >> > > > > > > > >> > > > > > > On Mon, Jun 1, 2026 at 2:17 PM Dmitri Bourlatchkov < > >> > > [email protected] > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Hi Huaxin, > >> > > > > > > > > >> > > > > > > > How about storing idempotency keys in the Polaris Entity > >> > > properties > >> > > > > > (not > >> > > > > > > > Iceberg metadata)? > >> > > > > > > > > >> > > > > > > > I understand that entities can be deleted thus discarding > >> > > > previously > >> > > > > > > > recorded keys, but based on the use cases discussed so > far, > >> it > >> > > does > >> > > > > not > >> > > > > > > > look like deleted entities should be a functional concern. > >> > > > > > > > > >> > > > > > > > Storing idempotency keys inside the entity will ensure > that > >> > their > >> > > > > > updates > >> > > > > > > > are processed in the same logical change set as the entity > >> > > changes > >> > > > > from > >> > > > > > > the > >> > > > > > > > IRC request payload. > >> > > > > > > > > >> > > > > > > > This will ensure uniform operations across all Persistence > >> > > > > > > implementations > >> > > > > > > > and will not require any Idempotency-specific Persistence > >> > > changes. > >> > > > > > > > > >> > > > > > > > WDYT? > >> > > > > > > > > >> > > > > > > > Thanks, > >> > > > > > > > Dmitri. > >> > > > > > > > > >> > > > > > > > On Sun, May 31, 2026 at 2:35 PM huaxin gao < > >> > > [email protected] > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Hi Dmitri, Robert, > >> > > > > > > > > > >> > > > > > > > > Thanks both. > >> > > > > > > > > > >> > > > > > > > > Dmitri — I agree with both of your points. > >> > > > > > > > > > >> > > > > > > > > - Idempotency storage will stay separate from the > >> > metastore. > >> > > It > >> > > > > > will > >> > > > > > > > > be separate in code and in transactions. We make the > >> > > > > idempotency > >> > > > > > > > > decision before the handler runs, or after it > commits > >> — > >> > > never > >> > > > > > > inside > >> > > > > > > > > the metastore transaction. > >> > > > > > > > > - I'll document the assumption you raised. Model B is > >> only > >> > as > >> > > > > > strict > >> > > > > > > as > >> > > > > > > > > the spec wants if the client builds the request so > >> that > >> > at > >> > > > most > >> > > > > > one > >> > > > > > > > > try can commit (for example, update requirements). > The > >> > > > > catalog's > >> > > > > > > > > optimistic concurrency makes sure of this. Model B > >> just > >> > > > records > >> > > > > > the > >> > > > > > > > > result on top of it. I'll say this clearly in the > >> Polaris > >> > > > docs. > >> > > > > > > > > > >> > > > > > > > > Robert — I see why the operation-id-in-metadata idea is > >> > > > appealing. > >> > > > > If > >> > > > > > > we > >> > > > > > > > > write the id inside the commit, it is atomic with the > >> change. > >> > > > That > >> > > > > > > would > >> > > > > > > > > close the in-flight gap for table and view operations. > >> That > >> > is > >> > > a > >> > > > > real > >> > > > > > > > > plus. > >> > > > > > > > > > >> > > > > > > > > But I don't think we should put the idempotency key in > >> table > >> > > > > > metadata. > >> > > > > > > > > Here is why: > >> > > > > > > > > > >> > > > > > > > > 1. It only works for table and view operations. It can't > >> help > >> > > > > > namespace > >> > > > > > > > > operations, grants, or other writes. A separate store > >> > > handles > >> > > > > all > >> > > > > > of > >> > > > > > > > > them with one mechanism. > >> > > > > > > > > > >> > > > > > > > > 2. It mixes two concerns. Idempotency is a REST/catalog > >> > > concern. > >> > > > > > Table > >> > > > > > > > > metadata should describe the table — schema, > snapshots, > >> > > > > > > partitioning, > >> > > > > > > > > sort order. A per-request id is not table state. I'd > >> > rather > >> > > > not > >> > > > > > mix > >> > > > > > > > > the two. > >> > > > > > > > > > >> > > > > > > > > 3. It bloats the metadata. To support retries we'd have > to > >> > keep > >> > > > > > > > > operation-ids with some retention/TTL. metadata.json > is > >> > > > > rewritten > >> > > > > > on > >> > > > > > > > > every commit and read on every table load. For tables > >> with > >> > > > many > >> > > > > > > > > writes, this adds real cost. And every client and > >> engine > >> > > that > >> > > > > > reads > >> > > > > > > > > the table pays it, not just the idempotency path. > >> > > > > > > > > > >> > > > > > > > > 4. It doesn't match the spec. The Iceberg REST spec > >> defines > >> > > > > > idempotency > >> > > > > > > > > at the protocol layer — an Idempotency-Key header > with > >> a > >> > > > > > server-side > >> > > > > > > > > contract. It does not store idempotency in table > >> metadata. > >> > > > > Putting > >> > > > > > > an > >> > > > > > > > > operation-id there would be a new mechanism that > isn't > >> in > >> > > the > >> > > > > spec > >> > > > > > > > > today. So it's a change to how the spec > >> > > > > > > > > works, and a cross-project change too. > >> > > > > > > > > > >> > > > > > > > > So I'd prefer to keep the record in a separate > idempotency > >> > > store. > >> > > > > We > >> > > > > > > > > accept the in-flight gap, but it is bounded. The > catalog's > >> > > > > optimistic > >> > > > > > > > > concurrency stops a duplicate commit from landing. And > >> once a > >> > > > > record > >> > > > > > > > > exists, retries replay cleanly. > >> > > > > > > > > > >> > > > > > > > > Thanks, > >> > > > > > > > > Huaxin > >> > > > > > > > > > >> > > > > > > > > On Sat, May 30, 2026 at 3:15 AM Robert Stupp < > >> [email protected] > >> > > > >> > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Hi all, > >> > > > > > > > > > > >> > > > > > > > > > Thanks for the clarifications. Russell's explanation > is > >> > > > > especially > >> > > > > > > > > useful. > >> > > > > > > > > > I agree, ambiguous request outcomes, for example, > >> timeouts > >> > or > >> > > > > > network > >> > > > > > > > > > connections being reset, are hard to reason about. > >> > > > > > > > > > > >> > > > > > > > > > Clients often cannot reliably reconcile from the > current > >> > > state > >> > > > > > alone > >> > > > > > > > for > >> > > > > > > > > > table/view state mutating operations. > >> > > > > > > > > > > >> > > > > > > > > > I wonder whether the idempotency key should be > recorded > >> in > >> > > the > >> > > > > > > > table/view > >> > > > > > > > > > metadata as an "operation-id", with an explicit > >> retention > >> > > > > > guarantee, > >> > > > > > > > > maybe > >> > > > > > > > > > tied to a server-provided minimum TTL. > >> > > > > > > > > > The approach could reduce or change the role of a > >> separate > >> > > > > > > > > > idempotency-record table and handling of it. > >> > > > > > > > > > > >> > > > > > > > > > Request handling could roughly look like this: > >> > > > > > > > > > if the current history/metadata already contains > that > >> > > > > > > "operation-id", > >> > > > > > > > > > return equivalent-enough response without > re-running > >> > the > >> > > > > > > operation. > >> > > > > > > > > > > >> > > > > > > > > > try the committing operation: > >> > > > > > > > > > if the commit succeeds: > >> > > > > > > > > > record the "operation-id" in the table/view > >> metadata, > >> > and > >> > > > > > > > > > return the successful response. > >> > > > > > > > > > if the commit runs into a conflict: > >> > > > > > > > > > re-check whether the current metadata/history > >> contains > >> > > that > >> > > > > > > > > > "operation-id" > >> > > > > > > > > > if so: > >> > > > > > > > > > return equivalent-enough response. > >> > > > > > > > > > otherwise: > >> > > > > > > > > > return the conflict response. > >> > > > > > > > > > > >> > > > > > > > > > This is not perfect either and needs spec work, > >> retention > >> > > > rules, > >> > > > > > and > >> > > > > > > > may > >> > > > > > > > > > only work for table and view operations. > >> > > > > > > > > > > >> > > > > > > > > > I mostly want to separate the questions: > >> > > > > > > > > > 1. What guarantees do clients actually need after an > >> > > ambiguous > >> > > > > > > outcome? > >> > > > > > > > > > 2. Where should the durable evidence for the guarantee > >> > live? > >> > > > > > > > > > > >> > > > > > > > > > Robert > >> > > > > > > > > > > >> > > > > > > > > > On Sat, May 30, 2026 at 4:30 AM Dmitri Bourlatchkov < > >> > > > > > > [email protected]> > >> > > > > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > Hi Russell, > >> > > > > > > > > > > > >> > > > > > > > > > > Thanks for the information! It clarifies the use > case > >> a > >> > lot > >> > > > (at > >> > > > > > > least > >> > > > > > > > > for > >> > > > > > > > > > > me :) > >> > > > > > > > > > > > >> > > > > > > > > > > In short, I'd say the main benefit is allowing > >> clients to > >> > > > avoid > >> > > > > > > > > conflicts > >> > > > > > > > > > > (409) on re-submitting changes that got committed by > >> the > >> > > > server > >> > > > > > > > without > >> > > > > > > > > > the > >> > > > > > > > > > > client receiving confirmation of the success. > >> > > > > > > > > > > > >> > > > > > > > > > > I believe the Iceberg REST Catalog spec [1] is > >> formally > >> > > > > stricter > >> > > > > > > than > >> > > > > > > > > > Model > >> > > > > > > > > > > B when it states "the server ensures no additional > >> > effects > >> > > > for > >> > > > > > > > requests > >> > > > > > > > > > > that carry the same Idempotency-Key". Since Model B > >> > permits > >> > > > > > request > >> > > > > > > > > > > re-execution, the possibility of additional side > >> effects > >> > > > cannot > >> > > > > > be > >> > > > > > > > > ruled > >> > > > > > > > > > > out completely based on the proposed server-side > >> > algorithm > >> > > > > alone. > >> > > > > > > The > >> > > > > > > > > > > server must assume that the client forms the > (change) > >> > > request > >> > > > > in > >> > > > > > > > such a > >> > > > > > > > > > way > >> > > > > > > > > > > that only one execution attempt can succeed (e.g. by > >> > using > >> > > > > > "update > >> > > > > > > > > > > requirements"). This is also mentioned in comments > on > >> > the > >> > > > doc > >> > > > > > [2]. > >> > > > > > > > > > > > >> > > > > > > > > > > This is probably worth mentioning in the Polaris > docs > >> > > related > >> > > > > to > >> > > > > > > > > > > our Idempotency-Key implementation. > >> > > > > > > > > > > > >> > > > > > > > > > > Assuming this kind of cooperation on the client > side, > >> I > >> > > > believe > >> > > > > > > > Model B > >> > > > > > > > > > can > >> > > > > > > > > > > be considered compliant with the spec [1]. > >> > > > > > > > > > > > >> > > > > > > > > > > In anticipation of fresh implementation PRs for this > >> > > feature, > >> > > > > I'd > >> > > > > > > > like > >> > > > > > > > > to > >> > > > > > > > > > > re-emphasize (IIRC I mentioned this before) that, I > >> > think, > >> > > we > >> > > > > > > should > >> > > > > > > > > > avoid > >> > > > > > > > > > > coupling Idempotency persistence with MetaStore > >> > persistence > >> > > > > (both > >> > > > > > > > > > code-wise > >> > > > > > > > > > > and transaction-wise). Model B processes > >> > > Idempotency-related > >> > > > > data > >> > > > > > > > > outside > >> > > > > > > > > > > the original change request's execution scope. > >> > Idempotency > >> > > > > > > decisions > >> > > > > > > > > are > >> > > > > > > > > > > made either before the request starts executing or > >> after > >> > it > >> > > > is > >> > > > > > > > > committed > >> > > > > > > > > > to > >> > > > > > > > > > > the MetaStore. > >> > > > > > > > > > > > >> > > > > > > > > > > [1] > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/polaris/blob/4e4eaf840bf71d431b13034b0dd6f338261d8e8b/spec/iceberg-rest-catalog-open-api.yaml#L2098 > >> > > > > > > > > > > > >> > > > > > > > > > > [2] > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/1hqTejVyYXDpL5MJcVc7NyhCslKaGH82QoqMEcUYPvkE/edit?tab=t.0 > >> > > > > > > > > > > > >> > > > > > > > > > > Cheers, > >> > > > > > > > > > > Dmitri. > >> > > > > > > > > > > > >> > > > > > > > > > > On Fri, May 29, 2026 at 8:26 PM Russell Spitzer < > >> > > > > > > > > > [email protected] > >> > > > > > > > > > > > > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > >> > > > > > > > > > > > The problem with a client attempting to determine > if > >> > it’s > >> > > > > > > > operations > >> > > > > > > > > > > > succeeded via load table, and the reason all this > >> work > >> > > has > >> > > > > > > > > proceeded, > >> > > > > > > > > > is > >> > > > > > > > > > > > that there is no way for a client to guaranteed > >> path to > >> > > > > > actually > >> > > > > > > > > > > determine > >> > > > > > > > > > > > if a commit occurred. There are too many > legitimate > >> > > > > mechanisms > >> > > > > > to > >> > > > > > > > > erase > >> > > > > > > > > > > > history from an Iceberg table to guarantee an > >> operation > >> > > > > > occurred. > >> > > > > > > > > > > > > >> > > > > > > > > > > > For example, you could check if your snapshot > >> exists in > >> > > > > > snapshot > >> > > > > > > > > > history > >> > > > > > > > > > > > but this could have been erased by expire > snapshots. > >> > > > > > > > > > > > > >> > > > > > > > > > > > Or you could check if the schema was modified > >> according > >> > > to > >> > > > > your > >> > > > > > > > > update, > >> > > > > > > > > > > but > >> > > > > > > > > > > > this too could have been undone by another > >> operation. > >> > > > Client > >> > > > > A > >> > > > > > > adds > >> > > > > > > > > > > column > >> > > > > > > > > > > > but gets time out, Client B removes the Column, > >> Client > >> > A > >> > > > > > retries > >> > > > > > > > and > >> > > > > > > > > > adds > >> > > > > > > > > > > > the column again. > >> > > > > > > > > > > > > >> > > > > > > > > > > > Because of this the Iceberg client usually just > >> bails > >> > out > >> > > > to > >> > > > > he > >> > > > > > > > user > >> > > > > > > > > > with > >> > > > > > > > > > > > an exception if it doesn’t get an actual > >> confirmation > >> > > that > >> > > > > the > >> > > > > > > > commit > >> > > > > > > > > > > > succeeded from the server. This leaves the “can I > >> retry > >> > > or > >> > > > > not” > >> > > > > > > as > >> > > > > > > > an > >> > > > > > > > > > > > exercise to the end user. > >> > > > > > > > > > > > > >> > > > > > > > > > > > In practice, actual Iceberg users work around this > >> sort > >> > > of > >> > > > > > thing > >> > > > > > > by > >> > > > > > > > > > > adding > >> > > > > > > > > > > > all sorts of custom metadata to hopefully persist > >> > history > >> > > > in > >> > > > > > the > >> > > > > > > > > table > >> > > > > > > > > > > > itself in some way that can’t be touched by expire > >> > > > snapshots, > >> > > > > > but > >> > > > > > > > > this > >> > > > > > > > > > is > >> > > > > > > > > > > > usually very fragile and also relies on all > clients > >> > > > behaving > >> > > > > > > well. > >> > > > > > > > > I’ve > >> > > > > > > > > > > > seen folks use custom table properties for example > >> > > > “batch-5: > >> > > > > > > > > committed” > >> > > > > > > > > > > > then manually have their own retry logic check > >> whether > >> > > this > >> > > > > > > > property > >> > > > > > > > > is > >> > > > > > > > > > > > set. Then, of course, they also have to add a > bunch > >> > > custom > >> > > > > > logic > >> > > > > > > to > >> > > > > > > > > > make > >> > > > > > > > > > > > sure they clean up this state as well. > >> > > > > > > > > > > > > >> > > > > > > > > > > > This is why Iceberg added the Idempotency path in > >> the > >> > > first > >> > > > > > > place, > >> > > > > > > > it > >> > > > > > > > > > > gives > >> > > > > > > > > > > > us a guaranteed way for clients to retry in case > of > >> a > >> > > > network > >> > > > > > > issue > >> > > > > > > > > or > >> > > > > > > > > > > > catalog issue with a guarantee they will not do > >> > duplicate > >> > > > > work > >> > > > > > be > >> > > > > > > > > > > retrying. > >> > > > > > > > > > > > With this in place the client can now cleanly > retry > >> > > (within > >> > > > > the > >> > > > > > > > > > > idempotency > >> > > > > > > > > > > > window) the same operation over and over without > >> > throwing > >> > > > an > >> > > > > > > > > exception > >> > > > > > > > > > to > >> > > > > > > > > > > > the end user. Only in a situation where the > catalog > >> > > cannot > >> > > > > > > respond > >> > > > > > > > > > over a > >> > > > > > > > > > > > very long time will the user actually have to do > >> some > >> > > sort > >> > > > of > >> > > > > > > > > > > > reconciliation. You can look at the history of the > >> > > Iceberg > >> > > > > > > client’s > >> > > > > > > > > > retry > >> > > > > > > > > > > > behavior with ambiguous server side or network > >> errors > >> > to > >> > > > see > >> > > > > > how > >> > > > > > > > this > >> > > > > > > > > > has > >> > > > > > > > > > > > been a problem in the past. > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Fri, May 29, 2026 at 1:24 PM huaxin gao < > >> > > > > > > [email protected] > >> > > > > > > > > > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > > Hi Robert, > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks for your reply! > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > You're right that Model B does not prevent > >> duplicate > >> > > > > > execution. > >> > > > > > > > The > >> > > > > > > > > > > > > record is written only after success. So if a > >> client > >> > > > times > >> > > > > > out > >> > > > > > > > > while > >> > > > > > > > > > > the > >> > > > > > > > > > > > > first request is still running, a retry can run > >> the > >> > > > handler > >> > > > > > > > again. > >> > > > > > > > > > > There > >> > > > > > > > > > > > > is no record yet to stop it. So Model B is > >> "remember > >> > > and > >> > > > > > > replay a > >> > > > > > > > > > > > > successful result," not "run exactly once." > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On the trade-off: Model A gives a stronger > >> guarantee, > >> > > but > >> > > > > it > >> > > > > > > > needs > >> > > > > > > > > > > > > reserve/heartbeat/purge state, which adds > >> complexity > >> > > and > >> > > > > > > > overhead. > >> > > > > > > > > > > Model > >> > > > > > > > > > > > > B is simpler and cheaper. The window it leaves > >> open > >> > is > >> > > > > small, > >> > > > > > > > and a > >> > > > > > > > > > > > > client only retries after a timeout, so racing > >> first > >> > > > > requests > >> > > > > > > > > should > >> > > > > > > > > > be > >> > > > > > > > > > > > > rare in practice. Every design is a trade-off, > >> and my > >> > > > view > >> > > > > is > >> > > > > > > > that > >> > > > > > > > > > > Model > >> > > > > > > > > > > > > B is the right one here. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > It also helps to be clear about where > >> duplicate-work > >> > > > > > protection > >> > > > > > > > > > really > >> > > > > > > > > > > > > comes from. It comes from the catalog itself, > not > >> > from > >> > > > > > > > idempotency. > >> > > > > > > > > > The > >> > > > > > > > > > > > > catalog uses optimistic concurrency. If wo first > >> > > attempts > >> > > > > > race, > >> > > > > > > > at > >> > > > > > > > > > most > >> > > > > > > > > > > > > one commit wins and the other gets a 409. > >> Idempotency > >> > > > sits > >> > > > > on > >> > > > > > > top > >> > > > > > > > > of > >> > > > > > > > > > > > that. > >> > > > > > > > > > > > > It does not replace it. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > So what does Model B add over "the client just > >> calls > >> > > > > > loadTable > >> > > > > > > > and > >> > > > > > > > > > > > > reconciles"? Two things that I think are real: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > 1. The 422 check. loadTable can tell a client > >> that > >> > a > >> > > > > table > >> > > > > > > > > exists. > >> > > > > > > > > > It > >> > > > > > > > > > > > > cannot tell the client that the table THEY > >> > created > >> > > > > with > >> > > > > > > THIS > >> > > > > > > > > key > >> > > > > > > > > > > is > >> > > > > > > > > > > > > the one that succeeded. The record binds > the > >> key > >> > > to > >> > > > > > > > > (principal, > >> > > > > > > > > > > > > operation, resource). If the same key is > >> reused > >> > > for > >> > > > a > >> > > > > > > > > different > >> > > > > > > > > > > > > request, the server returns 422. The client > >> > cannot > >> > > > > > detect > >> > > > > > > > this > >> > > > > > > > > > on > >> > > > > > > > > > > > > its own. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > 2. One server-side behavior for all mutating > >> ops. > >> > > > > > > create-table > >> > > > > > > > > > > happens > >> > > > > > > > > > > > > to reconcile cleanly with loadTable. But > the > >> > point > >> > > > of > >> > > > > > the > >> > > > > > > > > > > > > Idempotency-Key header is that the client > >> should > >> > > not > >> > > > > > have > >> > > > > > > to > >> > > > > > > > > > write > >> > > > > > > > > > > > > reconciliation logic for every operation. > >> For a > >> > > > known > >> > > > > > key, > >> > > > > > > > the > >> > > > > > > > > > > > > server turns what would be a 409 into an > >> > > equivalent > >> > > > > 2xx > >> > > > > > > > > replay. > >> > > > > > > > > > > The > >> > > > > > > > > > > > > client gets a clean success instead of an > >> error > >> > it > >> > > > has > >> > > > > > to > >> > > > > > > > > > special- > >> > > > > > > > > > > > > case. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > There is a third, weaker benefit: once a record > >> > exists, > >> > > > > > retries > >> > > > > > > > > stop > >> > > > > > > > > > > > > seeing flip-flopping results. But that only > helps > >> > > after a > >> > > > > > > record > >> > > > > > > > > > > exists, > >> > > > > > > > > > > > > which is exactly the window you pointed out is > >> > > > unprotected. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > So I'll correct my earlier wording. This is not > >> > > > convergence > >> > > > > > on > >> > > > > > > > > > exactly- > >> > > > > > > > > > > > > once idempotency. It is a narrower guarantee: > >> replay > >> > a > >> > > > > > recorded > >> > > > > > > > > > result, > >> > > > > > > > > > > > > plus detect key misuse. It sits on top of the > >> > catalog's > >> > > > > > > existing > >> > > > > > > > > > > > > concurrency control. The real question for the > >> list > >> > is > >> > > > > > simple: > >> > > > > > > is > >> > > > > > > > > > that > >> > > > > > > > > > > > > narrower guarantee worth shipping on its own? Or > >> do > >> > we > >> > > > need > >> > > > > > > Model > >> > > > > > > > > A's > >> > > > > > > > > > > > > in-flight protection to have a strong > idempotency > >> > > > > guarantee? > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > My view is that the narrow version is worth it > for > >> > now: > >> > > > > it's > >> > > > > > > the > >> > > > > > > > > > > > > behavior the spec asks for, the 422 check can't > be > >> > done > >> > > > > > > > > client-side, > >> > > > > > > > > > > and > >> > > > > > > > > > > > > it's a small change we can strengthen toward > >> Model A > >> > > > later > >> > > > > > if a > >> > > > > > > > > real > >> > > > > > > > > > > use > >> > > > > > > > > > > > > case needs it. Happy to hear what others think. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Best, > >> > > > > > > > > > > > > Huaxin > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Fri, May 29, 2026 at 7:36 AM Robert Stupp < > >> > > > > [email protected] > >> > > > > > > > >> > > > > > > > > wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Hi Huaxin, > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Thanks for writing this up and moving the > design > >> > > > > discussion > >> > > > > > > > back > >> > > > > > > > > to > >> > > > > > > > > > > > dev@ > >> > > > > > > > > > > > > . > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Since you’re asking before locking in the > >> > > > > implementation, I > >> > > > > > > > think > >> > > > > > > > > > we > >> > > > > > > > > > > > > should > >> > > > > > > > > > > > > > clarify one point. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Model B is certainly simpler than the > >> lease-based > >> > > > > approach, > >> > > > > > > but > >> > > > > > > > > I’m > >> > > > > > > > > > > not > >> > > > > > > > > > > > > > sure I fully understand what problem it still > >> > solves. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > As I read it, if a client times out while the > >> > > original > >> > > > > > > request > >> > > > > > > > is > >> > > > > > > > > > > still > >> > > > > > > > > > > > > > running, a retry with the same key may not see > >> an > >> > > > > > idempotency > >> > > > > > > > > > record > >> > > > > > > > > > > > yet > >> > > > > > > > > > > > > > and could run the handler again. > >> > > > > > > > > > > > > > So this feels less like preventing duplicate > >> > > execution > >> > > > > and > >> > > > > > > more > >> > > > > > > > > > like > >> > > > > > > > > > > > > > remembering a successful result after the > fact. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > For the create-table case, couldn’t a client > >> > achieve > >> > > > > > roughly > >> > > > > > > > the > >> > > > > > > > > > same > >> > > > > > > > > > > > > > recovery by calling loadTable after an > ambiguous > >> > > > timeout > >> > > > > > and > >> > > > > > > > > > > > reconciling > >> > > > > > > > > > > > > > from there? > >> > > > > > > > > > > > > > Since Model B also rebuilds the response from > >> > current > >> > > > > > catalog > >> > > > > > > > > > state, > >> > > > > > > > > > > > I’m > >> > > > > > > > > > > > > > trying to understand what it gives us beyond > >> that. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > I’m not against simplifying the design, but I > >> think > >> > > we > >> > > > > > should > >> > > > > > > > be > >> > > > > > > > > > > clear > >> > > > > > > > > > > > > > about the narrower guarantee before calling > this > >> > > > > > convergence. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Best, > >> > > > > > > > > > > > > > Robert > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Fri, May 29, 2026 at 12:29 AM huaxin gao < > >> > > > > > > > > > [email protected]> > >> > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Hi all, > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > I've simplified the proposed design for > >> > > > Idempotency-Key > >> > > > > > > > support > >> > > > > > > > > > in > >> > > > > > > > > > > > > > Polaris > >> > > > > > > > > > > > > > > (Iceberg REST spec — retries with the same > key > >> > must > >> > > > not > >> > > > > > > > produce > >> > > > > > > > > > > > > > additional > >> > > > > > > > > > > > > > > side effects), and I'd like a wider review > >> before > >> > > > > > updating > >> > > > > > > > the > >> > > > > > > > > > > > > > > implementation PR (#4269 < > >> > > > > > > > > > > > https://github.com/apache/polaris/pull/4269 > >> > > > > > > > > > > > > >). > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > What changed > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > - Before (Model A, lease-based): reserve > an > >> > > > > idempotency > >> > > > > > > row > >> > > > > > > > > > > before > >> > > > > > > > > > > > > > doing > >> > > > > > > > > > > > > > > work → IN_PROGRESS / heartbeat → finalize > >> after. > >> > > > > > > > > > > > > > > - After (Model B, optimistic commit): run > >> the > >> > > > handler > >> > > > > > > > first → > >> > > > > > > > > > > > record > >> > > > > > > > > > > > > > only > >> > > > > > > > > > > > > > > after a successful (2xx) outcome. The record > >> > stores > >> > > > > > > binding + > >> > > > > > > > > > > status, > >> > > > > > > > > > > > > not > >> > > > > > > > > > > > > > > the HTTP response body. Retries with the > same > >> key > >> > > > > > re-derive > >> > > > > > > > an > >> > > > > > > > > > > > > equivalent > >> > > > > > > > > > > > > > > response from current catalog state > >> > > > > > > > > > > > > > > instead of replaying a stored payload. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > The design doc still compares Model A and > >> Model B > >> > > > > > > > side-by-side > >> > > > > > > > > so > >> > > > > > > > > > > the > >> > > > > > > > > > > > > > > trade-offs are explicit. So far the > discussion > >> > has > >> > > > been > >> > > > > > > > leaning > >> > > > > > > > > > > > toward > >> > > > > > > > > > > > > > > Model B — mutating REST operations only, > >> 2xx-only > >> > > > > > > > persistence, > >> > > > > > > > > no > >> > > > > > > > > > > > > > > response-body storage, and the known > >> > > > > > > > > > > > > > > trade-offs (e.g. concurrent first-request > >> races; > >> > > see > >> > > > > the > >> > > > > > > > NOTES > >> > > > > > > > > > > > section > >> > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > the doc). > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Does this direction look right before we > lock > >> in > >> > > the > >> > > > > > > > > > > implementation? > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Comments on the doc > >> > > > > > > > > > > > > > > < > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/1hqTejVyYXDpL5MJcVc7NyhCslKaGH82QoqMEcUYPvkE/edit?tab=t.0 > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > or replies on this thread both work. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > > Huaxin > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
