Hi All, It's good to see this discussion progressing :)
I do not insist on strongly binding REST requests to JDBC (or other) transactions. However, I think we have to keep in mind that the Persistence SPI should be implementable via JDBC. With that in mind, a trivial CAS on multiple changed rows may not be sufficient. For example, if we change table A, but have to read entities for tables B and C to check for location overlaps and B and C are served from memory (cache) in node X but are updated in parallel by node Y, how will Persistence be able to detect that conflict? Even "UPDATE where version = ?" for each of the changed entities does not guarantee that the database will detect the conflict. Not even with serializable Tx isolation. Serializable isolation might help, but then the cache always has to send all reads through to the database. That is, on every retry. Right now this is achieved by a combination of the Entity Cache and Resolver logic. However, the need to read through stems only from knowing a specific database implementation (RDBMS with Serializable isolation). So, the Cache and Resolver apparently make certain assumptions about that. At the same time there are call paths into Persistence that do not involve the Resolver. Those paths are not covered by the (implicit) read-through guarantees of the Resolver (and Entity Cache) code. It would be nice to abstract those aspects so that higher level (REST service) code could stay agnostic of Persistence technology and Persistence _implementation_ would then deal with caching and all database-specific aspects. More concretely, I think read-through for every entity does not have to be forced by the high-level code (where the Resolver currently lives). It is a concern of the database-specific layer. WDYT? Thanks, Dmitri. On Thu, Oct 1, 2026 at 12:14 AM Jean-Baptiste Onofré <[email protected]> wrote: > Hi Dennis > > I agree with you. I think "multi object conditional commits" rather > than a request-scoped transaction is aligned with what Robert and I > argued: commit attempts stay short, it maps to both JDBC and NoSQL. An > explicit read set checked at commit time stays compatible with > in-memory caching (I don't think a serializable only approach does > not). > > I believe it is also a small step forward from what we already have. > The multi-table commit path already does a multi-entity CAS through > updateEntitiesPropertiesIfNotChanged. We should extend it. > > I have some concerns about using PolarisResolutionManifest as is: > 1. As Dmitri mentioned, the location overlap check does not go through > the manifest at all. So the commit "contract" needs predicate checks > that the backend should re-evaluate at commit time. > 2. The manifest is not an "open book" today. > getPassthroughResolvedPath creates a single user Resolver on every > call and does not record the version it returned. I don't see how to > store the version easily today. > 3. In your example the catalog is not pinned, but validating a table > location against the allowed locations depends on the catalog storage > configuration, so for a create it would have to be. I would prefer > recording versions on read, or pinning by default with an explicit > opt-out. > 4. To simplify, I would define the read set as a small type in the > Persistence SPI, which the manifest produces, rather than making the > manifest itself the SPI. > > I would propose to define the read set in Persistence SPI terms, > independent from the resolver. and also include predicate (location > overlap, name conflict) in the contract. > > I think it's pretty close to what Dmitri is proposing (the > "Persistence Session "iidea). > > Regards > JB > > On Thu, Oct 1, 2026 at 2:41 AM Dennis Huo <[email protected]> wrote: > > > > Agree we should formulate in terms of Persistence SPIs first. > > > > IIRC we talked about this in one of the community syncs but I forgot to > > bring it here -- the PolarisResolutionManifest in concept already > > represents *exactly* the set of "which elements were engaged during > request > > processing". > > > > Adherence may have drifted, but at least early on, we made sure all the > > IcebergCatalog logic did *not* get to do new direct reads from the > > persistence layer, but instead needs to "read" from the *authorized* set > of > > entities contained in a PolarisResolutionManifest (or declared as a > > "passthroughPath" == allow reading fresh from DB but had to pre-register > it > > for authz purposes). > > > > It's important to keep this unified anyways because it really serves two > > things: > > > > 1. Declares the set of entities whose state you expect to > resolve/snapshot > > up-front in the operation-handling, that the IcebergCatalog logic then > > operates on later; also explicitly declare which ones require fetching > > *fresh* versions mid-operation (addPassthroughPath) > > 2. Enforces that RBAC authz is performed against that pre-declared set of > > entities -- if later processign were allowed to access arbitrary other > > entities to make handler decisions, then how did we "prove" that the > > authorization engine actually allowed the request to access those > arbitrary > > other entities? Even if it's just for internal-processing purposes, the > > very fact that something got accessed means there's a route to accidental > > leakage of unauthorized data. > > > > So in a way, solid/provable authz and solid/provable consistency go > > hand-in-hand. > > > > I think PolarisResolutionManifest in its current form is probably a bit > > opaque and doesn't 100% convey the semantics we want quite yet, but it > > seems like the right SPI starting point to extend or at least evolve > from. > > Basically, aside from addPath and addPassthroughPath, there could be a > > notion of annotating a given path as being one of the "dependent states" > to > > carry over into the later commit. Conceptually: > > > > // Prepare operation processing > > resolutionManifest.prefetch(rootCatalog, mustBeUnchangedAtCommitTime = > > false); // For this operation let's pretend we don't care if the root > > catalog changes before we commit the change to the tables > > resolutionManifest.prefetch(parentNamespace, mustBeUnchangedAtCommitTime > = > > false); // For this operation let's pretend we don't care if the parent > > namespace changes before we commit the change to the tables > > resolutionManifest.prefetch(table1, mustBeUnchangedAtCommitTime = true); > > // Assume multi-table commit - table1 will have changes and also must > not > > have already changed at commit time > > resolutionManifest.prefetch(table2, mustBeUnchangedAtCommitTime = true); > > // Same for table2 > > > > // .... Do all the Iceberg processing work > > > > // When we commit, we use the resolutionManifest as the ledger of things > we > > used that must still be at their unchanged version at time of commit > > metastore.doConditionalBatchOmmit(List.of(table1Updated, table2Updated), > > resolutionManifest.getSetOfUnchangedEntityRequirements()); > > > > Overall this fits the optimistic-concurrency theme of Iceberg and Polaris > > in general, and would *intentionally* not try to express a full-fledged > > "Transaction" session in which you do a bunch of > reads/writes/reads/writes > > on a single entity as if you're operating in an uncommitted block on a > > RDBMS, instead opting for the middle-ground of "multi-object conditional > > commits". > > > > On Tue, Sep 22, 2026 at 4:10 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi EJ, > > > > > > > 1. Which backend assumptions should we design for? [...] > > > > > > I know I used the term "transaction" a lot during the call, but the > real > > > challenge is about establishing an SPI with clear contracts that could > be > > > implemented by JDBC and NoSQL persistence using the database features > > > natural to each model. > > > > > > I am fairly confident that a JDBC impl. is possible with Serializable > > > transaction. We should be able to solve all the problems raised in this > > > thread. However, that will make in-memory caching impossible because > the > > > database needs to see all the reads (and writes) in a Tx to be able to > > > ensure serializable guarantees. If a cache prevents a read from > hitting the > > > database, that read (e.g. storage config) will not be a factor in > > > serializability. > > > > > > This connects to my earlier question about the in-memory cache and JDBC > > > (title: Usefulness of InMemoryEntityCache) > > > > > > Are different approaches possible with JDBC? Probably. What is > involved? > > > This depends on the SPI contract, I think. > > > > > > > 2. How do we protect the data used by validation? [...] > > > > > > The point mentioned by Robert previously IIRC, is that we need to > ground > > > all runtime decisions to a particular state of the catalog. > > > > > > If at commit time the state of the catalog in the database shifts from > what > > > was used during validation, the request has to restart from scratch. > > > > > > Current JDBC Persistence does this only for relationships between > entities > > > and grants and between versions of the same entity. > > > > > > Relationships between table locations and storage config are not > actualized > > > for the purpose of state consistency tracking. > > > > > > From my POV we need to begin by formulating this Persistence state in > the > > > Persistence SPI terms. Then, think how we can implement that for JDBC. > > > > > > NoSQL Persistence already has a per-catalog state (not granular), so > wiring > > > the new SPI to NoSQL should not be too hard, I hope. > > > > > > The state in the SPI does not have to be global to the catalog (as in > > > NoSQL), but it needs to be able to express the idea of which elements > were > > > engaged during request processing so serve as input to Persistence > impl. > > > for cosistency guarantees. If Persistence caches data in memory (for > > > example), it will have to do so in a way that does not violate > consistency > > > expectations. > > > > > > From my POV we could begin by defining a "Persistence Session" concept. > > > Each request is 1:1 with a session. All reads and writes go through the > > > session. The next step would be the JDBC impl., which probably needs > some > > > brainstorming... but I wonder if we could reach a consensus at a high > level > > > first. > > > > > > WDYT? > > > > > > Thanks, > > > Dmitri. > > > > > > On Tue, Sep 22, 2026 at 5:55 PM EJ Wang < > [email protected]> > > > wrote: > > > > > > > Thanks for the recap, Dmitri. I'd like to follow up on the three > points I > > > > raised during the sync. We didn't have much time to discuss them, so > it > > > > would help to confirm these before choosing an SPI design. > > > > > > > > 1. Which backend assumptions should we design for? > > > > > > > > Within one Polaris server instance, do we assume entities, grants and > > > > secrets all use the same database backend, or must we support mixing > > > > backends across them? > > > > Must every supported persistence implementation provide transactional > > > > guarantees? Does the database itself need to provide transactions, > or can > > > > Polaris build that capability on top of simpler database operations? > > > > > > > > I'm happy to work with a homogeneous assumption if that's the > intended > > > > scope. Existing NoSQL illustrates the second question: it can publish > > > > changes to multiple objects under one reference using a single > > > conditional > > > > pointer update. > > > > > > > > 2. How do we protect the data used by validation? > > > > > > > > Your storage-configuration and location-overlap examples make the > problem > > > > concrete. Consider these races: > > > > > > > > A table creation passes validation against the catalog's allowed > > > locations, > > > > then another request changes that configuration before the table is > > > > committed. > > > > Two table-creation requests each check for overlapping locations, > both > > > see > > > > no overlap, and then create tables at conflicting locations. > > > > > > > > Checking only the record being written cannot cover these > dependencies. > > > The > > > > first also depends on the catalog configuration. The second depends > on > > > the > > > > absence of another overlapping table. > > > > > > > > Could the persistence layer protect these reads together with the > writes, > > > > using backend-internal versions/tokens or database isolation? Shared > code > > > > would still need to identify the required reads and checks. Could we > keep > > > > the mechanism for protecting them inside each backend? > > > > > > > > 3. Business rules are duplicated across backend implementations > > > > > > > > Today, the TreeMap > > > > < > > > > > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481 > > > > > > > > > and JDBC > > > > < > > > > > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046 > > > > > > > > > implementations each load a secret, check its principal, apply the > > > > rotation/reset steps and write it back. > > > > > > > > Changing these business rules therefore requires keeping multiple > > > > implementations in sync. A fix applied to one backend can be missed > in > > > > another, causing the same Polaris operation to behave differently > > > depending > > > > on the database. > > > > > > > > My proposal is to implement each business workflow once, against a > common > > > > set of storage operations with defined consistency and failure > semantics. > > > > The shared workflow should not need to know whether those operations > are > > > > implemented through JDBC transactions, NoSQL reference CAS or another > > > > mechanism. Each backend would implement the same storage contract, > > > keeping > > > > its execution details internal. A new database backend could then > reuse > > > the > > > > existing business workflows. > > > > > > > > Do these seem like reasonable starting points? We can then evaluate > the > > > SPI > > > > against the original failure cases on both JDBC and existing NoSQL. > > > > > > > > -ej > > > > > > > > On Fri, Sep 18, 2026 at 12:10 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Side note: I just came across [5541], which shows a case where a > failed > > > > > request leaves persisted side effects. > > > > > > > > > > This can be seen as a coding mistake, of course. However, this is > about > > > > > executing one request as a single, atomic unit. This is something > that > > > > > probably needs foundational support and coding patterns in Polaris > > > code. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > [5541] https://github.com/apache/polaris/pull/5541 > > > > > > > > > > On Fri, Sep 18, 2026 at 3:02 PM Dmitri Bourlatchkov < > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > Here's a recap of the Community Sync discussion as I remember > it. I'm > > > > > sure > > > > > > my recollection is not complete, so please add your thoughts to > this > > > > > thread. > > > > > > > > > > > > * Polaris traditionally relies on entity version number to ensure > > > > > > consistency of changes. > > > > > > > > > > > > This works well for internal RBAC grants, which also have version > > > > numbers > > > > > > cross-referenced at the Persistence layer. > > > > > > > > > > > > I do not think this works in more general cases like validating > > > > locations > > > > > > wrt catalog-level storage configuration. > > > > > > > > > > > > Another difficult case involves location overlaps between > concurrent > > > > > table > > > > > > creations. > > > > > > > > > > > > * We talked about how Persistence calls related to RDBMS > transactions > > > > (in > > > > > > the JDBC persistence case). > > > > > > > > > > > > There were some concerns raised about whether we need to bind > > > > Persistence > > > > > > API to transactions explicitly. > > > > > > > > > > > > I think it is a valid concern. At the same time, the Persistence > API > > > > > > should be clear about consistency guarantees across all backend > > > > > > implementations. I think this needs more attention now that the > > > matter > > > > of > > > > > > milti-entioty changes became prominent in [5035]. > > > > > > > > > > > > I personally believe that RDBMS-based Persistence in Polaris has > to > > > use > > > > > > Serializable transaction isolation. Consequently, the core code > needs > > > > to > > > > > > provide mechanisms to allow the Persistence implementation to > > > properly > > > > > > connect reads and writes from API requests to JDBC > transactions... > > > > > whether > > > > > > we use "transation" as an explicit terms in the SPI or not. > > > > > > > > > > > > Note that currently, the same API request performs reads and > writes > > > in > > > > > > _separate_ RDBMS transactions (even separate connections). > > > > > > > > > > > > * We did not discuss Persistence SPI implications in the call, > but > > > > > > currently each Persistence call is assumed to be one distinct and > > > > atomic > > > > > > change (please correct me if I'm wrong). > > > > > > > > > > > > Consequently, supporting new use cases involves introducing new > SPI > > > > > > methods with many parameters. This can be seen in PR [5035]. This > > > tends > > > > > to > > > > > > overcomplicate the SPI. > > > > > > > > > > > > Ideally, I think the SPI should be simplified to deal with > > > multi-entity > > > > > > and single-entity changes in a coherent manner to avoid any > ambiguity > > > > > about > > > > > > which method the caller should use. > > > > > > > > > > > > [5035] https://github.com/apache/polaris/pull/5035 > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > >> Heads up: This discussion is on the Community Sync call agenda > for > > > > Sept > > > > > >> 17. > > > > > >> > > > > > >> Interested parties, please plan to attend, if possible :) > > > > > >> > > > > > >> Cheers, > > > > > >> Dmitri. > > > > > >> > > > > > >> On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov < > > > [email protected]> > > > > > >> wrote: > > > > > >> > > > > > >>> Hi JB, > > > > > >>> > > > > > >>> I've added an agenda item to the next community sync call for > this > > > > > (Sept > > > > > >>> 17, 2026). > > > > > >>> > > > > > >>> Re: key points: did you mean a real GH discussion or dev email > > > (this > > > > > >>> thread)? Just double checking :) I'm fine with either approach. > > > > > >>> > > > > > >>> Cheers, > > > > > >>> Dmitri. > > > > > >>> > > > > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré < > > > > [email protected]> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Hi Dmitri, > > > > > >>>> > > > > > >>>> I agree on the need to align on these points, though I'm not > > > > entirely > > > > > >>>> sure a dedicated meeting is necessary. Let's start by using > some > > > > time > > > > > >>>> during the next community meeting to discuss it. > > > > > >>>> > > > > > >>>> In the meantime, what if we outline the key points in a GitHub > > > > > >>>> Discussion first? We can then use that as a reference during > the > > > > > >>>> meeting. > > > > > >>>> > > > > > >>>> Regards, > > > > > >>>> JB > > > > > >>>> > > > > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < > > > > [email protected] > > > > > > > > > > > >>>> wrote: > > > > > >>>> > > > > > > >>>> > Hi All, > > > > > >>>> > > > > > > >>>> > My impression from this thread is that it might be time for > a > > > call > > > > > to > > > > > >>>> > discuss all the related issues and try to arrive at a shared > > > > > >>>> implementation > > > > > >>>> > plan. > > > > > >>>> > > > > > > >>>> > I know meetings are not ideal, but in this case it might be > > > > > >>>> beneficial as a > > > > > >>>> > means for achieving a common understanding of the set of > > > problems > > > > > and > > > > > >>>> > priorities related to this thread. A dedicated metrics > meeting > > > > > worked > > > > > >>>> well > > > > > >>>> > from my POV. > > > > > >>>> > > > > > > >>>> > Allocating a time slot in the next community meeting might > be an > > > > > >>>> option, > > > > > >>>> > although I think we might need the full hour in this case. > > > > > >>>> > > > > > > >>>> > WDYT? > > > > > >>>> > > > > > > >>>> > Thanks, > > > > > >>>> > Dmitri. > > > > > >>>> > > > > > > >>>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S < > > > > > >>>> [email protected]> > > > > > >>>> > wrote: > > > > > >>>> > > > > > > >>>> > > Hi all, > > > > > >>>> > > > > > > > >>>> > > I rewrote https://github.com/apache/polaris/pull/5035, > > > > following > > > > > >>>> the > > > > > >>>> > > discussion to focus only on the SPI foundation. > > > > > >>>> > > > > > > > >>>> > > What is in the PR now: > > > > > >>>> > > > > > > > >>>> > > - A written manager-level consistency contract in > > > > > >>>> > > > site/content/in-dev/unreleased/persistence-consistency.md. > > > > > >>>> > > - CAS-aware EntityMutation / GrantMutation records. > > > > > >>>> > > - MetaStoreChangeSet and > BasePersistence#commitChangeSet, > > > > with > > > > > >>>> backends > > > > > >>>> > > opting in via supportsAtomicMixedCommit(). > > > > > >>>> > > - commitChangeSet implementations for the in-memory > TreeMap > > > > > >>>> backend and > > > > > >>>> > > JDBC. > > > > > >>>> > > - AtomicOperationMetaStoreManager and > > > > > >>>> TransactionalMetaStoreManagerImpl > > > > > >>>> > > grant/revoke paths now build one change set per > operation, > > > > > >>>> falling back > > > > > >>>> > > to > > > > > >>>> > > individual operations when the backend does not support > > > mixed > > > > > >>>> commits. > > > > > >>>> > > - Tests for the change-set API and TreeMap atomic > commits. > > > > > >>>> > > > > > > > >>>> > > I intentionally left out: > > > > > >>>> > > > > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > > > > >>>> > > - Refactoring createCatalog, dropEntity, and > renameEntity > > > to > > > > > use > > > > > >>>> change > > > > > >>>> > > sets. > > > > > >>>> > > - NoSQL support for commitChangeSet. > > > > > >>>> > > > > > > > >>>> > > I am treating this as Phase 1 so the contract and the SPI > > > > > primitive > > > > > >>>> can be > > > > > >>>> > > reviewed before the larger operation migrations land. > > > > > >>>> > > > > > > > >>>> > > Could you please take a look? In particular, I would like > to > > > > know > > > > > >>>> whether > > > > > >>>> > > the contract captures the consensus so far, or if it > needs to > > > > > >>>> address retry > > > > > >>>> > > / external-work coordination before we merge this > foundation. > > > > > >>>> > > > > > > > >>>> > > Thanks, > > > > > >>>> > > Prithvi S > > > > > >>>> > > > > > > > >>>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov < > > > > > >>>> [email protected]> > > > > > >>>> > > wrote: > > > > > >>>> > > > > > > > >>>> > > > Hi Robert, > > > > > >>>> > > > > > > > > >>>> > > > I agree that the client-visible operation (e.g. REST API > > > > > request) > > > > > >>>> > > consists > > > > > >>>> > > > of many distinct phases. The database / persistence > changes > > > > are > > > > > >>>> just one > > > > > >>>> > > of > > > > > >>>> > > > these phases. STS (as an example) is another. My point > about > > > > > >>>> transactions > > > > > >>>> > > > in JDBC Persistence applies to the former (database > changes) > > > > > >>>> phase - one > > > > > >>>> > > > retry attempt there should ideally involve exactly one > > > > > >>>> transaction. > > > > > >>>> > > > > > > > > >>>> > > > We certainly need logic inside Polaris Servers to to > > > > coordinate > > > > > >>>> requests > > > > > >>>> > > to > > > > > >>>> > > > various external systems depending on outcomes from > previous > > > > > >>>> request > > > > > >>>> > > > processing phases. > > > > > >>>> > > > > > > > > >>>> > > > I also agree that validation based on database state > should > > > be > > > > > >>>> repeated > > > > > >>>> > > > from scratch if we retry the database changes. > > > > > >>>> > > > > > > > > >>>> > > > However, internal RBAC authorization naturally has to > happen > > > > > >>>> inside the > > > > > >>>> > > > database update phase (read-check-write). IIRC, that is > part > > > > of > > > > > >>>> the > > > > > >>>> > > > implicit consistency guarantees, which were discussed > during > > > > > early > > > > > >>>> > > project > > > > > >>>> > > > months (but never got written down in full clarity, > > > > > >>>> unfortunately). If > > > > > >>>> > > the > > > > > >>>> > > > Authorization check is inside that phase for internal > RBAC, > > > I > > > > > >>>> suppose it > > > > > >>>> > > > will be there for other Authorizers too and that can > involve > > > > > >>>> external > > > > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly expect > > > > > >>>> authorization to be > > > > > >>>> > > > efficient, but I'm not sure how quick it can be in > practice. > > > > > >>>> There's > > > > > >>>> > > > certainly potential for delays in some (perhaps > infrequent) > > > > > cases. > > > > > >>>> > > > > > > > > >>>> > > > Cheers, > > > > > >>>> > > > Dmitri. > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp < > > > [email protected]> > > > > > >>>> wrote: > > > > > >>>> > > > > > > > > >>>> > > > > Hi Dmitri, > > > > > >>>> > > > > > > > > > >>>> > > > > I agree that one logical operation needs a consistent > view > > > > of > > > > > >>>> the whole > > > > > >>>> > > > > backend state. > > > > > >>>> > > > > What worries me is treating one long JDBC transaction > as > > > the > > > > > >>>> boundary > > > > > >>>> > > of > > > > > >>>> > > > > that operation and then retrying the complete request > if > > > the > > > > > >>>> commit > > > > > >>>> > > > fails. > > > > > >>>> > > > > > > > > > >>>> > > > > The request can also write metadata to object storage > or > > > > call > > > > > >>>> STS. > > > > > >>>> > > > > Each system can leave us with an outcome whose > certainty > > > we > > > > > >>>> cannot > > > > > >>>> > > know. > > > > > >>>> > > > > The database may have committed before the connection > was > > > > > lost, > > > > > >>>> an > > > > > >>>> > > > > object-store write may have succeeded before a > timeout, or > > > > STS > > > > > >>>> may have > > > > > >>>> > > > > issued credentials before its response was lost. > > > > > >>>> > > > > A database rollback cannot undo any of those effects, > and > > > > > >>>> retrying the > > > > > >>>> > > > > whole request may repeat them. > > > > > >>>> > > > > > > > > > >>>> > > > > So I think the contract has to separate the overall > > > > operation > > > > > >>>> from each > > > > > >>>> > > > > attempt to commit it. > > > > > >>>> > > > > Each database or backend commit attempt should stay > short. > > > > > >>>> > > > > When we retry, validation and authorization need to > use > > > the > > > > > >>>> state for > > > > > >>>> > > > that > > > > > >>>> > > > > new attempt, and we need clear rules for which > external > > > work > > > > > >>>> can be > > > > > >>>> > > > reused, > > > > > >>>> > > > > repeated, cleaned up, or reconciled after an uncertain > > > > > outcome. > > > > > >>>> > > > > > > > > > >>>> > > > > Some contextual state could be useful for carrying > that > > > > state. > > > > > >>>> > > > > But the JDBC connection and transaction should be an > > > > > >>>> implementation > > > > > >>>> > > > detail > > > > > >>>> > > > > inside it, not the boundary of the complete operation. > > > > > >>>> > > > > > > > > > >>>> > > > > Cheers, > > > > > >>>> > > > > Robert > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > > > > >>>> [email protected] > > > > > >>>> > > > > > > > > >>>> > > > > wrote: > > > > > >>>> > > > > > > > > > >>>> > > > > > Hi All, > > > > > >>>> > > > > > > > > > > >>>> > > > > > The idea of a backend-agnostic change-set primitive > > > sounds > > > > > >>>> good. > > > > > >>>> > > > > > > > > > > >>>> > > > > > However, I am not sure it is sufficient for all the > use > > > > > cases > > > > > >>>> we > > > > > >>>> > > > touched > > > > > >>>> > > > > > here. More specifically, the state read by > validation > > > code > > > > > is > > > > > >>>> not > > > > > >>>> > > > > > necessarily reflected in the change set: > > > > > >>>> > > > > > > > > > > >>>> > > > > > * Unchanged entities might be considered by > validation, > > > > but > > > > > >>>> changed > > > > > >>>> > > in > > > > > >>>> > > > a > > > > > >>>> > > > > > parallel request. > > > > > >>>> > > > > > * Even for changed entities the state read by > validation > > > > may > > > > > >>>> differ > > > > > >>>> > > > from > > > > > >>>> > > > > > the state being altered. Some validation code talks > to > > > the > > > > > >>>> MetaStore > > > > > >>>> > > > > > directly, outside of the data produced by the > Resolver. > > > > > >>>> > > > > > > > > > > >>>> > > > > > I do not think Polaris offers any explicit > mechanisms > > > > (ATM) > > > > > >>>> to ensure > > > > > >>>> > > > > > consistency between these reads and subsequent > writes. > > > > > >>>> > > > > > > > > > > >>>> > > > > > As far as JDBC goes, running an overarching > Transaction > > > > > >>>> across all > > > > > >>>> > > > reads > > > > > >>>> > > > > > and writes in the same request, at the SERIALIZABLE > > > > > isolation > > > > > >>>> level > > > > > >>>> > > in > > > > > >>>> > > > > the > > > > > >>>> > > > > > backing RDBMS could solve the problem, I think. > > > > > >>>> > > > > > > > > > > >>>> > > > > > Indeed, such a transaction might be long to > accommodate > > > > > calls > > > > > >>>> made by > > > > > >>>> > > > > > Polaris to external storage, etc. However, is that a > > > > > problem? > > > > > >>>> I think > > > > > >>>> > > > the > > > > > >>>> > > > > > alternative is for the JDBC Persistence impl. to > > > > "manually" > > > > > >>>> track all > > > > > >>>> > > > > reads > > > > > >>>> > > > > > under the same request and redo them in the small > > > > > transaction > > > > > >>>> that > > > > > >>>> > > > > persists > > > > > >>>> > > > > > the writes. This will also add RDBMS overhead and > > > require > > > > > >>>> complex > > > > > >>>> > > code > > > > > >>>> > > > in > > > > > >>>> > > > > > Polaris to handle the data properly. > > > > > >>>> > > > > > > > > > > >>>> > > > > > Tracking the request-wide transaction can be done > only > > > for > > > > > >>>> JDBC > > > > > >>>> > > without > > > > > >>>> > > > > > leaking "transaction" concepts to the NoSQL > > > Persistence, I > > > > > >>>> think. > > > > > >>>> > > NoSQL > > > > > >>>> > > > > > will use other mechanisms to ensure read/write > > > > consistency. > > > > > >>>> > > > > > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch in > this > > > > > >>>> discussion), > > > > > >>>> > > > I'd > > > > > >>>> > > > > > like to propose this approach: > > > > > >>>> > > > > > > > > > > >>>> > > > > > * Each request establishes a "Data Context" > > > > > >>>> > > > > > - In JDBC the Data Context corresponds to a JDBC > > > > > Connection > > > > > >>>> + Tx > > > > > >>>> > > > > > - In NoSQL the Data Context tracks one or more > > > reference > > > > > >>>> hashes > > > > > >>>> > > > > > * All Persistence access in the same request goes > > > through > > > > > the > > > > > >>>> same > > > > > >>>> > > Data > > > > > >>>> > > > > > Context > > > > > >>>> > > > > > * All Persistence changes are committed once at the > end > > > of > > > > > the > > > > > >>>> > > request > > > > > >>>> > > > > > - Not all changes have to be globally atomic. For > > > > example, > > > > > >>>> changes > > > > > >>>> > > in > > > > > >>>> > > > > > Catalogs A and B do not have to be atomic with > respect > > > to > > > > > >>>> each other. > > > > > >>>> > > > We > > > > > >>>> > > > > > can go deeper into this later. This is relevant to > > > NoSQL. > > > > > >>>> > > > > > - Transactional backends like JDBC can, of course, > > > > choose > > > > > >>>> to make > > > > > >>>> > > all > > > > > >>>> > > > > > changes globally atomic. > > > > > >>>> > > > > > * A commit can fail in two main ways: > > > > > >>>> > > > > > - A retriable failure like an optimistic lock > error or > > > > Tx > > > > > >>>> > > > > serializability > > > > > >>>> > > > > > error > > > > > >>>> > > > > > - A non-triable logical error (e.g. entity not > found) > > > > > >>>> > > > > > * On a retriable error the whole request is > re-attempted > > > > (as > > > > > >>>> if > > > > > >>>> > > > > resubmitted > > > > > >>>> > > > > > by a client) a few times (configurable timeout). > > > > > >>>> > > > > > > > > > > >>>> > > > > > WDYT? > > > > > >>>> > > > > > > > > > > >>>> > > > > > Cheers, > > > > > >>>> > > > > > Dmitri. > > > > > >>>> > > > > > > > > > > >>>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste > Onofré < > > > > > >>>> > > [email protected] > > > > > >>>> > > > > > > > > > >>>> > > > > > wrote: > > > > > >>>> > > > > > > > > > > >>>> > > > > > > Hi all > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > I think Robert has a good point. > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > If the atomicity guarantee lives only in > > > > BasePersistence, > > > > > >>>> then the > > > > > >>>> > > > > > > manager contract can't tell a caller whether the > state > > > > it > > > > > >>>> read for > > > > > >>>> > > > > > > validation/authorization/credential-vending is the > > > same > > > > > >>>> state that > > > > > >>>> > > > > > > eventually commits. That's the actual bug class > behind > > > > the > > > > > >>>> JDBC > > > > > >>>> > > > > > > symptoms (not any single operation being > non-atomic, > > > but > > > > > the > > > > > >>>> > > contract > > > > > >>>> > > > > > > being silent about it. Every operation-specific > fix > > > > (like > > > > > >>>> #4939, > > > > > >>>> > > > #5035 > > > > > >>>> > > > > > > or #5095) re-answers this question locally and it > > > keeps > > > > > >>>> recurring. > > > > > >>>> > > So > > > > > >>>> > > > > > > the deliverable should start with a written > > > consistency > > > > > >>>> contract at > > > > > >>>> > > > > > > the manager level. > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > I'm not sure migrating everyone to > > > > > >>>> > > TransactionalMetaStoreManagerImpl > > > > > >>>> > > > > > > is a good idea. It would tie the logical change > set > > > to a > > > > > DB > > > > > >>>> > > > > > > transaction spanning the REST request. It means: > > > > > >>>> > > > > > > - it holds a durable transaction open across slow > > > > external > > > > > >>>> work > > > > > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > > > > >>>> > > > > > > - it doesn't map to NoSQL > > > > > >>>> > > > > > > - It wraps single-row updates in > > > runWiithinTransaction, > > > > > >>>> which is a > > > > > >>>> > > > > > overhead > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > So, I think the transactional manager isn't a > portable > > > > > >>>> target. It's > > > > > >>>> > > > > > > "only" one backend's strategy. > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > I think Privthi's approach is right. A > > > backend-agnostic > > > > > >>>> change-set > > > > > >>>> > > > > > > primitive with a documented fallback is the > correct > > > > shape. > > > > > >>>> There is > > > > > >>>> > > > > > > one caveat: without carrying the original entitiy > (not > > > > > just > > > > > >>>> the new > > > > > >>>> > > > > > > one) the change set can't express optimistic > > > > concurrency. > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > I propose the following multi-steps approach: > > > > > >>>> > > > > > > 1. We write the manager-level consistency > contract as > > > > > >>>> Robert asked. > > > > > >>>> > > > It > > > > > >>>> > > > > > > should include the explicit statement that a > logical > > > > > change > > > > > >>>> set is > > > > > >>>> > > > not > > > > > >>>> > > > > > > a request-scope DB transaction. > > > > > >>>> > > > > > > 2. We make Compare And Swap > (optimistic-concurrency > > > > > pattern) > > > > > >>>> > > baseline > > > > > >>>> > > > > > > first-class in EntityMutation before merging the > SPI > > > > > >>>> > > > > > > 3. Then we refactor > > > > createCatalog/dropEntity/renameEntity. > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > Thoughts? > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > Regards > > > > > >>>> > > > > > > JB > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > > > > >>>> [email protected]> > > > > > >>>> > > wrote: > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > Hi all, > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > Yufei's clarification about where the atomicity > > > > > guarantee > > > > > >>>> is > > > > > >>>> > > > defined > > > > > >>>> > > > > > > seems > > > > > >>>> > > > > > > > important. > > > > > >>>> > > > > > > > If it is a property of the lower-level > > > BasePersistence > > > > > >>>> contract > > > > > >>>> > > > > rather > > > > > >>>> > > > > > > than > > > > > >>>> > > > > > > > the general PolarisMetaStoreManager contract, > the > > > > > general > > > > > >>>> > > contract > > > > > >>>> > > > > does > > > > > >>>> > > > > > > not > > > > > >>>> > > > > > > > tell callers whether the state used for > validation, > > > > > >>>> > > authorization, > > > > > >>>> > > > or > > > > > >>>> > > > > > > > credential vending is consistent with the change > > > that > > > > > >>>> eventually > > > > > >>>> > > > > > commits. > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > The current PRs suggest that operation-specific > > > > > >>>> multi-object > > > > > >>>> > > > methods > > > > > >>>> > > > > > can > > > > > >>>> > > > > > > > fix individual cases, while leaving the same > > > contract > > > > > >>>> question to > > > > > >>>> > > > > recur > > > > > >>>> > > > > > > for > > > > > >>>> > > > > > > > each new case. > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > I would also avoid defining a logical change > set as > > > a > > > > > >>>> database > > > > > >>>> > > > > > > transaction > > > > > >>>> > > > > > > > around an entire REST request. > > > > > >>>> > > > > > > > That would tie the contract to the backend and > > > > database, > > > > > >>>> and > > > > > >>>> > > could > > > > > >>>> > > > > keep > > > > > >>>> > > > > > > the > > > > > >>>> > > > > > > > durable attempt open across slow external work. > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > So is the choice really between the two current > > > > manager > > > > > >>>> > > > > > implementations, > > > > > >>>> > > > > > > or > > > > > >>>> > > > > > > > do we first need to revisit the boundary and > > > > guarantees > > > > > >>>> exposed > > > > > >>>> > > to > > > > > >>>> > > > > > their > > > > > >>>> > > > > > > > callers? > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > Cheers, > > > > > >>>> > > > > > > > Robert > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri > > > Bourlatchkov < > > > > > >>>> > > > > [email protected] > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > wrote: > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > Hi Yufei, > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > Thanks for the link. I stand corrected. The > "one > > > > > atomic > > > > > >>>> change > > > > > >>>> > > > per > > > > > >>>> > > > > > > method" > > > > > >>>> > > > > > > > > contract as defined in javadoc does apply > only to > > > > > >>>> > > BasePersistence > > > > > >>>> > > > > > > > > and AtomicOperationMetaStoreManager (which > > > delegates > > > > > to > > > > > >>>> > > > > > > BasePersistence). > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > Note that all non-test Persistence > implementations > > > > in > > > > > >>>> the > > > > > >>>> > > Polaris > > > > > >>>> > > > > > > codebase > > > > > >>>> > > > > > > > > extend those classes (which is probably why I > was > > > > > >>>> confused > > > > > >>>> > > about > > > > > >>>> > > > > > > atomicity > > > > > >>>> > > > > > > > > expectations). > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > However, this creates a gap in the > Persistence SPI > > > > > >>>> > > specification. > > > > > >>>> > > > > If > > > > > >>>> > > > > > > > > other PolarisMetaStoreManager implementations > do > > > not > > > > > >>>> have to > > > > > >>>> > > > comply > > > > > >>>> > > > > > > with > > > > > >>>> > > > > > > > > this principle, it will create a conceptual > > > > difficulty > > > > > >>>> at call > > > > > >>>> > > > > sites. > > > > > >>>> > > > > > > How > > > > > >>>> > > > > > > > > can PolarisMetaStoreManager callers reason > about > > > > > >>>> consistency > > > > > >>>> > > and > > > > > >>>> > > > > > > durability > > > > > >>>> > > > > > > > > behaviours in general? > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > I believe we need to address that as part of > this > > > > > >>>> discussion. > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > Cheers, > > > > > >>>> > > > > > > > > Dmitri. > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu < > > > > > >>>> [email protected] > > > > > >>>> > > > > > > > > >>>> > > > > > wrote: > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined > with > > > the > > > > > >>>> idea that > > > > > >>>> > > one > > > > > >>>> > > > > > > method > > > > > >>>> > > > > > > > > call > > > > > >>>> > > > > > > > > > means one atomic change. > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > If the "MetaStore SPI" refers to the > interface > > > > > >>>> > > > > > > PolarisMetaStoreManager, I > > > > > >>>> > > > > > > > > > don't think we've ever state each method to > be > > > > > >>>> atomic. We did > > > > > >>>> > > > > > clarify > > > > > >>>> > > > > > > > > > atomicity[1] in the interface > BasePersistence > > > > > though. > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > 1. > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48 > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > Yufei > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri > > > > > Bourlatchkov < > > > > > >>>> > > > > > > [email protected]> > > > > > >>>> > > > > > > > > > wrote: > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > Hi Yufei, > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > I agree that JDBC transactions must be > handled > > > > > more > > > > > >>>> > > > explicitly. > > > > > >>>> > > > > > > > > However, > > > > > >>>> > > > > > > > > > > I'm not sure that simply moving to > > > > > >>>> > > > > > > TransactionalMetaStoreManagerImpl is > > > > > >>>> > > > > > > > > > > sufficient. > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > The MetaStore SPI is currently defined > with > > > the > > > > > >>>> idea that > > > > > >>>> > > one > > > > > >>>> > > > > > > method > > > > > >>>> > > > > > > > > call > > > > > >>>> > > > > > > > > > > means one atomic change [1]. The > > > "transactional" > > > > > >>>> MetaStore > > > > > >>>> > > > > impl. > > > > > >>>> > > > > > is > > > > > >>>> > > > > > > > > but a > > > > > >>>> > > > > > > > > > > sub-case of that. It cannot alter the > > > high-level > > > > > >>>> contract. > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > We could add SPI methods having multiple > > > object > > > > > >>>> parameters > > > > > >>>> > > to > > > > > >>>> > > > > > > represent > > > > > >>>> > > > > > > > > > > grouped changes, but I am not sure it > will be > > > a > > > > > >>>> sound > > > > > >>>> > > design. > > > > > >>>> > > > > > This > > > > > >>>> > > > > > > will > > > > > >>>> > > > > > > > > > > bloat the interface surfaces and require > extra > > > > > >>>> impl. effort > > > > > >>>> > > > for > > > > > >>>> > > > > > > each > > > > > >>>> > > > > > > > > > > backend type. More importantly, adding > > > multi-arg > > > > > >>>> change > > > > > >>>> > > > methods > > > > > >>>> > > > > > > still > > > > > >>>> > > > > > > > > > won't > > > > > >>>> > > > > > > > > > > address the problem of reads being > consistent > > > > with > > > > > >>>> writes, > > > > > >>>> > > > > > because > > > > > >>>> > > > > > > each > > > > > >>>> > > > > > > > > > > method call will still be independent > > > regarding > > > > > the > > > > > >>>> data > > > > > >>>> > > > stored > > > > > >>>> > > > > > in > > > > > >>>> > > > > > > the > > > > > >>>> > > > > > > > > > > database. > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > I tend to think we need to introduce a > "change > > > > > set" > > > > > >>>> or > > > > > >>>> > > > "atomic > > > > > >>>> > > > > > > batch" > > > > > >>>> > > > > > > > > > > concept to core Persistence and associate > each > > > > > REST > > > > > >>>> API > > > > > >>>> > > > request > > > > > >>>> > > > > > > with > > > > > >>>> > > > > > > > > one > > > > > >>>> > > > > > > > > > > such change set, which will be committed > (or > > > > > rolled > > > > > >>>> back) > > > > > >>>> > > at > > > > > >>>> > > > > the > > > > > >>>> > > > > > > end of > > > > > >>>> > > > > > > > > > the > > > > > >>>> > > > > > > > > > > request. I believe Ayush mentioned a > similar > > > > > >>>> concept in PR > > > > > >>>> > > > 4939 > > > > > >>>> > > > > > > [2]. In > > > > > >>>> > > > > > > > > > > JDBC each change set will naturally be > > > > associated > > > > > >>>> with an > > > > > >>>> > > > RDBMS > > > > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, each > atomic > > > > > >>>> change set > > > > > >>>> > > > will > > > > > >>>> > > > > be > > > > > >>>> > > > > > > > > > > associated with one CAS operation on the > > > > > underlying > > > > > >>>> > > database. > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > [1] > > > > > >>>> > > > > > > > > > > > >>>> > https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > > > > >>>> > > > > > > > > > > > >>>> > > > https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > > > > > >>>> > > > > > > > > > > Dmitri. > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > > > > >>>> > > > [email protected] > > > > > >>>> > > > > > > > > > > >>>> > > > > > > wrote: > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for raising this, Dmitri. > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > These are valid concerns, and they were > > > > already > > > > > >>>> > > recognized > > > > > >>>> > > > > when > > > > > >>>> > > > > > > we > > > > > >>>> > > > > > > > > > > > introduced JDBC persistence to Polaris. > At > > > > that > > > > > >>>> time, we > > > > > >>>> > > > > chose > > > > > >>>> > > > > > > to use > > > > > >>>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the > JDBC > > > > due > > > > > >>>> to the > > > > > >>>> > > > > > > simplicity. I > > > > > >>>> > > > > > > > > > > think > > > > > >>>> > > > > > > > > > > > most of the issues mentioned here can > > > already > > > > be > > > > > >>>> > > addressed > > > > > >>>> > > > by > > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl. > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > For example, rename is already wrapped > in a > > > > > >>>> transaction > > > > > >>>> > > in > > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. > > > > > Similarly, > > > > > >>>> catalog > > > > > >>>> > > > > > > creation, > > > > > >>>> > > > > > > > > > which > > > > > >>>> > > > > > > > > > > > involves reading and creating multiple > > > > objects, > > > > > >>>> is also > > > > > >>>> > > > > > executed > > > > > >>>> > > > > > > > > > within a > > > > > >>>> > > > > > > > > > > > transaction [2]. > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > I see two possible directions: > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Modify > AtomicOperationMetaStoreManager > > > > > >>>> together with > > > > > >>>> > > the > > > > > >>>> > > > > > > > > persistence > > > > > >>>> > > > > > > > > > > > backends (such as JDBC) to provide > the > > > > > required > > > > > >>>> > > > > consistency > > > > > >>>> > > > > > > > > > guarantees > > > > > >>>> > > > > > > > > > > > for > > > > > >>>> > > > > > > > > > > > specific operations, similar to what > > > > > >>>> > > > > > > > > > TransactionalMetaStoreManagerImpl > > > > > >>>> > > > > > > > > > > > does. > > > > > >>>> > > > > > > > > > > > 2. > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Migrate the persistence backends > (such as > > > > > >>>> JDBC) to use > > > > > >>>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl > > > directly. > > > > > We > > > > > >>>> may > > > > > >>>> > > have > > > > > >>>> > > > to > > > > > >>>> > > > > > > deal > > > > > >>>> > > > > > > > > with > > > > > >>>> > > > > > > > > > > > transactional semantic mismatches > across > > > > > >>>> different > > > > > >>>> > > > > > persistence > > > > > >>>> > > > > > > > > > > backends. > > > > > >>>> > > > > > > > > > > > For example, we would likely avoid > using > > > > > JDBC's > > > > > >>>> > > > > > > > > > `runWithinTransaction` > > > > > >>>> > > > > > > > > > > > for > > > > > >>>> > > > > > > > > > > > single row updates, which adds > additional > > > > > >>>> overhead and > > > > > >>>> > > > > > > complexity > > > > > >>>> > > > > > > > > > > > without > > > > > >>>> > > > > > > > > > > > benefits. > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > References: > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 > > > > > >>>> > > > > > > > > > > > 2. > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > > > > >>>> Bourlatchkov < > > > > > >>>> > > > > > > > > [email protected]> > > > > > >>>> > > > > > > > > > > > wrote: > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Hi all, > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Ayush and Prithvi recently > contributed a > > > > > couple > > > > > >>>> of > > > > > >>>> > > > > > interesting > > > > > >>>> > > > > > > PRs: > > > > > >>>> > > > > > > > > > > > > [4939], [5035]. > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > It looks like people are starting to > > > > encounter > > > > > >>>> > > > consistency > > > > > >>>> > > > > > > issues > > > > > >>>> > > > > > > > > in > > > > > >>>> > > > > > > > > > > > > JDBC persistence. > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight into > the > > > > > >>>> underlying > > > > > >>>> > > > > issues. > > > > > >>>> > > > > > > They > > > > > >>>> > > > > > > > > > offer > > > > > >>>> > > > > > > > > > > > > incremental fixes that can work. > However, > > > I > > > > > >>>> believe it > > > > > >>>> > > is > > > > > >>>> > > > > > time > > > > > >>>> > > > > > > for > > > > > >>>> > > > > > > > > > the > > > > > >>>> > > > > > > > > > > > > Polaris community to review and > improve > > > this > > > > > >>>> area of > > > > > >>>> > > the > > > > > >>>> > > > > > > codebase > > > > > >>>> > > > > > > > > > > > > holistically. > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding a solution > that > > > can > > > > be > > > > > >>>> applied > > > > > >>>> > > to > > > > > >>>> > > > > all > > > > > >>>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, > > > > NoSQL) > > > > > >>>> and > > > > > >>>> > > > addresses > > > > > >>>> > > > > > > these > > > > > >>>> > > > > > > > > > > > > aspects: > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent > > > > changes > > > > > >>>> where > > > > > >>>> > > the > > > > > >>>> > > > > > > service > > > > > >>>> > > > > > > > > > reads > > > > > >>>> > > > > > > > > > > > > and validates current catalog state, > > > then > > > > > >>>> commits a > > > > > >>>> > > > > change > > > > > >>>> > > > > > > (e.g. > > > > > >>>> > > > > > > > > > > > > name clashes during renames). > > > > > >>>> > > > > > > > > > > > > * Supporting consistent but > independent > > > > > changes > > > > > >>>> to RBAC > > > > > >>>> > > > > > grants > > > > > >>>> > > > > > > and > > > > > >>>> > > > > > > > > > > > > MetaStore entities. This > independence is > > > > > >>>> needed to > > > > > >>>> > > > > support > > > > > >>>> > > > > > > > > > > > > external authorizers like OPA and > > > Ranger. > > > > > >>>> > > > > > > > > > > > > * Supporting atomic changes across > > > multiple > > > > > >>>> similar > > > > > >>>> > > > > entities. > > > > > >>>> > > > > > > > > > > > > * Supporting authorization-based > filtering > > > > of > > > > > >>>> list > > > > > >>>> > > > > operations > > > > > >>>> > > > > > > (cf. > > > > > >>>> > > > > > > > > > > > > [4831]). > > > > > >>>> > > > > > > > > > > > > * Supporting credential-vending > decisions > > > > that > > > > > >>>> are > > > > > >>>> > > rooted > > > > > >>>> > > > > in > > > > > >>>> > > > > > > the > > > > > >>>> > > > > > > > > > > > > exact state of the catalog. > > > > > >>>> > > > > > > > > > > > > * Supporting server-side retries for > > > > transient > > > > > >>>> > > > persistence > > > > > >>>> > > > > > > failures > > > > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability > > > failures). > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please share your comments and ideas. > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4831] > > > > > >>>> https://github.com/apache/polaris/pull/4831 > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4939] > > > > > >>>> https://github.com/apache/polaris/pull/4939 > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [5035] > > > > > >>>> https://github.com/apache/polaris/pull/5035 > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Thanks, > > > > > >>>> > > > > > > > > > > > > Dmitri > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > >>> > > > > > > > > > > > > >
