Hi All, Here's a recap of the Community Sync discussion as I remember it. I'm sure my recollection is not complete, so please add your thoughts to this thread.
* Polaris traditionally relies on entity version number to ensure consistency of changes. This works well for internal RBAC grants, which also have version numbers cross-referenced at the Persistence layer. I do not think this works in more general cases like validating locations wrt catalog-level storage configuration. Another difficult case involves location overlaps between concurrent table creations. * We talked about how Persistence calls related to RDBMS transactions (in the JDBC persistence case). There were some concerns raised about whether we need to bind Persistence API to transactions explicitly. I think it is a valid concern. At the same time, the Persistence API should be clear about consistency guarantees across all backend implementations. I think this needs more attention now that the matter of milti-entioty changes became prominent in [5035]. I personally believe that RDBMS-based Persistence in Polaris has to use Serializable transaction isolation. Consequently, the core code needs to provide mechanisms to allow the Persistence implementation to properly connect reads and writes from API requests to JDBC transactions... whether we use "transation" as an explicit terms in the SPI or not. Note that currently, the same API request performs reads and writes in _separate_ RDBMS transactions (even separate connections). * We did not discuss Persistence SPI implications in the call, but currently each Persistence call is assumed to be one distinct and atomic change (please correct me if I'm wrong). Consequently, supporting new use cases involves introducing new SPI methods with many parameters. This can be seen in PR [5035]. This tends to overcomplicate the SPI. Ideally, I think the SPI should be simplified to deal with multi-entity and single-entity changes in a coherent manner to avoid any ambiguity about which method the caller should use. [5035] https://github.com/apache/polaris/pull/5035 Cheers, Dmitri. On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov <[email protected]> wrote: > Heads up: This discussion is on the Community Sync call agenda for Sept 17. > > Interested parties, please plan to attend, if possible :) > > Cheers, > Dmitri. > > On Wed, Sep 9, 2026 at 1:48 PM Dmitri Bourlatchkov <[email protected]> > wrote: > >> Hi JB, >> >> I've added an agenda item to the next community sync call for this (Sept >> 17, 2026). >> >> Re: key points: did you mean a real GH discussion or dev email (this >> thread)? Just double checking :) I'm fine with either approach. >> >> Cheers, >> Dmitri. >> >> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <[email protected]> >> wrote: >> >>> Hi Dmitri, >>> >>> I agree on the need to align on these points, though I'm not entirely >>> sure a dedicated meeting is necessary. Let's start by using some time >>> during the next community meeting to discuss it. >>> >>> In the meantime, what if we outline the key points in a GitHub >>> Discussion first? We can then use that as a reference during the >>> meeting. >>> >>> Regards, >>> JB >>> >>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <[email protected]> >>> wrote: >>> > >>> > Hi All, >>> > >>> > My impression from this thread is that it might be time for a call to >>> > discuss all the related issues and try to arrive at a shared >>> implementation >>> > plan. >>> > >>> > I know meetings are not ideal, but in this case it might be beneficial >>> as a >>> > means for achieving a common understanding of the set of problems and >>> > priorities related to this thread. A dedicated metrics meeting worked >>> well >>> > from my POV. >>> > >>> > Allocating a time slot in the next community meeting might be an >>> option, >>> > although I think we might need the full hour in this case. >>> > >>> > WDYT? >>> > >>> > Thanks, >>> > Dmitri. >>> > >>> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <[email protected] >>> > >>> > wrote: >>> > >>> > > Hi all, >>> > > >>> > > I rewrote https://github.com/apache/polaris/pull/5035, following the >>> > > discussion to focus only on the SPI foundation. >>> > > >>> > > What is in the PR now: >>> > > >>> > > - A written manager-level consistency contract in >>> > > site/content/in-dev/unreleased/persistence-consistency.md. >>> > > - CAS-aware EntityMutation / GrantMutation records. >>> > > - MetaStoreChangeSet and BasePersistence#commitChangeSet, with >>> backends >>> > > opting in via supportsAtomicMixedCommit(). >>> > > - commitChangeSet implementations for the in-memory TreeMap >>> backend and >>> > > JDBC. >>> > > - AtomicOperationMetaStoreManager and >>> TransactionalMetaStoreManagerImpl >>> > > grant/revoke paths now build one change set per operation, >>> falling back >>> > > to >>> > > individual operations when the backend does not support mixed >>> commits. >>> > > - Tests for the change-set API and TreeMap atomic commits. >>> > > >>> > > I intentionally left out: >>> > > >>> > > - Entity deletes in MetaStoreChangeSet. >>> > > - Refactoring createCatalog, dropEntity, and renameEntity to use >>> change >>> > > sets. >>> > > - NoSQL support for commitChangeSet. >>> > > >>> > > I am treating this as Phase 1 so the contract and the SPI primitive >>> can be >>> > > reviewed before the larger operation migrations land. >>> > > >>> > > Could you please take a look? In particular, I would like to know >>> whether >>> > > the contract captures the consensus so far, or if it needs to >>> address retry >>> > > / external-work coordination before we merge this foundation. >>> > > >>> > > Thanks, >>> > > Prithvi S >>> > > >>> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov < >>> [email protected]> >>> > > wrote: >>> > > >>> > > > Hi Robert, >>> > > > >>> > > > I agree that the client-visible operation (e.g. REST API request) >>> > > consists >>> > > > of many distinct phases. The database / persistence changes are >>> just one >>> > > of >>> > > > these phases. STS (as an example) is another. My point about >>> transactions >>> > > > in JDBC Persistence applies to the former (database changes) phase >>> - one >>> > > > retry attempt there should ideally involve exactly one transaction. >>> > > > >>> > > > We certainly need logic inside Polaris Servers to to coordinate >>> requests >>> > > to >>> > > > various external systems depending on outcomes from previous >>> request >>> > > > processing phases. >>> > > > >>> > > > I also agree that validation based on database state should be >>> repeated >>> > > > from scratch if we retry the database changes. >>> > > > >>> > > > However, internal RBAC authorization naturally has to happen >>> inside the >>> > > > database update phase (read-check-write). IIRC, that is part of the >>> > > > implicit consistency guarantees, which were discussed during early >>> > > project >>> > > > months (but never got written down in full clarity, >>> unfortunately). If >>> > > the >>> > > > Authorization check is inside that phase for internal RBAC, I >>> suppose it >>> > > > will be there for other Authorizers too and that can involve >>> external >>> > > > systems (e.g. OPA / Ranger). We can certainly expect authorization >>> to be >>> > > > efficient, but I'm not sure how quick it can be in practice. >>> There's >>> > > > certainly potential for delays in some (perhaps infrequent) cases. >>> > > > >>> > > > Cheers, >>> > > > Dmitri. >>> > > > >>> > > > >>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <[email protected]> >>> wrote: >>> > > > >>> > > > > Hi Dmitri, >>> > > > > >>> > > > > I agree that one logical operation needs a consistent view of >>> the whole >>> > > > > backend state. >>> > > > > What worries me is treating one long JDBC transaction as the >>> boundary >>> > > of >>> > > > > that operation and then retrying the complete request if the >>> commit >>> > > > fails. >>> > > > > >>> > > > > The request can also write metadata to object storage or call >>> STS. >>> > > > > Each system can leave us with an outcome whose certainty we >>> cannot >>> > > know. >>> > > > > The database may have committed before the connection was lost, >>> an >>> > > > > object-store write may have succeeded before a timeout, or STS >>> may have >>> > > > > issued credentials before its response was lost. >>> > > > > A database rollback cannot undo any of those effects, and >>> retrying the >>> > > > > whole request may repeat them. >>> > > > > >>> > > > > So I think the contract has to separate the overall operation >>> from each >>> > > > > attempt to commit it. >>> > > > > Each database or backend commit attempt should stay short. >>> > > > > When we retry, validation and authorization need to use the >>> state for >>> > > > that >>> > > > > new attempt, and we need clear rules for which external work can >>> be >>> > > > reused, >>> > > > > repeated, cleaned up, or reconciled after an uncertain outcome. >>> > > > > >>> > > > > Some contextual state could be useful for carrying that state. >>> > > > > But the JDBC connection and transaction should be an >>> implementation >>> > > > detail >>> > > > > inside it, not the boundary of the complete operation. >>> > > > > >>> > > > > Cheers, >>> > > > > Robert >>> > > > > >>> > > > > >>> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov < >>> [email protected] >>> > > > >>> > > > > wrote: >>> > > > > >>> > > > > > Hi All, >>> > > > > > >>> > > > > > The idea of a backend-agnostic change-set primitive sounds >>> good. >>> > > > > > >>> > > > > > However, I am not sure it is sufficient for all the use cases >>> we >>> > > > touched >>> > > > > > here. More specifically, the state read by validation code is >>> not >>> > > > > > necessarily reflected in the change set: >>> > > > > > >>> > > > > > * Unchanged entities might be considered by validation, but >>> changed >>> > > in >>> > > > a >>> > > > > > parallel request. >>> > > > > > * Even for changed entities the state read by validation may >>> differ >>> > > > from >>> > > > > > the state being altered. Some validation code talks to the >>> MetaStore >>> > > > > > directly, outside of the data produced by the Resolver. >>> > > > > > >>> > > > > > I do not think Polaris offers any explicit mechanisms (ATM) to >>> ensure >>> > > > > > consistency between these reads and subsequent writes. >>> > > > > > >>> > > > > > As far as JDBC goes, running an overarching Transaction across >>> all >>> > > > reads >>> > > > > > and writes in the same request, at the SERIALIZABLE isolation >>> level >>> > > in >>> > > > > the >>> > > > > > backing RDBMS could solve the problem, I think. >>> > > > > > >>> > > > > > Indeed, such a transaction might be long to accommodate calls >>> made by >>> > > > > > Polaris to external storage, etc. However, is that a problem? >>> I think >>> > > > the >>> > > > > > alternative is for the JDBC Persistence impl. to "manually" >>> track all >>> > > > > reads >>> > > > > > under the same request and redo them in the small transaction >>> that >>> > > > > persists >>> > > > > > the writes. This will also add RDBMS overhead and require >>> complex >>> > > code >>> > > > in >>> > > > > > Polaris to handle the data properly. >>> > > > > > >>> > > > > > Tracking the request-wide transaction can be done only for JDBC >>> > > without >>> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I >>> think. >>> > > NoSQL >>> > > > > > will use other mechanisms to ensure read/write consistency. >>> > > > > > >>> > > > > > Connecting to Robert's email (a parallel branch in this >>> discussion), >>> > > > I'd >>> > > > > > like to propose this approach: >>> > > > > > >>> > > > > > * Each request establishes a "Data Context" >>> > > > > > - In JDBC the Data Context corresponds to a JDBC Connection >>> + Tx >>> > > > > > - In NoSQL the Data Context tracks one or more reference >>> hashes >>> > > > > > * All Persistence access in the same request goes through the >>> same >>> > > Data >>> > > > > > Context >>> > > > > > * All Persistence changes are committed once at the end of the >>> > > request >>> > > > > > - Not all changes have to be globally atomic. For example, >>> changes >>> > > in >>> > > > > > Catalogs A and B do not have to be atomic with respect to each >>> other. >>> > > > We >>> > > > > > can go deeper into this later. This is relevant to NoSQL. >>> > > > > > - Transactional backends like JDBC can, of course, choose to >>> make >>> > > all >>> > > > > > changes globally atomic. >>> > > > > > * A commit can fail in two main ways: >>> > > > > > - A retriable failure like an optimistic lock error or Tx >>> > > > > serializability >>> > > > > > error >>> > > > > > - A non-triable logical error (e.g. entity not found) >>> > > > > > * On a retriable error the whole request is re-attempted (as if >>> > > > > resubmitted >>> > > > > > by a client) a few times (configurable timeout). >>> > > > > > >>> > > > > > WDYT? >>> > > > > > >>> > > > > > Cheers, >>> > > > > > Dmitri. >>> > > > > > >>> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < >>> > > [email protected] >>> > > > > >>> > > > > > wrote: >>> > > > > > >>> > > > > > > Hi all >>> > > > > > > >>> > > > > > > I think Robert has a good point. >>> > > > > > > >>> > > > > > > If the atomicity guarantee lives only in BasePersistence, >>> then the >>> > > > > > > manager contract can't tell a caller whether the state it >>> read for >>> > > > > > > validation/authorization/credential-vending is the same >>> state that >>> > > > > > > eventually commits. That's the actual bug class behind the >>> JDBC >>> > > > > > > symptoms (not any single operation being non-atomic, but the >>> > > contract >>> > > > > > > being silent about it. Every operation-specific fix (like >>> #4939, >>> > > > #5035 >>> > > > > > > or #5095) re-answers this question locally and it keeps >>> recurring. >>> > > So >>> > > > > > > the deliverable should start with a written consistency >>> contract at >>> > > > > > > the manager level. >>> > > > > > > >>> > > > > > > I'm not sure migrating everyone to >>> > > TransactionalMetaStoreManagerImpl >>> > > > > > > is a good idea. It would tie the logical change set to a DB >>> > > > > > > transaction spanning the REST request. It means: >>> > > > > > > - it holds a durable transaction open across slow external >>> work >>> > > > > > > (credential vending, OPA, Ranger, ...) >>> > > > > > > - it doesn't map to NoSQL >>> > > > > > > - It wraps single-row updates in runWiithinTransaction, >>> which is a >>> > > > > > overhead >>> > > > > > > >>> > > > > > > So, I think the transactional manager isn't a portable >>> target. It's >>> > > > > > > "only" one backend's strategy. >>> > > > > > > >>> > > > > > > I think Privthi's approach is right. A backend-agnostic >>> change-set >>> > > > > > > primitive with a documented fallback is the correct shape. >>> There is >>> > > > > > > one caveat: without carrying the original entitiy (not just >>> the new >>> > > > > > > one) the change set can't express optimistic concurrency. >>> > > > > > > >>> > > > > > > I propose the following multi-steps approach: >>> > > > > > > 1. We write the manager-level consistency contract as Robert >>> asked. >>> > > > It >>> > > > > > > should include the explicit statement that a logical change >>> set is >>> > > > not >>> > > > > > > a request-scope DB transaction. >>> > > > > > > 2. We make Compare And Swap (optimistic-concurrency pattern) >>> > > baseline >>> > > > > > > first-class in EntityMutation before merging the SPI >>> > > > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity. >>> > > > > > > >>> > > > > > > Thoughts? >>> > > > > > > >>> > > > > > > Regards >>> > > > > > > JB >>> > > > > > > >>> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <[email protected] >>> > >>> > > wrote: >>> > > > > > > > >>> > > > > > > > Hi all, >>> > > > > > > > >>> > > > > > > > Yufei's clarification about where the atomicity guarantee >>> is >>> > > > defined >>> > > > > > > seems >>> > > > > > > > important. >>> > > > > > > > If it is a property of the lower-level BasePersistence >>> contract >>> > > > > rather >>> > > > > > > than >>> > > > > > > > the general PolarisMetaStoreManager contract, the general >>> > > contract >>> > > > > does >>> > > > > > > not >>> > > > > > > > tell callers whether the state used for validation, >>> > > authorization, >>> > > > or >>> > > > > > > > credential vending is consistent with the change that >>> eventually >>> > > > > > commits. >>> > > > > > > > >>> > > > > > > > The current PRs suggest that operation-specific >>> multi-object >>> > > > methods >>> > > > > > can >>> > > > > > > > fix individual cases, while leaving the same contract >>> question to >>> > > > > recur >>> > > > > > > for >>> > > > > > > > each new case. >>> > > > > > > > >>> > > > > > > > I would also avoid defining a logical change set as a >>> database >>> > > > > > > transaction >>> > > > > > > > around an entire REST request. >>> > > > > > > > That would tie the contract to the backend and database, >>> and >>> > > could >>> > > > > keep >>> > > > > > > the >>> > > > > > > > durable attempt open across slow external work. >>> > > > > > > > >>> > > > > > > > So is the choice really between the two current manager >>> > > > > > implementations, >>> > > > > > > or >>> > > > > > > > do we first need to revisit the boundary and guarantees >>> exposed >>> > > to >>> > > > > > their >>> > > > > > > > callers? >>> > > > > > > > >>> > > > > > > > Cheers, >>> > > > > > > > Robert >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov < >>> > > > > [email protected] >>> > > > > > > >>> > > > > > > > wrote: >>> > > > > > > > >>> > > > > > > > > Hi Yufei, >>> > > > > > > > > >>> > > > > > > > > Thanks for the link. I stand corrected. The "one atomic >>> change >>> > > > per >>> > > > > > > method" >>> > > > > > > > > contract as defined in javadoc does apply only to >>> > > BasePersistence >>> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates to >>> > > > > > > BasePersistence). >>> > > > > > > > > >>> > > > > > > > > Note that all non-test Persistence implementations in the >>> > > Polaris >>> > > > > > > codebase >>> > > > > > > > > extend those classes (which is probably why I was >>> confused >>> > > about >>> > > > > > > atomicity >>> > > > > > > > > expectations). >>> > > > > > > > > >>> > > > > > > > > However, this creates a gap in the Persistence SPI >>> > > specification. >>> > > > > If >>> > > > > > > > > other PolarisMetaStoreManager implementations do not >>> have to >>> > > > comply >>> > > > > > > with >>> > > > > > > > > this principle, it will create a conceptual difficulty >>> at call >>> > > > > sites. >>> > > > > > > How >>> > > > > > > > > can PolarisMetaStoreManager callers reason about >>> consistency >>> > > and >>> > > > > > > durability >>> > > > > > > > > behaviours in general? >>> > > > > > > > > >>> > > > > > > > > I believe we need to address that as part of this >>> discussion. >>> > > > > > > > > >>> > > > > > > > > Cheers, >>> > > > > > > > > Dmitri. >>> > > > > > > > > >>> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu < >>> [email protected] >>> > > > >>> > > > > > wrote: >>> > > > > > > > > >>> > > > > > > > > > > The MetaStore SPI is currently defined with the idea >>> that >>> > > one >>> > > > > > > method >>> > > > > > > > > call >>> > > > > > > > > > means one atomic change. >>> > > > > > > > > > >>> > > > > > > > > > If the "MetaStore SPI" refers to the interface >>> > > > > > > PolarisMetaStoreManager, I >>> > > > > > > > > > don't think we've ever state each method to be atomic. >>> We did >>> > > > > > clarify >>> > > > > > > > > > atomicity[1] in the interface BasePersistence though. >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > 1. >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48 >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > Yufei >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov < >>> > > > > > > [email protected]> >>> > > > > > > > > > wrote: >>> > > > > > > > > > >>> > > > > > > > > > > Hi Yufei, >>> > > > > > > > > > > >>> > > > > > > > > > > I agree that JDBC transactions must be handled more >>> > > > explicitly. >>> > > > > > > > > However, >>> > > > > > > > > > > I'm not sure that simply moving to >>> > > > > > > TransactionalMetaStoreManagerImpl is >>> > > > > > > > > > > sufficient. >>> > > > > > > > > > > >>> > > > > > > > > > > The MetaStore SPI is currently defined with the idea >>> that >>> > > one >>> > > > > > > method >>> > > > > > > > > call >>> > > > > > > > > > > means one atomic change [1]. The "transactional" >>> MetaStore >>> > > > > impl. >>> > > > > > is >>> > > > > > > > > but a >>> > > > > > > > > > > sub-case of that. It cannot alter the high-level >>> contract. >>> > > > > > > > > > > >>> > > > > > > > > > > We could add SPI methods having multiple object >>> parameters >>> > > to >>> > > > > > > represent >>> > > > > > > > > > > grouped changes, but I am not sure it will be a sound >>> > > design. >>> > > > > > This >>> > > > > > > will >>> > > > > > > > > > > bloat the interface surfaces and require extra impl. >>> effort >>> > > > for >>> > > > > > > each >>> > > > > > > > > > > backend type. More importantly, adding multi-arg >>> change >>> > > > methods >>> > > > > > > still >>> > > > > > > > > > won't >>> > > > > > > > > > > address the problem of reads being consistent with >>> writes, >>> > > > > > because >>> > > > > > > each >>> > > > > > > > > > > method call will still be independent regarding the >>> data >>> > > > stored >>> > > > > > in >>> > > > > > > the >>> > > > > > > > > > > database. >>> > > > > > > > > > > >>> > > > > > > > > > > I tend to think we need to introduce a "change set" >>> or >>> > > > "atomic >>> > > > > > > batch" >>> > > > > > > > > > > concept to core Persistence and associate each REST >>> API >>> > > > request >>> > > > > > > with >>> > > > > > > > > one >>> > > > > > > > > > > such change set, which will be committed (or rolled >>> back) >>> > > at >>> > > > > the >>> > > > > > > end of >>> > > > > > > > > > the >>> > > > > > > > > > > request. I believe Ayush mentioned a similar concept >>> in PR >>> > > > 4939 >>> > > > > > > [2]. In >>> > > > > > > > > > > JDBC each change set will naturally be associated >>> with an >>> > > > RDBMS >>> > > > > > > > > > > transaction. In NoSQL persistence, each atomic >>> change set >>> > > > will >>> > > > > be >>> > > > > > > > > > > associated with one CAS operation on the underlying >>> > > database. >>> > > > > > > > > > > >>> > > > > > > > > > > [1] >>> > > > > > > >>> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r >>> > > > > > > > > > > >>> > > > > > > > > > > [2] >>> > > > > > > >>> https://github.com/apache/polaris/pull/4939#discussion_r3575719158 >>> > > > > > > > > > > >>> > > > > > > > > > > Cheers, >>> > > > > > > > > > > Dmitri. >>> > > > > > > > > > > >>> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < >>> > > > [email protected] >>> > > > > > >>> > > > > > > wrote: >>> > > > > > > > > > > >>> > > > > > > > > > > > Thanks for raising this, Dmitri. >>> > > > > > > > > > > > >>> > > > > > > > > > > > These are valid concerns, and they were already >>> > > recognized >>> > > > > when >>> > > > > > > we >>> > > > > > > > > > > > introduced JDBC persistence to Polaris. At that >>> time, we >>> > > > > chose >>> > > > > > > to use >>> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due >>> to the >>> > > > > > > simplicity. I >>> > > > > > > > > > > think >>> > > > > > > > > > > > most of the issues mentioned here can already be >>> > > addressed >>> > > > by >>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl. >>> > > > > > > > > > > > >>> > > > > > > > > > > > For example, rename is already wrapped in a >>> transaction >>> > > in >>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. Similarly, >>> catalog >>> > > > > > > creation, >>> > > > > > > > > > which >>> > > > > > > > > > > > involves reading and creating multiple objects, is >>> also >>> > > > > > executed >>> > > > > > > > > > within a >>> > > > > > > > > > > > transaction [2]. >>> > > > > > > > > > > > >>> > > > > > > > > > > > I see two possible directions: >>> > > > > > > > > > > > >>> > > > > > > > > > > > 1. >>> > > > > > > > > > > > >>> > > > > > > > > > > > Modify AtomicOperationMetaStoreManager together >>> with >>> > > the >>> > > > > > > > > persistence >>> > > > > > > > > > > > backends (such as JDBC) to provide the required >>> > > > > consistency >>> > > > > > > > > > guarantees >>> > > > > > > > > > > > for >>> > > > > > > > > > > > specific operations, similar to what >>> > > > > > > > > > TransactionalMetaStoreManagerImpl >>> > > > > > > > > > > > does. >>> > > > > > > > > > > > 2. >>> > > > > > > > > > > > >>> > > > > > > > > > > > Migrate the persistence backends (such as JDBC) >>> to use >>> > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. We >>> may >>> > > have >>> > > > to >>> > > > > > > deal >>> > > > > > > > > with >>> > > > > > > > > > > > transactional semantic mismatches across >>> different >>> > > > > > persistence >>> > > > > > > > > > > backends. >>> > > > > > > > > > > > For example, we would likely avoid using JDBC's >>> > > > > > > > > > `runWithinTransaction` >>> > > > > > > > > > > > for >>> > > > > > > > > > > > single row updates, which adds additional >>> overhead and >>> > > > > > > complexity >>> > > > > > > > > > > > without >>> > > > > > > > > > > > benefits. >>> > > > > > > > > > > > >>> > > > > > > > > > > > References: >>> > > > > > > > > > > > >>> > > > > > > > > > > > 1. >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 >>> > > > > > > > > > > > 2. >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 >>> > > > > > > > > > > > >>> > > > > > > > > > > > Yufei >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri >>> Bourlatchkov < >>> > > > > > > > > [email protected]> >>> > > > > > > > > > > > wrote: >>> > > > > > > > > > > > >>> > > > > > > > > > > > > Hi all, >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > Ayush and Prithvi recently contributed a couple >>> of >>> > > > > > interesting >>> > > > > > > PRs: >>> > > > > > > > > > > > > [4939], [5035]. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > It looks like people are starting to encounter >>> > > > consistency >>> > > > > > > issues >>> > > > > > > > > in >>> > > > > > > > > > > > > JDBC persistence. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > The PRs provide valuable insight into the >>> underlying >>> > > > > issues. >>> > > > > > > They >>> > > > > > > > > > offer >>> > > > > > > > > > > > > incremental fixes that can work. However, I >>> believe it >>> > > is >>> > > > > > time >>> > > > > > > for >>> > > > > > > > > > the >>> > > > > > > > > > > > > Polaris community to review and improve this >>> area of >>> > > the >>> > > > > > > codebase >>> > > > > > > > > > > > > holistically. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > By this, I mean finding a solution that can be >>> applied >>> > > to >>> > > > > all >>> > > > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL) and >>> > > > addresses >>> > > > > > > these >>> > > > > > > > > > > > > aspects: >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > * Supporting concurrent and consistent changes >>> where >>> > > the >>> > > > > > > service >>> > > > > > > > > > reads >>> > > > > > > > > > > > > and validates current catalog state, then >>> commits a >>> > > > > change >>> > > > > > > (e.g. >>> > > > > > > > > > > > > name clashes during renames). >>> > > > > > > > > > > > > * Supporting consistent but independent changes >>> to RBAC >>> > > > > > grants >>> > > > > > > and >>> > > > > > > > > > > > > MetaStore entities. This independence is >>> needed to >>> > > > > support >>> > > > > > > > > > > > > external authorizers like OPA and Ranger. >>> > > > > > > > > > > > > * Supporting atomic changes across multiple >>> similar >>> > > > > entities. >>> > > > > > > > > > > > > * Supporting authorization-based filtering of >>> list >>> > > > > operations >>> > > > > > > (cf. >>> > > > > > > > > > > > > [4831]). >>> > > > > > > > > > > > > * Supporting credential-vending decisions that >>> are >>> > > rooted >>> > > > > in >>> > > > > > > the >>> > > > > > > > > > > > > exact state of the catalog. >>> > > > > > > > > > > > > * Supporting server-side retries for transient >>> > > > persistence >>> > > > > > > failures >>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures). >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > Please share your comments and ideas. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > [4831] >>> https://github.com/apache/polaris/pull/4831 >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > [4939] >>> https://github.com/apache/polaris/pull/4939 >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > [5035] >>> https://github.com/apache/polaris/pull/5035 >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > Thanks, >>> > > > > > > > > > > > > Dmitri >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> >>
