> The MetaStore SPI is currently defined with the idea that one method call means one atomic change.
If the "MetaStore SPI" refers to the interface PolarisMetaStoreManager, I don't think we've ever state each method to be atomic. We did clarify atomicity[1] in the interface BasePersistence though. 1. https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48 Yufei On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Yufei, > > I agree that JDBC transactions must be handled more explicitly. However, > I'm not sure that simply moving to TransactionalMetaStoreManagerImpl is > sufficient. > > The MetaStore SPI is currently defined with the idea that one method call > means one atomic change [1]. The "transactional" MetaStore impl. is but a > sub-case of that. It cannot alter the high-level contract. > > We could add SPI methods having multiple object parameters to represent > grouped changes, but I am not sure it will be a sound design. This will > bloat the interface surfaces and require extra impl. effort for each > backend type. More importantly, adding multi-arg change methods still won't > address the problem of reads being consistent with writes, because each > method call will still be independent regarding the data stored in the > database. > > I tend to think we need to introduce a "change set" or "atomic batch" > concept to core Persistence and associate each REST API request with one > such change set, which will be committed (or rolled back) at the end of the > request. I believe Ayush mentioned a similar concept in PR 4939 [2]. In > JDBC each change set will naturally be associated with an RDBMS > transaction. In NoSQL persistence, each atomic change set will be > associated with one CAS operation on the underlying database. > > [1] https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > [2] https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > > Cheers, > Dmitri. > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <[email protected]> wrote: > > > Thanks for raising this, Dmitri. > > > > These are valid concerns, and they were already recognized when we > > introduced JDBC persistence to Polaris. At that time, we chose to use > > AtomicOperationMetaStoreManager for the JDBC due to the simplicity. I > think > > most of the issues mentioned here can already be addressed by > > TransactionalMetaStoreManagerImpl. > > > > For example, rename is already wrapped in a transaction in > > TransactionalMetaStoreManagerImpl [1]. Similarly, catalog creation, which > > involves reading and creating multiple objects, is also executed within a > > transaction [2]. > > > > I see two possible directions: > > > > 1. > > > > Modify AtomicOperationMetaStoreManager together with the persistence > > backends (such as JDBC) to provide the required consistency guarantees > > for > > specific operations, similar to what TransactionalMetaStoreManagerImpl > > does. > > 2. > > > > Migrate the persistence backends (such as JDBC) to use > > TransactionalMetaStoreManagerImpl directly. We may have to deal with > > transactional semantic mismatches across different persistence > backends. > > For example, we would likely avoid using JDBC's `runWithinTransaction` > > for > > single row updates, which adds additional overhead and complexity > > without > > benefits. > > > > References: > > > > 1. > > > > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 > > 2. > > > > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 > > > > Yufei > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi all, > > > > > > Ayush and Prithvi recently contributed a couple of interesting PRs: > > > [4939], [5035]. > > > > > > It looks like people are starting to encounter consistency issues in > > > JDBC persistence. > > > > > > The PRs provide valuable insight into the underlying issues. They offer > > > incremental fixes that can work. However, I believe it is time for the > > > Polaris community to review and improve this area of the codebase > > > holistically. > > > > > > By this, I mean finding a solution that can be applied to all > > > persistence backends (in-memory, JDBC, NoSQL) and addresses these > > > aspects: > > > > > > * Supporting concurrent and consistent changes where the service reads > > > and validates current catalog state, then commits a change (e.g. > > > name clashes during renames). > > > * Supporting consistent but independent changes to RBAC grants and > > > MetaStore entities. This independence is needed to support > > > external authorizers like OPA and Ranger. > > > * Supporting atomic changes across multiple similar entities. > > > * Supporting authorization-based filtering of list operations (cf. > > > [4831]). > > > * Supporting credential-vending decisions that are rooted in the > > > exact state of the catalog. > > > * Supporting server-side retries for transient persistence failures > > > (e.g. RDBMS Tx serializability failures). > > > > > > Please share your comments and ideas. > > > > > > [4831] https://github.com/apache/polaris/pull/4831 > > > > > > [4939] https://github.com/apache/polaris/pull/4939 > > > > > > [5035] https://github.com/apache/polaris/pull/5035 > > > > > > Thanks, > > > Dmitri > > > > > >
