HI All, Related area for improvement: https://github.com/apache/polaris/issues/5087
Cheers, Dmitri. On Thu, Jul 16, 2026 at 2:26 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Yufei, > > I agree that JDBC transactions must be handled more explicitly. However, > I'm not sure that simply moving to TransactionalMetaStoreManagerImpl is > sufficient. > > The MetaStore SPI is currently defined with the idea that one method call > means one atomic change [1]. The "transactional" MetaStore impl. is but a > sub-case of that. It cannot alter the high-level contract. > > We could add SPI methods having multiple object parameters to represent > grouped changes, but I am not sure it will be a sound design. This will > bloat the interface surfaces and require extra impl. effort for each > backend type. More importantly, adding multi-arg change methods still won't > address the problem of reads being consistent with writes, because each > method call will still be independent regarding the data stored in the > database. > > I tend to think we need to introduce a "change set" or "atomic batch" > concept to core Persistence and associate each REST API request with one > such change set, which will be committed (or rolled back) at the end of the > request. I believe Ayush mentioned a similar concept in PR 4939 [2]. In > JDBC each change set will naturally be associated with an RDBMS > transaction. In NoSQL persistence, each atomic change set will be > associated with one CAS operation on the underlying database. > > [1] https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > [2] https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > > Cheers, > Dmitri. > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <[email protected]> wrote: > >> Thanks for raising this, Dmitri. >> >> These are valid concerns, and they were already recognized when we >> introduced JDBC persistence to Polaris. At that time, we chose to use >> AtomicOperationMetaStoreManager for the JDBC due to the simplicity. I >> think >> most of the issues mentioned here can already be addressed by >> TransactionalMetaStoreManagerImpl. >> >> For example, rename is already wrapped in a transaction in >> TransactionalMetaStoreManagerImpl [1]. Similarly, catalog creation, which >> involves reading and creating multiple objects, is also executed within a >> transaction [2]. >> >> I see two possible directions: >> >> 1. >> >> Modify AtomicOperationMetaStoreManager together with the persistence >> backends (such as JDBC) to provide the required consistency guarantees >> for >> specific operations, similar to what TransactionalMetaStoreManagerImpl >> does. >> 2. >> >> Migrate the persistence backends (such as JDBC) to use >> TransactionalMetaStoreManagerImpl directly. We may have to deal with >> transactional semantic mismatches across different persistence >> backends. >> For example, we would likely avoid using JDBC's `runWithinTransaction` >> for >> single row updates, which adds additional overhead and complexity >> without >> benefits. >> >> References: >> >> 1. >> >> >> >> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 >> 2. >> >> >> >> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 >> >> Yufei >> >> >> On Thu, Jul 16, 2026 at 8:22 AM Dmitri Bourlatchkov <[email protected]> >> wrote: >> >> > Hi all, >> > >> > Ayush and Prithvi recently contributed a couple of interesting PRs: >> > [4939], [5035]. >> > >> > It looks like people are starting to encounter consistency issues in >> > JDBC persistence. >> > >> > The PRs provide valuable insight into the underlying issues. They offer >> > incremental fixes that can work. However, I believe it is time for the >> > Polaris community to review and improve this area of the codebase >> > holistically. >> > >> > By this, I mean finding a solution that can be applied to all >> > persistence backends (in-memory, JDBC, NoSQL) and addresses these >> > aspects: >> > >> > * Supporting concurrent and consistent changes where the service reads >> > and validates current catalog state, then commits a change (e.g. >> > name clashes during renames). >> > * Supporting consistent but independent changes to RBAC grants and >> > MetaStore entities. This independence is needed to support >> > external authorizers like OPA and Ranger. >> > * Supporting atomic changes across multiple similar entities. >> > * Supporting authorization-based filtering of list operations (cf. >> > [4831]). >> > * Supporting credential-vending decisions that are rooted in the >> > exact state of the catalog. >> > * Supporting server-side retries for transient persistence failures >> > (e.g. RDBMS Tx serializability failures). >> > >> > Please share your comments and ideas. >> > >> > [4831] https://github.com/apache/polaris/pull/4831 >> > >> > [4939] https://github.com/apache/polaris/pull/4939 >> > >> > [5035] https://github.com/apache/polaris/pull/5035 >> > >> > Thanks, >> > Dmitri >> > >> >
