Hi JB,

I've added an agenda item to the next community sync call for this (Sept
17, 2026).

Re: key points: did you mean a real GH discussion or dev email (this
thread)? Just double checking :) I'm fine with either approach.

Cheers,
Dmitri.

On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste Onofré <[email protected]>
wrote:

> Hi Dmitri,
>
> I agree on the need to align on these points, though I'm not entirely
> sure a dedicated meeting is necessary. Let's start by using some time
> during the next community meeting to discuss it.
>
> In the meantime, what if we outline the key points in a GitHub
> Discussion first? We can then use that as a reference during the
> meeting.
>
> Regards,
> JB
>
> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov <[email protected]>
> wrote:
> >
> > Hi All,
> >
> > My impression from this thread is that it might be time for a call to
> > discuss all the related issues and try to arrive at a shared
> implementation
> > plan.
> >
> > I know meetings are not ideal, but in this case it might be beneficial
> as a
> > means for achieving a common understanding of the set of problems and
> > priorities related to this thread. A dedicated metrics meeting worked
> well
> > from my POV.
> >
> > Allocating a time slot in the next community meeting might be an option,
> > although I think we might need the full hour in this case.
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > On Mon, Aug 17, 2026 at 6:39 PM Prithvi S <[email protected]>
> > wrote:
> >
> > > Hi all,
> > >
> > > I rewrote https://github.com/apache/polaris/pull/5035, following the
> > > discussion to focus only on the SPI foundation.
> > >
> > > What is in the PR now:
> > >
> > >    - A written manager-level consistency contract in
> > >    site/content/in-dev/unreleased/persistence-consistency.md.
> > >    - CAS-aware EntityMutation / GrantMutation records.
> > >    - MetaStoreChangeSet and BasePersistence#commitChangeSet, with
> backends
> > >    opting in via supportsAtomicMixedCommit().
> > >    - commitChangeSet implementations for the in-memory TreeMap backend
> and
> > >    JDBC.
> > >    - AtomicOperationMetaStoreManager and
> TransactionalMetaStoreManagerImpl
> > >    grant/revoke paths now build one change set per operation, falling
> back
> > > to
> > >    individual operations when the backend does not support mixed
> commits.
> > >    - Tests for the change-set API and TreeMap atomic commits.
> > >
> > > I intentionally left out:
> > >
> > >    - Entity deletes in MetaStoreChangeSet.
> > >    - Refactoring createCatalog, dropEntity, and renameEntity to use
> change
> > >    sets.
> > >    - NoSQL support for commitChangeSet.
> > >
> > > I am treating this as Phase 1 so the contract and the SPI primitive
> can be
> > > reviewed before the larger operation migrations land.
> > >
> > > Could you please take a look? In particular, I would like to know
> whether
> > > the contract captures the consensus so far, or if it needs to address
> retry
> > > / external-work coordination before we merge this foundation.
> > >
> > > Thanks,
> > > Prithvi S
> > >
> > > On Mon, Aug 17, 2026 at 7:40 PM Dmitri Bourlatchkov <[email protected]>
> > > wrote:
> > >
> > > > Hi Robert,
> > > >
> > > > I agree that the client-visible operation (e.g. REST API request)
> > > consists
> > > > of many distinct phases. The database / persistence changes are just
> one
> > > of
> > > > these phases. STS (as an example) is another. My point about
> transactions
> > > > in JDBC Persistence applies to the former (database changes) phase -
> one
> > > > retry attempt there should ideally involve exactly one transaction.
> > > >
> > > > We certainly need logic inside Polaris Servers to to coordinate
> requests
> > > to
> > > > various external systems depending on outcomes from previous request
> > > > processing phases.
> > > >
> > > > I also agree that validation based on database state should be
> repeated
> > > > from scratch if we retry the database changes.
> > > >
> > > > However, internal RBAC authorization naturally has to happen inside
> the
> > > > database update phase (read-check-write). IIRC, that is part of the
> > > > implicit consistency guarantees, which were discussed during early
> > > project
> > > > months (but never got written down in full clarity, unfortunately).
> If
> > > the
> > > > Authorization check is inside that phase for internal RBAC, I
> suppose it
> > > > will be there for other Authorizers too and that can involve external
> > > > systems (e.g. OPA / Ranger). We can certainly expect authorization
> to be
> > > > efficient, but I'm not sure how quick it can be in practice. There's
> > > > certainly potential for delays in some (perhaps infrequent) cases.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > >
> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp <[email protected]> wrote:
> > > >
> > > > > Hi Dmitri,
> > > > >
> > > > > I agree that one logical operation needs a consistent view of the
> whole
> > > > > backend state.
> > > > > What worries me is treating one long JDBC transaction as the
> boundary
> > > of
> > > > > that operation and then retrying the complete request if the commit
> > > > fails.
> > > > >
> > > > > The request can also write metadata to object storage or call STS.
> > > > > Each system can leave us with an outcome whose certainty we cannot
> > > know.
> > > > > The database may have committed before the connection was lost, an
> > > > > object-store write may have succeeded before a timeout, or STS may
> have
> > > > > issued credentials before its response was lost.
> > > > > A database rollback cannot undo any of those effects, and retrying
> the
> > > > > whole request may repeat them.
> > > > >
> > > > > So I think the contract has to separate the overall operation from
> each
> > > > > attempt to commit it.
> > > > > Each database or backend commit attempt should stay short.
> > > > > When we retry, validation and authorization need to use the state
> for
> > > > that
> > > > > new attempt, and we need clear rules for which external work can be
> > > > reused,
> > > > > repeated, cleaned up, or reconciled after an uncertain outcome.
> > > > >
> > > > > Some contextual state could be useful for carrying that state.
> > > > > But the JDBC connection and transaction should be an implementation
> > > > detail
> > > > > inside it, not the boundary of the complete operation.
> > > > >
> > > > > Cheers,
> > > > > Robert
> > > > >
> > > > >
> > > > > On Mon, Aug 10, 2026 at 11:04 PM Dmitri Bourlatchkov <
> [email protected]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > The idea of a backend-agnostic change-set primitive sounds good.
> > > > > >
> > > > > > However, I am not sure it is sufficient for all the use cases we
> > > > touched
> > > > > > here. More specifically, the state read by validation code is not
> > > > > > necessarily reflected in the change set:
> > > > > >
> > > > > > * Unchanged entities might be considered by validation, but
> changed
> > > in
> > > > a
> > > > > > parallel request.
> > > > > > * Even for changed entities the state read by validation may
> differ
> > > > from
> > > > > > the state being altered. Some validation code talks to the
> MetaStore
> > > > > > directly, outside of the data produced by the Resolver.
> > > > > >
> > > > > > I do not think Polaris offers any explicit mechanisms (ATM) to
> ensure
> > > > > > consistency between these reads and subsequent writes.
> > > > > >
> > > > > > As far as JDBC goes, running an overarching Transaction across
> all
> > > > reads
> > > > > > and writes in the same request, at the SERIALIZABLE isolation
> level
> > > in
> > > > > the
> > > > > > backing RDBMS could solve the problem, I think.
> > > > > >
> > > > > > Indeed, such a transaction might be long to accommodate calls
> made by
> > > > > > Polaris to external storage, etc. However, is that a problem? I
> think
> > > > the
> > > > > > alternative is for the JDBC Persistence impl. to "manually"
> track all
> > > > > reads
> > > > > > under the same request and redo them in the small transaction
> that
> > > > > persists
> > > > > > the writes. This will also add RDBMS overhead and require complex
> > > code
> > > > in
> > > > > > Polaris to handle the data properly.
> > > > > >
> > > > > > Tracking the request-wide transaction can be done only for JDBC
> > > without
> > > > > > leaking "transaction" concepts to the NoSQL Persistence, I think.
> > > NoSQL
> > > > > > will use other mechanisms to ensure read/write consistency.
> > > > > >
> > > > > > Connecting to Robert's email (a parallel branch in this
> discussion),
> > > > I'd
> > > > > > like to propose this approach:
> > > > > >
> > > > > > * Each request establishes a "Data Context"
> > > > > >   - In JDBC the Data Context corresponds to a JDBC Connection +
> Tx
> > > > > >   - In NoSQL the Data Context tracks one or more reference hashes
> > > > > > * All Persistence access in the same request goes through the
> same
> > > Data
> > > > > > Context
> > > > > > * All Persistence changes are committed once at the end of the
> > > request
> > > > > >   - Not all changes have to be globally atomic. For example,
> changes
> > > in
> > > > > > Catalogs A and B do not have to be atomic with respect to each
> other.
> > > > We
> > > > > > can go deeper into this later. This is relevant to NoSQL.
> > > > > >   - Transactional backends like JDBC can, of course, choose to
> make
> > > all
> > > > > > changes globally atomic.
> > > > > > * A commit can fail in two main ways:
> > > > > >   - A retriable failure like an optimistic lock error or Tx
> > > > > serializability
> > > > > > error
> > > > > >   - A non-triable logical error (e.g. entity not found)
> > > > > > * On a retriable error the whole request is re-attempted (as if
> > > > > resubmitted
> > > > > > by a client) a few times (configurable timeout).
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré <
> > > [email protected]
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all
> > > > > > >
> > > > > > > I think Robert has a good point.
> > > > > > >
> > > > > > > If the atomicity guarantee lives only in BasePersistence, then
> the
> > > > > > > manager contract can't tell a caller whether the state it read
> for
> > > > > > > validation/authorization/credential-vending is the same state
> that
> > > > > > > eventually commits. That's the actual bug class behind the JDBC
> > > > > > > symptoms (not any single operation being non-atomic, but the
> > > contract
> > > > > > > being silent about it. Every operation-specific fix (like
> #4939,
> > > > #5035
> > > > > > > or #5095) re-answers this question locally and it keeps
> recurring.
> > > So
> > > > > > > the deliverable should start with a written consistency
> contract at
> > > > > > > the manager level.
> > > > > > >
> > > > > > > I'm not sure migrating everyone to
> > > TransactionalMetaStoreManagerImpl
> > > > > > > is a good idea. It would tie the logical change set to a DB
> > > > > > > transaction spanning the REST request. It means:
> > > > > > > - it holds a durable transaction open across slow external work
> > > > > > > (credential vending, OPA, Ranger, ...)
> > > > > > > - it doesn't map to NoSQL
> > > > > > > - It wraps single-row updates in runWiithinTransaction, which
> is a
> > > > > > overhead
> > > > > > >
> > > > > > > So, I think the transactional manager isn't a portable target.
> It's
> > > > > > > "only" one backend's strategy.
> > > > > > >
> > > > > > > I think Privthi's approach is right. A backend-agnostic
> change-set
> > > > > > > primitive with a documented fallback is the correct shape.
> There is
> > > > > > > one caveat: without carrying the original entitiy (not just
> the new
> > > > > > > one) the change set can't express optimistic concurrency.
> > > > > > >
> > > > > > > I propose the following multi-steps approach:
> > > > > > > 1. We write the manager-level consistency contract as Robert
> asked.
> > > > It
> > > > > > > should include the explicit statement that a logical change
> set is
> > > > not
> > > > > > > a request-scope DB transaction.
> > > > > > > 2. We make Compare And Swap (optimistic-concurrency pattern)
> > > baseline
> > > > > > > first-class in EntityMutation before merging the SPI
> > > > > > > 3. Then we refactor createCatalog/dropEntity/renameEntity.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Regards
> > > > > > > JB
> > > > > > >
> > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp <[email protected]>
> > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Yufei's clarification about where the atomicity guarantee is
> > > > defined
> > > > > > > seems
> > > > > > > > important.
> > > > > > > > If it is a property of the lower-level BasePersistence
> contract
> > > > > rather
> > > > > > > than
> > > > > > > > the general PolarisMetaStoreManager contract, the general
> > > contract
> > > > > does
> > > > > > > not
> > > > > > > > tell callers whether the state used for validation,
> > > authorization,
> > > > or
> > > > > > > > credential vending is consistent with the change that
> eventually
> > > > > > commits.
> > > > > > > >
> > > > > > > > The current PRs suggest that operation-specific multi-object
> > > > methods
> > > > > > can
> > > > > > > > fix individual cases, while leaving the same contract
> question to
> > > > > recur
> > > > > > > for
> > > > > > > > each new case.
> > > > > > > >
> > > > > > > > I would also avoid defining a logical change set as a
> database
> > > > > > > transaction
> > > > > > > > around an entire REST request.
> > > > > > > > That would tie the contract to the backend and database, and
> > > could
> > > > > keep
> > > > > > > the
> > > > > > > > durable attempt open across slow external work.
> > > > > > > >
> > > > > > > > So is the choice really between the two current manager
> > > > > > implementations,
> > > > > > > or
> > > > > > > > do we first need to revisit the boundary and guarantees
> exposed
> > > to
> > > > > > their
> > > > > > > > callers?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Robert
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri Bourlatchkov <
> > > > > [email protected]
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Yufei,
> > > > > > > > >
> > > > > > > > > Thanks for the link. I stand corrected. The "one atomic
> change
> > > > per
> > > > > > > method"
> > > > > > > > > contract as defined in javadoc does apply only to
> > > BasePersistence
> > > > > > > > > and AtomicOperationMetaStoreManager (which delegates to
> > > > > > > BasePersistence).
> > > > > > > > >
> > > > > > > > > Note that all non-test Persistence implementations in the
> > > Polaris
> > > > > > > codebase
> > > > > > > > > extend those classes (which is probably why I was confused
> > > about
> > > > > > > atomicity
> > > > > > > > > expectations).
> > > > > > > > >
> > > > > > > > > However, this creates a gap in the Persistence SPI
> > > specification.
> > > > > If
> > > > > > > > > other PolarisMetaStoreManager implementations do not have
> to
> > > > comply
> > > > > > > with
> > > > > > > > > this principle, it will create a conceptual difficulty at
> call
> > > > > sites.
> > > > > > > How
> > > > > > > > > can PolarisMetaStoreManager callers reason about
> consistency
> > > and
> > > > > > > durability
> > > > > > > > > behaviours in general?
> > > > > > > > >
> > > > > > > > > I believe we need to address that as part of this
> discussion.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Thu, Jul 16, 2026 at 9:29 PM Yufei Gu <
> [email protected]
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > The MetaStore SPI is currently defined with the idea
> that
> > > one
> > > > > > > method
> > > > > > > > > call
> > > > > > > > > > means one atomic change.
> > > > > > > > > >
> > > > > > > > > > If the "MetaStore SPI" refers to the interface
> > > > > > > PolarisMetaStoreManager, I
> > > > > > > > > > don't think we've ever state each method to be atomic.
> We did
> > > > > > clarify
> > > > > > > > > > atomicity[1] in the interface BasePersistence though.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yufei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 16, 2026 at 11:27 AM Dmitri Bourlatchkov <
> > > > > > > [email protected]>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Yufei,
> > > > > > > > > > >
> > > > > > > > > > > I agree that JDBC transactions must be handled more
> > > > explicitly.
> > > > > > > > > However,
> > > > > > > > > > > I'm not sure that simply moving to
> > > > > > > TransactionalMetaStoreManagerImpl is
> > > > > > > > > > > sufficient.
> > > > > > > > > > >
> > > > > > > > > > > The MetaStore SPI is currently defined with the idea
> that
> > > one
> > > > > > > method
> > > > > > > > > call
> > > > > > > > > > > means one atomic change [1]. The "transactional"
> MetaStore
> > > > > impl.
> > > > > > is
> > > > > > > > > but a
> > > > > > > > > > > sub-case of that. It cannot alter the high-level
> contract.
> > > > > > > > > > >
> > > > > > > > > > > We could add SPI methods having multiple object
> parameters
> > > to
> > > > > > > represent
> > > > > > > > > > > grouped changes, but I am not sure it will be a sound
> > > design.
> > > > > > This
> > > > > > > will
> > > > > > > > > > > bloat the interface surfaces and require extra impl.
> effort
> > > > for
> > > > > > > each
> > > > > > > > > > > backend type. More importantly, adding multi-arg change
> > > > methods
> > > > > > > still
> > > > > > > > > > won't
> > > > > > > > > > > address the problem of reads being consistent with
> writes,
> > > > > > because
> > > > > > > each
> > > > > > > > > > > method call will still be independent regarding the
> data
> > > > stored
> > > > > > in
> > > > > > > the
> > > > > > > > > > > database.
> > > > > > > > > > >
> > > > > > > > > > > I tend to think we need to introduce a "change set" or
> > > > "atomic
> > > > > > > batch"
> > > > > > > > > > > concept to core Persistence and associate each REST API
> > > > request
> > > > > > > with
> > > > > > > > > one
> > > > > > > > > > > such change set, which will be committed (or rolled
> back)
> > > at
> > > > > the
> > > > > > > end of
> > > > > > > > > > the
> > > > > > > > > > > request. I believe Ayush mentioned a similar concept
> in PR
> > > > 4939
> > > > > > > [2]. In
> > > > > > > > > > > JDBC each change set will naturally be associated with
> an
> > > > RDBMS
> > > > > > > > > > > transaction. In NoSQL persistence, each atomic change
> set
> > > > will
> > > > > be
> > > > > > > > > > > associated with one CAS operation on the underlying
> > > database.
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > >
> https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r
> > > > > > > > > > >
> > > > > > > > > > > [2]
> > > > > > >
> https://github.com/apache/polaris/pull/4939#discussion_r3575719158
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Dmitri.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu <
> > > > [email protected]
> > > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for raising this, Dmitri.
> > > > > > > > > > > >
> > > > > > > > > > > > These are valid concerns, and they were already
> > > recognized
> > > > > when
> > > > > > > we
> > > > > > > > > > > > introduced JDBC persistence to Polaris. At that
> time, we
> > > > > chose
> > > > > > > to use
> > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC due to
> the
> > > > > > > simplicity. I
> > > > > > > > > > > think
> > > > > > > > > > > > most of the issues mentioned here can already be
> > > addressed
> > > > by
> > > > > > > > > > > > TransactionalMetaStoreManagerImpl.
> > > > > > > > > > > >
> > > > > > > > > > > > For example, rename is already wrapped in a
> transaction
> > > in
> > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. Similarly,
> catalog
> > > > > > > creation,
> > > > > > > > > > which
> > > > > > > > > > > > involves reading and creating multiple objects, is
> also
> > > > > > executed
> > > > > > > > > > within a
> > > > > > > > > > > > transaction [2].
> > > > > > > > > > > >
> > > > > > > > > > > > I see two possible directions:
> > > > > > > > > > > >
> > > > > > > > > > > >    1.
> > > > > > > > > > > >
> > > > > > > > > > > >    Modify AtomicOperationMetaStoreManager together
> with
> > > the
> > > > > > > > > persistence
> > > > > > > > > > > >    backends (such as JDBC) to provide the required
> > > > > consistency
> > > > > > > > > > guarantees
> > > > > > > > > > > > for
> > > > > > > > > > > >    specific operations, similar to what
> > > > > > > > > > TransactionalMetaStoreManagerImpl
> > > > > > > > > > > >    does.
> > > > > > > > > > > >    2.
> > > > > > > > > > > >
> > > > > > > > > > > >    Migrate the persistence backends (such as JDBC)
> to use
> > > > > > > > > > > >    TransactionalMetaStoreManagerImpl directly. We may
> > > have
> > > > to
> > > > > > > deal
> > > > > > > > > with
> > > > > > > > > > > >    transactional semantic mismatches across different
> > > > > > persistence
> > > > > > > > > > > backends.
> > > > > > > > > > > >    For example, we would likely avoid using JDBC's
> > > > > > > > > > `runWithinTransaction`
> > > > > > > > > > > > for
> > > > > > > > > > > >    single row updates, which adds additional
> overhead and
> > > > > > > complexity
> > > > > > > > > > > > without
> > > > > > > > > > > >    benefits.
> > > > > > > > > > > >
> > > > > > > > > > > > References:
> > > > > > > > > > > >
> > > > > > > > > > > >    1.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286
> > > > > > > > > > > >    2.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965
> > > > > > > > > > > >
> > > > > > > > > > > > Yufei
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri Bourlatchkov <
> > > > > > > > > [email protected]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ayush and Prithvi recently contributed a couple of
> > > > > > interesting
> > > > > > > PRs:
> > > > > > > > > > > > > [4939], [5035].
> > > > > > > > > > > > >
> > > > > > > > > > > > > It looks like people are starting to encounter
> > > > consistency
> > > > > > > issues
> > > > > > > > > in
> > > > > > > > > > > > > JDBC persistence.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The PRs provide valuable insight into the
> underlying
> > > > > issues.
> > > > > > > They
> > > > > > > > > > offer
> > > > > > > > > > > > > incremental fixes that can work. However, I
> believe it
> > > is
> > > > > > time
> > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > Polaris community to review and improve this area
> of
> > > the
> > > > > > > codebase
> > > > > > > > > > > > > holistically.
> > > > > > > > > > > > >
> > > > > > > > > > > > > By this, I mean finding a solution that can be
> applied
> > > to
> > > > > all
> > > > > > > > > > > > > persistence backends (in-memory, JDBC, NoSQL) and
> > > > addresses
> > > > > > > these
> > > > > > > > > > > > > aspects:
> > > > > > > > > > > > >
> > > > > > > > > > > > > * Supporting concurrent and consistent changes
> where
> > > the
> > > > > > > service
> > > > > > > > > > reads
> > > > > > > > > > > > >   and validates current catalog state, then
> commits a
> > > > > change
> > > > > > > (e.g.
> > > > > > > > > > > > >   name clashes during renames).
> > > > > > > > > > > > > * Supporting consistent but independent changes to
> RBAC
> > > > > > grants
> > > > > > > and
> > > > > > > > > > > > >   MetaStore entities. This independence is needed
> to
> > > > > support
> > > > > > > > > > > > >   external authorizers like OPA and Ranger.
> > > > > > > > > > > > > * Supporting atomic changes across multiple similar
> > > > > entities.
> > > > > > > > > > > > > * Supporting authorization-based filtering of list
> > > > > operations
> > > > > > > (cf.
> > > > > > > > > > > > >   [4831]).
> > > > > > > > > > > > > * Supporting credential-vending decisions that are
> > > rooted
> > > > > in
> > > > > > > the
> > > > > > > > > > > > >   exact state of the catalog.
> > > > > > > > > > > > > * Supporting server-side retries for transient
> > > > persistence
> > > > > > > failures
> > > > > > > > > > > > >   (e.g. RDBMS Tx serializability failures).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please share your comments and ideas.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [4831] https://github.com/apache/polaris/pull/4831
> > > > > > > > > > > > >
> > > > > > > > > > > > > [4939] https://github.com/apache/polaris/pull/4939
> > > > > > > > > > > > >
> > > > > > > > > > > > > [5035] https://github.com/apache/polaris/pull/5035
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Dmitri
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Reply via email to