Hi EJ, Thanks for sharing the PoC!
I'll try to explore that code, but cannot promise a quick turnaround :) Reviewing this may take me a couple of weeks (due to other priorities). Cheers, Dmitri. On Mon, Oct 5, 2026 at 4:47 PM EJ Wang <[email protected]> wrote: > One detail I omitted: the PoC also includes FoundationDB and Spanner > adapters alongside JDBC. They reuse the same business-aware Manager. The > adapters implement generic storage operations without table, catalog or > grant rules. > > This exercises both goals: addressing consistency problems such as > concurrent creation at overlapping table locations, and reusing that logic > across databases. The concurrency tests have run on FoundationDB, > PostgreSQL and CockroachDB. Spanner coverage currently uses the emulator, > with three concurrency tests skipped. > > These are targeted feasibility checks. Production validation and > integration with Polaris’s existing NoSQL backend remain follow-up work. > > -ej > > On Mon, Oct 5, 2026 at 1:36 PM EJ Wang <[email protected]> > wrote: > > > Hi Dmitri, all, > > > > Two requests can both check for overlapping table locations, see none, > and > > then create conflicting tables. We need the checks and the resulting > writes > > to be protected together. > > > > My proposal is to implement these rules once in shared Polaris code (the > > Manager). This business-aware layer checks them in the transaction that > > creates the table. Database adapters would implement a business-agnostic > > interface (Primitives) for generic reads, writes and commit. They would > not > > implement table, catalog or grant rules, so adding a database reuses the > > same Polaris behavior. > > > > This PoC assumes the database itself can protect the relevant reads and > > writes in one serializable transaction. Calls to object storage or > > credential services happen outside it. > > > > The Java PoC > > < > https://github.com/flyingImer/polaris/tree/prototype/durable-java-primitives/persistence/primitives > > > > reuses the existing transactional Manager and tests selected races, > > authorization retries and uncertain commits. It uses a temporary data > > layout and reads more data than we'd want in production. Existing NoSQL > > integration, production tuning and migration remain follow-up work. > > > > I'm not proposing an additional request-wide Persistence Session yet. > > Existing business-aware request code still owns authorization, external > > work and retries. > > > > The PoC allows an already-authorized metadata change to finish. A fresh > > request retry checks authorization again. An uncertain commit stops > > automatic retries and retains files that may already be referenced. These > > policies still need agreement. > > > > Would this separation be a useful starting point? > > > > -ej > > > > > > On Tue, 22 Sep 2026 19:08:49 -0400, Dmitri Bourlatchkov < > [email protected]> > > wrote: > > > > Hi EJ, > 1. Which backend assumptions should we design for? [...] I know > I > > used the term "transaction" a lot during the call, but the real challenge > > is about establishing an SPI with clear contracts that could be > implemented > > by JDBC and NoSQL persistence using the database features natural to each > > model. I am fairly confident that a JDBC impl. is possible with > > Serializable transaction. We should be able to solve all the problems > > raised in this thread. However, that will make in-memory caching > impossible > > because the database needs to see all the reads (and writes) in a Tx to > be > > able to ensure serializable guarantees. If a cache prevents a read from > > hitting the database, that read (e.g. storage config) will not be a > factor > > in serializability. This connects to my earlier question about the > > in-memory cache and JDBC (title: Usefulness of InMemoryEntityCache) Are > > different approaches possible with JDBC? Probably. What is involved? This > > depends on the SPI contract, I think. > 2. How do we protect the data > used > > by validation? [...] The point mentioned by Robert previously IIRC, is > that > > we need to ground all runtime decisions to a particular state of the > > catalog. If at commit time the state of the catalog in the database > shifts > > from what was used during validation, the request has to restart from > > scratch. Current JDBC Persistence does this only for relationships > between > > entities and grants and between versions of the same entity. > Relationships > > between table locations and storage config are not actualized for the > > purpose of state consistency tracking. From my POV we need to begin by > > formulating this Persistence state in the Persistence SPI terms. Then, > > think how we can implement that for JDBC. NoSQL Persistence already has a > > per-catalog state (not granular), so wiring the new SPI to NoSQL should > not > > be too hard, I hope. The state in the SPI does not have to be global to > the > > catalog (as in NoSQL), but it needs to be able to express the idea of > which > > elements were engaged during request processing so serve as input to > > Persistence impl. for cosistency guarantees. If Persistence caches data > in > > memory (for example), it will have to do so in a way that does not > violate > > consistency expectations. From my POV we could begin by defining a > > "Persistence Session" concept. Each request is 1:1 with a session. All > > reads and writes go through the session. The next step would be the JDBC > > impl., which probably needs some brainstorming... but I wonder if we > could > > reach a consensus at a high level first. WDYT? Thanks, Dmitri. On Tue, > Sep > > 22, 2026 at 5:55 PM EJ Wang <[email protected]> wrote: > > > Thanks for the recap, Dmitri. I'd like to follow up on the three points > I > > > raised during the sync. We didn't have much time to discuss them, so it > > > would help to confirm these before choosing an SPI design. > > 1. Which > > backend assumptions should we design for? > > Within one Polaris server > > instance, do we assume entities, grants and > secrets all use the same > > database backend, or must we support mixing > backends across them? > > Must > > every supported persistence implementation provide transactional > > > guarantees? Does the database itself need to provide transactions, or > can > > > Polaris build that capability on top of simpler database operations? > > > > I'm happy to work with a homogeneous assumption if that's the intended > > > scope. Existing NoSQL illustrates the second question: it can publish > > > changes to multiple objects under one reference using a single > conditional > > > pointer update. > > 2. How do we protect the data used by validation? > > > > > Your storage-configuration and location-overlap examples make the > problem > > > concrete. Consider these races: > > A table creation passes validation > > against the catalog's allowed locations, > then another request changes > > that configuration before the table is > committed. > Two table-creation > > requests each check for overlapping locations, both see > no overlap, and > > then create tables at conflicting locations. > > Checking only the record > > being written cannot cover these dependencies. The > first also depends > on > > the catalog configuration. The second depends on the > absence of another > > overlapping table. > > Could the persistence layer protect these reads > > together with the writes, > using backend-internal versions/tokens or > > database isolation? Shared code > would still need to identify the > required > > reads and checks. Could we keep > the mechanism for protecting them > inside > > each backend? > > 3. Business rules are duplicated across backend > > implementations > > Today, the TreeMap > < > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java#L446-L481 > > > > > and JDBC > < > > > > https://github.com/apache/polaris/blob/5de0c6a900a7fad77e7d0663222ce2ee23621917/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java#L1002-L1046 > > > > > implementations each load a secret, check its principal, apply the > > > > rotation/reset steps and write it back. > > Changing these business rules > > therefore requires keeping multiple > implementations in sync. A fix > > applied to one backend can be missed in > another, causing the same > Polaris > > operation to behave differently depending > on the database. > > My > > proposal is to implement each business workflow once, against a common > > > set of storage operations with defined consistency and failure > semantics. > > > The shared workflow should not need to know whether those operations are > > > > implemented through JDBC transactions, NoSQL reference CAS or another > > > mechanism. Each backend would implement the same storage contract, > keeping > > > its execution details internal. A new database backend could then reuse > > the > existing business workflows. > > Do these seem like reasonable > > starting points? We can then evaluate the SPI > against the original > > failure cases on both JDBC and existing NoSQL. > > -ej > > On Fri, Sep > 18, > > 2026 at 12:10 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > > > Side note: I just came across [5541], which shows a case where a failed > > > > > request leaves persisted side effects. > > > > This can be seen as a > coding > > mistake, of course. However, this is about > > executing one request as a > > single, atomic unit. This is something that > > probably needs > foundational > > support and coding patterns in Polaris code. > > > > Cheers, > > Dmitri. > > > > > [5541] https://github.com/apache/polaris/pull/5541 > > > > On Fri, Sep > > 18, 2026 at 3:02 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > > > > Hi All, > > > > > > Here's a recap of the Community Sync discussion > > as I remember it. I'm > > sure > > > my recollection is not complete, so > > please add your thoughts to this > > thread. > > > > > > * Polaris > > traditionally relies on entity version number to ensure > > > consistency > > of changes. > > > > > > This works well for internal RBAC grants, which > > also have version > numbers > > > cross-referenced at the Persistence > > layer. > > > > > > I do not think this works in more general cases like > > validating > locations > > > wrt catalog-level storage configuration. > > > > > > > > > Another difficult case involves location overlaps between > concurrent > > > > table > > > creations. > > > > > > * We talked about how Persistence > > calls related to RDBMS transactions > (in > > > the JDBC persistence > case). > > > > > > > > There were some concerns raised about whether we need to > bind > > > Persistence > > > API to transactions explicitly. > > > > > > I think it > is > > a valid concern. At the same time, the Persistence API > > > should be > > clear about consistency guarantees across all backend > > > > > implementations. I think this needs more attention now that the matter > > of > > > > > milti-entioty changes became prominent in [5035]. > > > > > > I > > personally believe that RDBMS-based Persistence in Polaris has to use > > > > > > Serializable transaction isolation. Consequently, the core code needs > > to > > > > > provide mechanisms to allow the Persistence implementation to > > properly > > > connect reads and writes from API requests to JDBC > > transactions... > > whether > > > we use "transation" as an explicit > terms > > in the SPI or not. > > > > > > Note that currently, the same API request > > performs reads and writes in > > > _separate_ RDBMS transactions (even > > separate connections). > > > > > > * We did not discuss Persistence SPI > > implications in the call, but > > > currently each Persistence call is > > assumed to be one distinct and > atomic > > > change (please correct me > if > > I'm wrong). > > > > > > Consequently, supporting new use cases involves > > introducing new SPI > > > methods with many parameters. This can be seen > in > > PR [5035]. This tends > > to > > > overcomplicate the SPI. > > > > > > > > Ideally, I think the SPI should be simplified to deal with multi-entity > > > > > > and single-entity changes in a coherent manner to avoid any ambiguity > > > > > about > > > which method the caller should use. > > > > > > [5035] > > https://github.com/apache/polaris/pull/5035 > > > > > > Cheers, > > > > > Dmitri. > > > > > > On Tue, Sep 15, 2026 at 11:06 AM Dmitri Bourlatchkov > < > > [email protected] > > > > > wrote: > > > > > >> Heads up: This discussion > > is on the Community Sync call agenda for > Sept > > >> 17. > > >> > > >> > > Interested parties, please plan to attend, if possible :) > > >> > > >> > > Cheers, > > >> Dmitri. > > >> > > >> On Wed, Sep 9, 2026 at 1:48 PM > Dmitri > > Bourlatchkov <[email protected]> > > >> wrote: > > >> > > >>> Hi JB, > > > > >>> > > >>> I've added an agenda item to the next community sync call for > > this > > (Sept > > >>> 17, 2026). > > >>> > > >>> Re: key points: did you > > mean a real GH discussion or dev email (this > > >>> thread)? Just double > > checking :) I'm fine with either approach. > > >>> > > >>> Cheers, > > > >>> > > Dmitri. > > >>> > > >>> On Wed, Sep 9, 2026 at 12:54 PM Jean-Baptiste > > Onofré < > [email protected]> > > >>> wrote: > > >>> > > >>>> Hi Dmitri, > > > > >>>> > > >>>> I agree on the need to align on these points, though I'm > > not > entirely > > >>>> sure a dedicated meeting is necessary. Let's > start > > by using some > time > > >>>> during the next community meeting to > discuss > > it. > > >>>> > > >>>> In the meantime, what if we outline the key points > in > > a GitHub > > >>>> Discussion first? We can then use that as a reference > > during the > > >>>> meeting. > > >>>> > > >>>> Regards, > > >>>> JB > > > > >>>> > > >>>> On Wed, Sep 9, 2026 at 12:50 AM Dmitri Bourlatchkov < > > > [email protected] > > > > > >>>> wrote: > > >>>> > > > >>>> > Hi All, > > > > >>>> > > > >>>> > My impression from this thread is that it might be time > > for a call > > to > > >>>> > discuss all the related issues and try to > > arrive at a shared > > >>>> implementation > > >>>> > plan. > > >>>> > > > > > > >>>> > I know meetings are not ideal, but in this case it might be > > > >>>> > > beneficial as a > > >>>> > means for achieving a common understanding of > > the set of problems > > and > > >>>> > priorities related to this > thread. A > > dedicated metrics meeting > > worked > > >>>> well > > >>>> > from my > POV. > > > > >>>> > > > >>>> > Allocating a time slot in the next community > meeting > > might be an > > >>>> option, > > >>>> > although I think we might need > the > > full hour in this case. > > >>>> > > > >>>> > WDYT? > > >>>> > > > >>>> > > > Thanks, > > >>>> > Dmitri. > > >>>> > > > >>>> > On Mon, Aug 17, 2026 at > > 6:39 PM Prithvi S < > > >>>> [email protected]> > > >>>> > > > wrote: > > >>>> > > > >>>> > > Hi all, > > >>>> > > > > >>>> > > I > rewrote > > https://github.com/apache/polaris/pull/5035, > following > > >>>> the > > > > > >>>> > > discussion to focus only on the SPI foundation. > > >>>> > > > > > > >>>> > > What is in the PR now: > > >>>> > > > > >>>> > > - A written > > manager-level consistency contract in > > >>>> > > > > site/content/in-dev/unreleased/persistence-consistency.md. > > >>>> > > - > > CAS-aware EntityMutation / GrantMutation records. > > >>>> > > - > > MetaStoreChangeSet and BasePersistence#commitChangeSet, > with > > >>>> > > backends > > >>>> > > opting in via supportsAtomicMixedCommit(). > > > >>>> > > > > - commitChangeSet implementations for the in-memory TreeMap > > >>>> > > backend and > > >>>> > > JDBC. > > >>>> > > - > > AtomicOperationMetaStoreManager and > > >>>> > > TransactionalMetaStoreManagerImpl > > >>>> > > grant/revoke paths now > build > > one change set per operation, > > >>>> falling back > > >>>> > > to > > > > >>>> > > individual operations when the backend does not support mixed > > > > > >>>> commits. > > >>>> > > - Tests for the change-set API and TreeMap > > atomic commits. > > >>>> > > > > >>>> > > I intentionally left out: > > > > >>>> > > > > >>>> > > - Entity deletes in MetaStoreChangeSet. > > >>>> > > > > > - Refactoring createCatalog, dropEntity, and renameEntity to > > use > > > > >>>> change > > >>>> > > sets. > > >>>> > > - NoSQL support for > > commitChangeSet. > > >>>> > > > > >>>> > > I am treating this as Phase 1 > so > > the contract and the SPI > > primitive > > >>>> can be > > >>>> > > > > reviewed before the larger operation migrations land. > > >>>> > > > > > >>>> > > > > Could you please take a look? In particular, I would like to > know > > > > > >>>> whether > > >>>> > > the contract captures the consensus so far, or > if > > it needs to > > >>>> address retry > > >>>> > > / external-work > > coordination before we merge this foundation. > > >>>> > > > > >>>> > > > > Thanks, > > >>>> > > Prithvi S > > >>>> > > > > >>>> > > On Mon, Aug 17, > > 2026 at 7:40 PM Dmitri Bourlatchkov < > > >>>> [email protected]> > > > >>>> > > > > wrote: > > >>>> > > > > >>>> > > > Hi Robert, > > >>>> > > > > > > >>>> > > > > > I agree that the client-visible operation (e.g. REST API > > > request) > > > > >>>> > > consists > > >>>> > > > of many distinct phases. The database > / > > persistence changes > are > > >>>> just one > > >>>> > > of > > >>>> > > > > > > these phases. STS (as an example) is another. My point about > > >>>> > > transactions > > >>>> > > > in JDBC Persistence applies to the former > > (database changes) > > >>>> phase - one > > >>>> > > > retry attempt > there > > should ideally involve exactly one > > >>>> transaction. > > >>>> > > > > > > > > >>>> > > > We certainly need logic inside Polaris Servers to to > > > coordinate > > >>>> requests > > >>>> > > to > > >>>> > > > various > > external systems depending on outcomes from previous > > >>>> request > > > > >>>> > > > processing phases. > > >>>> > > > > > >>>> > > > I also agree > > that validation based on database state should be > > >>>> repeated > > > > >>>> > > > from scratch if we retry the database changes. > > >>>> > > > > > > > > >>>> > > > However, internal RBAC authorization naturally has to > happen > > > > >>>> inside the > > >>>> > > > database update phase > (read-check-write). > > IIRC, that is part > of > > >>>> the > > >>>> > > > implicit consistency > > guarantees, which were discussed during > > early > > >>>> > > project > > > > > >>>> > > > months (but never got written down in full clarity, > > >>>> > > unfortunately). If > > >>>> > > the > > >>>> > > > Authorization check is > > inside that phase for internal RBAC, I > > >>>> suppose it > > >>>> > > > > > will be there for other Authorizers too and that can involve > > >>>> > > external > > >>>> > > > systems (e.g. OPA / Ranger). We can certainly > > expect > > >>>> authorization to be > > >>>> > > > efficient, but I'm not > > sure how quick it can be in practice. > > >>>> There's > > >>>> > > > > > certainly potential for delays in some (perhaps infrequent) > > cases. > > > > > >>>> > > > > > >>>> > > > Cheers, > > >>>> > > > Dmitri. > > >>>> > > > > > > > > >>>> > > > > > >>>> > > > On Mon, Aug 17, 2026 at 9:20 AM Robert Stupp < > > [email protected]> > > >>>> wrote: > > >>>> > > > > > >>>> > > > > Hi > > Dmitri, > > >>>> > > > > > > >>>> > > > > I agree that one logical > > operation needs a consistent view > of > > >>>> the whole > > >>>> > > > > > > > backend state. > > >>>> > > > > What worries me is treating one long JDBC > > transaction as the > > >>>> boundary > > >>>> > > of > > >>>> > > > > > that > > operation and then retrying the complete request if the > > >>>> commit > > > > > >>>> > > > fails. > > >>>> > > > > > > >>>> > > > > The request can also > > write metadata to object storage or > call > > >>>> STS. > > >>>> > > > > > > Each system can leave us with an outcome whose certainty we > > >>>> > cannot > > > > >>>> > > know. > > >>>> > > > > The database may have committed > before > > the connection was > > lost, > > >>>> an > > >>>> > > > > object-store > > write may have succeeded before a timeout, or > STS > > >>>> may have > > > > >>>> > > > > issued credentials before its response was lost. > > >>>> > > > > > > > A database rollback cannot undo any of those effects, and > > >>>> > > retrying the > > >>>> > > > > whole request may repeat them. > > >>>> > > > > > > > > > >>>> > > > > So I think the contract has to separate the overall > > > operation > > >>>> from each > > >>>> > > > > attempt to commit it. > > > > >>>> > > > > Each database or backend commit attempt should stay short. > > > > > >>>> > > > > When we retry, validation and authorization need to use the > > > > > >>>> state for > > >>>> > > > that > > >>>> > > > > new attempt, and we > > need clear rules for which external work > > >>>> can be > > >>>> > > > > > reused, > > >>>> > > > > repeated, cleaned up, or reconciled after an > > uncertain > > outcome. > > >>>> > > > > > > >>>> > > > > Some contextual > > state could be useful for carrying that > state. > > >>>> > > > > But the > > JDBC connection and transaction should be an > > >>>> implementation > > > > >>>> > > > detail > > >>>> > > > > inside it, not the boundary of the > > complete operation. > > >>>> > > > > > > >>>> > > > > Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > On Mon, Aug > > 10, 2026 at 11:04 PM Dmitri Bourlatchkov < > > >>>> [email protected] > > > > >>>> > > > > > >>>> > > > > wrote: > > >>>> > > > > > > >>>> > > > > > Hi > > All, > > >>>> > > > > > > > >>>> > > > > > The idea of a backend-agnostic > > change-set primitive sounds > > >>>> good. > > >>>> > > > > > > > >>>> > > > > > > > > However, I am not sure it is sufficient for all the use > > cases > > > > > >>>> we > > >>>> > > > touched > > >>>> > > > > > here. More > specifically, > > the state read by validation code > > is > > >>>> not > > >>>> > > > > > > > necessarily reflected in the change set: > > >>>> > > > > > > > >>>> > > > > > > > > * Unchanged entities might be considered by validation, > but > > > >>>> > > changed > > >>>> > > in > > >>>> > > > a > > >>>> > > > > > parallel > > request. > > >>>> > > > > > * Even for changed entities the state read by > > validation > may > > >>>> differ > > >>>> > > > from > > >>>> > > > > > > the > > state being altered. Some validation code talks to the > > >>>> > MetaStore > > > > >>>> > > > > > directly, outside of the data produced by the Resolver. > > > > > >>>> > > > > > > > >>>> > > > > > I do not think Polaris offers any > > explicit mechanisms > (ATM) > > >>>> to ensure > > >>>> > > > > > > > consistency between these reads and subsequent writes. > > >>>> > > > > > > > > > > >>>> > > > > > As far as JDBC goes, running an overarching Transaction > > > > > >>>> across all > > >>>> > > > reads > > >>>> > > > > > and writes in > the > > same request, at the SERIALIZABLE > > isolation > > >>>> level > > >>>> > > > > > in > > >>>> > > > > the > > >>>> > > > > > backing RDBMS could solve the > > problem, I think. > > >>>> > > > > > > > >>>> > > > > > Indeed, such a > > transaction might be long to accommodate > > calls > > >>>> made by > > > > >>>> > > > > > Polaris to external storage, etc. However, is that a > > > > problem? > > >>>> I think > > >>>> > > > the > > >>>> > > > > > > alternative > > is for the JDBC Persistence impl. to > "manually" > > >>>> track all > > > > >>>> > > > > reads > > >>>> > > > > > under the same request and redo > them > > in the small > > transaction > > >>>> that > > >>>> > > > > persists > > > > >>>> > > > > > the writes. This will also add RDBMS overhead and require > > > > > >>>> complex > > >>>> > > code > > >>>> > > > in > > >>>> > > > > > > > Polaris to handle the data properly. > > >>>> > > > > > > > >>>> > > > > > > > > Tracking the request-wide transaction can be done only for > > >>>> JDBC > > > > > >>>> > > without > > >>>> > > > > > leaking "transaction" concepts to > the > > NoSQL Persistence, I > > >>>> think. > > >>>> > > NoSQL > > >>>> > > > > > > > > will use other mechanisms to ensure read/write > consistency. > > >>>> > > > > > > > > > > >>>> > > > > > Connecting to Robert's email (a parallel branch > in > > this > > >>>> discussion), > > >>>> > > > I'd > > >>>> > > > > > like to > > propose this approach: > > >>>> > > > > > > > >>>> > > > > > * Each > request > > establishes a "Data Context" > > >>>> > > > > > - In JDBC the Data > Context > > corresponds to a JDBC > > Connection > > >>>> + Tx > > >>>> > > > > > - > In > > NoSQL the Data Context tracks one or more reference > > >>>> hashes > > > > >>>> > > > > > * All Persistence access in the same request goes through > > > > > the > > >>>> same > > >>>> > > Data > > >>>> > > > > > Context > > > >>>> > > > > > > > * All Persistence changes are committed once at the end of > > > the > > > > >>>> > > request > > >>>> > > > > > - Not all changes have to be > > globally atomic. For > example, > > >>>> changes > > >>>> > > in > > > >>>> > > > > > > > Catalogs A and B do not have to be atomic with respect to > > > >>>> > > each other. > > >>>> > > > We > > >>>> > > > > > can go deeper into this > > later. This is relevant to NoSQL. > > >>>> > > > > > - Transactional > > backends like JDBC can, of course, > choose > > >>>> to make > > >>>> > > > > all > > >>>> > > > > > changes globally atomic. > > >>>> > > > > > * A > > commit can fail in two main ways: > > >>>> > > > > > - A retriable > failure > > like an optimistic lock error or > Tx > > >>>> > > > > serializability > > > > > >>>> > > > > > error > > >>>> > > > > > - A non-triable logical error > (e.g. > > entity not found) > > >>>> > > > > > * On a retriable error the whole > > request is re-attempted > (as > > >>>> if > > >>>> > > > > resubmitted > > > > > >>>> > > > > > by a client) a few times (configurable timeout). > > >>>> > > > > > > > > > > >>>> > > > > > WDYT? > > >>>> > > > > > > > >>>> > > > > > > > Cheers, > > >>>> > > > > > Dmitri. > > >>>> > > > > > > > >>>> > > > > > > On > > Sun, Jul 26, 2026 at 12:44 PM Jean-Baptiste Onofré < > > >>>> > > > > [email protected] > > >>>> > > > > > > >>>> > > > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > Hi all > > >>>> > > > > > > > > >>>> > > > > > > > I > > think Robert has a good point. > > >>>> > > > > > > > > >>>> > > > > > > > If > > the atomicity guarantee lives only in > BasePersistence, > > >>>> then > the > > > > >>>> > > > > > > manager contract can't tell a caller whether the > state > > > it > > >>>> read for > > >>>> > > > > > > > > validation/authorization/credential-vending is the same > > >>>> state > that > > > > >>>> > > > > > > eventually commits. That's the actual bug class > behind > > > the > > >>>> JDBC > > >>>> > > > > > > symptoms (not any single > operation > > being non-atomic, but > > the > > >>>> > > contract > > >>>> > > > > > > > > being silent about it. Every operation-specific fix > (like > > >>>> > #4939, > > > > >>>> > > > #5035 > > >>>> > > > > > > or #5095) re-answers this > > question locally and it keeps > > >>>> recurring. > > >>>> > > So > > > >>>> > > > > > > > > the deliverable should start with a written consistency > > > > >>>> contract at > > >>>> > > > > > > the manager level. > > >>>> > > > > > > > > > > > >>>> > > > > > > I'm not sure migrating everyone to > > >>>> > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > is a good idea. It > > would tie the logical change set to a > > DB > > >>>> > > > > > > > > transaction spanning the REST request. It means: > > >>>> > > > > > > - > it > > holds a durable transaction open across slow > external > > >>>> work > > > > >>>> > > > > > > (credential vending, OPA, Ranger, ...) > > >>>> > > > > > > > > > - it doesn't map to NoSQL > > >>>> > > > > > > - It wraps single-row > > updates in runWiithinTransaction, > > >>>> which is a > > >>>> > > > > > > > overhead > > >>>> > > > > > > > > >>>> > > > > > > So, I think the > > transactional manager isn't a portable > > >>>> target. It's > > >>>> > > > > > > > > > "only" one backend's strategy. > > >>>> > > > > > > > > >>>> > > > > > > > > > I think Privthi's approach is right. A backend-agnostic > > >>>> > > change-set > > >>>> > > > > > > primitive with a documented fallback is > the > > correct > shape. > > >>>> There is > > >>>> > > > > > > one caveat: > without > > carrying the original entitiy (not > > just > > >>>> the new > > >>>> > > > > > > > > > one) the change set can't express optimistic > concurrency. > > > >>>> > > > > > > > > > > >>>> > > > > > > I propose the following multi-steps > > approach: > > >>>> > > > > > > 1. We write the manager-level consistency > > contract as > > >>>> Robert asked. > > >>>> > > > It > > >>>> > > > > > > > > should include the explicit statement that a logical > > change > > >>>> > > set is > > >>>> > > > not > > >>>> > > > > > > a request-scope DB > > transaction. > > >>>> > > > > > > 2. We make Compare And Swap > > (optimistic-concurrency > > pattern) > > >>>> > > baseline > > >>>> > > > > > > > > > first-class in EntityMutation before merging the SPI > > >>>> > > > > > > > > > 3. Then we refactor > createCatalog/dropEntity/renameEntity. > > >>>> > > > > > > > > > > > >>>> > > > > > > Thoughts? > > >>>> > > > > > > > > >>>> > > > > > > > > > Regards > > >>>> > > > > > > JB > > >>>> > > > > > > > > >>>> > > > > > > > > > On Fri, Jul 24, 2026 at 5:08 PM Robert Stupp < > > >>>> > [email protected]> > > > > >>>> > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > Hi > all, > > > > >>>> > > > > > > > > > >>>> > > > > > > > Yufei's clarification about > > where the atomicity > > guarantee > > >>>> is > > >>>> > > > defined > > > > >>>> > > > > > > seems > > >>>> > > > > > > > important. > > >>>> > > > > > > > > > > If it is a property of the lower-level BasePersistence > > >>>> > > contract > > >>>> > > > > rather > > >>>> > > > > > > than > > >>>> > > > > > > > > > > the general PolarisMetaStoreManager contract, the > > general > > > > >>>> > > contract > > >>>> > > > > does > > >>>> > > > > > > not > > > >>>> > > > > > > > > > tell callers whether the state used for validation, > > >>>> > > > > > authorization, > > >>>> > > > or > > >>>> > > > > > > > credential > > vending is consistent with the change that > > >>>> eventually > > >>>> > > > > > > > > commits. > > >>>> > > > > > > > > > >>>> > > > > > > > The current > > PRs suggest that operation-specific > > >>>> multi-object > > >>>> > > > > > methods > > >>>> > > > > > can > > >>>> > > > > > > > fix individual > cases, > > while leaving the same contract > > >>>> question to > > >>>> > > > > > recur > > > > >>>> > > > > > > for > > >>>> > > > > > > > each new case. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > I would also avoid defining a logical > > change set as a > > >>>> database > > >>>> > > > > > > transaction > > > >>>> > > > > > > > > > around an entire REST request. > > >>>> > > > > > > > That > > would tie the contract to the backend and > database, > > >>>> and > > > >>>> > > > > could > > >>>> > > > > keep > > >>>> > > > > > > the > > >>>> > > > > > > > > > > durable attempt open across slow external work. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > So is the choice really between the two current > > > manager > > >>>> > > > > > implementations, > > >>>> > > > > > > or > > > > >>>> > > > > > > > do we first need to revisit the boundary and > > > guarantees > > >>>> exposed > > >>>> > > to > > >>>> > > > > > their > > > > >>>> > > > > > > > callers? > > >>>> > > > > > > > > > >>>> > > > > > > > > > Cheers, > > >>>> > > > > > > > Robert > > >>>> > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > On Fri, Jul 17, 2026 at 10:21 PM Dmitri > > Bourlatchkov < > > >>>> > > > > [email protected] > > >>>> > > > > > > > > > > > >>>> > > > > > > > wrote: > > >>>> > > > > > > > > > >>>> > > > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > >>>> > > > > > > > > Thanks for > the > > link. I stand corrected. The "one > > atomic > > >>>> change > > >>>> > > > > > > per > > >>>> > > > > > > method" > > >>>> > > > > > > > > contract as > > defined in javadoc does apply only to > > >>>> > > BasePersistence > > > >>>> > > > > > > > > > > and AtomicOperationMetaStoreManager (which delegates > > > to > > > > >>>> > > > > > > BasePersistence). > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > > Note that all non-test Persistence implementations > in > > > > >>>> the > > >>>> > > Polaris > > >>>> > > > > > > codebase > > >>>> > > > > > > > > > > > extend those classes (which is probably why I was > > >>>> > > confused > > >>>> > > about > > >>>> > > > > > > atomicity > > >>>> > > > > > > > > > > > expectations). > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > However, this creates a gap in the Persistence SPI > > >>>> > > > > specification. > > >>>> > > > > If > > >>>> > > > > > > > > other > > PolarisMetaStoreManager implementations do not > > >>>> have to > > >>>> > > > > > > comply > > >>>> > > > > > > with > > >>>> > > > > > > > > this > > principle, it will create a conceptual > difficulty > > >>>> at call > > > > >>>> > > > > sites. > > >>>> > > > > > > How > > >>>> > > > > > > > > can > > PolarisMetaStoreManager callers reason about > > >>>> consistency > > > >>>> > > > > and > > >>>> > > > > > > durability > > >>>> > > > > > > > > > behaviours > > in general? > > >>>> > > > > > > > > > > >>>> > > > > > > > > I believe > we > > need to address that as part of this > > >>>> discussion. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > Cheers, > > >>>> > > > > > > > > > Dmitri. > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > On Thu, Jul 16, 2026 at > > 9:29 PM Yufei Gu < > > >>>> [email protected] > > >>>> > > > > > >>>> > > > > > > > wrote: > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > > The > > MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > > > > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > > > > > > > > > > > > means one atomic change. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > If the "MetaStore SPI" refers to the interface > > > >>>> > > > > > > > > PolarisMetaStoreManager, I > > >>>> > > > > > > > > > don't > think > > we've ever state each method to be > > >>>> atomic. We did > > >>>> > > > > > > > > clarify > > >>>> > > > > > > > > > atomicity[1] in the interface > > BasePersistence > > though. > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > 1. > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > > > https://github.com/apache/polaris/blob/e9039e12003a13e783b5130a3d30d30cfe78d93c/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java#L48 > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > Yufei > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > On Thu, Jul 16, 2026 at > > 11:27 AM Dmitri > > Bourlatchkov < > > >>>> > > > > > > [email protected] > > > > > > >>>> > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > > > Hi Yufei, > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > I agree that JDBC transactions must be handled > > more > > >>>> > > > > > > explicitly. > > >>>> > > > > > > > > However, > > >>>> > > > > > > > > > > > > > I'm not sure that simply moving to > > >>>> > > > > > > > > TransactionalMetaStoreManagerImpl is > > >>>> > > > > > > > > > > > > sufficient. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > The > > MetaStore SPI is currently defined with the > > >>>> idea that > > >>>> > > > > > one > > >>>> > > > > > > method > > >>>> > > > > > > > > call > > >>>> > > > > > > > > > > > > > means one atomic change [1]. The "transactional" > > >>>> > > MetaStore > > >>>> > > > > impl. > > >>>> > > > > > is > > >>>> > > > > > > > > > > > but a > > >>>> > > > > > > > > > > sub-case of that. It cannot alter > > the high-level > > >>>> contract. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > We could add SPI methods having multiple object > > >>>> > > parameters > > >>>> > > to > > >>>> > > > > > > represent > > >>>> > > > > > > > > > > > > > grouped changes, but I am not sure it will be a > > >>>> > sound > > > > >>>> > > design. > > >>>> > > > > > This > > >>>> > > > > > > will > > > > > >>>> > > > > > > > > > > bloat the interface surfaces and require extra > > > > > >>>> impl. effort > > >>>> > > > for > > >>>> > > > > > > each > > >>>> > > > > > > > > > > > > > backend type. More importantly, adding multi-arg > > >>>> > > change > > >>>> > > > methods > > >>>> > > > > > > still > > >>>> > > > > > > > > > > > > won't > > >>>> > > > > > > > > > > address the problem of reads > > being consistent > with > > >>>> writes, > > >>>> > > > > > because > > > > >>>> > > > > > > each > > >>>> > > > > > > > > > > method call will still > > be independent regarding > > the > > >>>> data > > >>>> > > > stored > > > > >>>> > > > > > in > > >>>> > > > > > > the > > >>>> > > > > > > > > > > > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > I > tend > > to think we need to introduce a "change > > set" > > >>>> or > > >>>> > > > > > > "atomic > > >>>> > > > > > > batch" > > >>>> > > > > > > > > > > concept > to > > core Persistence and associate each > > REST > > >>>> API > > >>>> > > > > > request > > >>>> > > > > > > with > > >>>> > > > > > > > > one > > >>>> > > > > > > > > > > > > > such change set, which will be committed (or > > rolled > > > > > >>>> back) > > >>>> > > at > > >>>> > > > > the > > >>>> > > > > > > end > of > > > > >>>> > > > > > > > > > the > > >>>> > > > > > > > > > > request. I > > believe Ayush mentioned a similar > > >>>> concept in PR > > >>>> > > > > > 4939 > > >>>> > > > > > > [2]. In > > >>>> > > > > > > > > > > JDBC each > > change set will naturally be > associated > > >>>> with an > > >>>> > > > > > RDBMS > > >>>> > > > > > > > > > > transaction. In NoSQL persistence, > each > > atomic > > >>>> change set > > >>>> > > > will > > >>>> > > > > be > > > >>>> > > > > > > > > > > > > associated with one CAS operation on the > > > underlying > > > > >>>> > > database. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > [1] > > >>>> > > > > > > > > >>>> > > https://lists.apache.org/thread/rf5orxs815zs4h64p4rwp03q3pbgxb5r > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > [2] > > >>>> > > > > > > > > > > > >>>> > https://github.com/apache/polaris/pull/4939#discussion_r3575719158 > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > Cheers, > > > >>>> > > > > > > > > > > > > Dmitri. > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > > On Thu, Jul 16, 2026 at 1:12 PM Yufei Gu < > > >>>> > > > > > [email protected] > > >>>> > > > > > > > >>>> > > > > > > wrote: > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > Thanks for > raising > > this, Dmitri. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > These are valid concerns, and they were > already > > >>>> > > > recognized > > > > >>>> > > > > when > > >>>> > > > > > > we > > >>>> > > > > > > > > > > > > > > introduced JDBC persistence to Polaris. At > that > > >>>> time, we > > > > >>>> > > > > chose > > >>>> > > > > > > to use > > >>>> > > > > > > > > > > > > > > AtomicOperationMetaStoreManager for the JDBC > due > > >>>> to the > > > > >>>> > > > > > > simplicity. I > > >>>> > > > > > > > > > > think > > > >>>> > > > > > > > > > > > > > most of the issues mentioned here can already > be > > > > > >>>> > > addressed > > >>>> > > > by > > >>>> > > > > > > > > > > > > > TransactionalMetaStoreManagerImpl. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > For example, rename is already wrapped in a > > > >>>> > > transaction > > >>>> > > in > > >>>> > > > > > > > > > > > > > TransactionalMetaStoreManagerImpl [1]. > > Similarly, > > >>>> catalog > > > > > >>>> > > > > > > creation, > > >>>> > > > > > > > > > which > > >>>> > > > > > > > > > > > > > > involves reading and creating multiple > objects, > > > >>>> > > is also > > >>>> > > > > > executed > > >>>> > > > > > > > > > within a > > > > > >>>> > > > > > > > > > > > transaction [2]. > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > I see two possible directions: > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Modify > > AtomicOperationMetaStoreManager > > >>>> together with > > >>>> > > the > > > > > >>>> > > > > > > > > persistence > > >>>> > > > > > > > > > > > backends > > (such as JDBC) to provide the > > required > > >>>> > > > > consistency > > > > > >>>> > > > > > > > > > guarantees > > >>>> > > > > > > > > > > > for > > > > >>>> > > > > > > > > > > > specific operations, similar to what > > >>>> > > > > > > > > > > > > TransactionalMetaStoreManagerImpl > > >>>> > > > > > > > > > > > > > > does. > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > Migrate the persistence backends (such > as > > > > >>>> JDBC) to use > > >>>> > > > > > > > > > > > > > TransactionalMetaStoreManagerImpl directly. > > We > > >>>> may > > >>>> > > > > > have > > >>>> > > > to > > >>>> > > > > > > deal > > >>>> > > > > > > > > > > > with > > >>>> > > > > > > > > > > > transactional semantic mismatches > > across > > >>>> different > > >>>> > > > > > persistence > > >>>> > > > > > > > > > > > > > backends. > > >>>> > > > > > > > > > > > For example, we would > > likely avoid using > > JDBC's > > >>>> > > > > > > > > > > > `runWithinTransaction` > > >>>> > > > > > > > > > > > for > > >>>> > > > > > > > > > > > > > > single row updates, which adds additional > > >>>> overhead > > and > > >>>> > > > > > > complexity > > >>>> > > > > > > > > > > > > without > > > > >>>> > > > > > > > > > > > benefits. > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > References: > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > 1. > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > >>>> > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1286 > > > > >>>> > > > > > > > > > > > 2. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > > > > https://github.com/apache/polaris/blob/5731c5cbee02257d1f21f78ca3befcd639b100a3/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L965 > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > Yufei > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > On Thu, Jul 16, 2026 at 8:22 AM Dmitri > > >>>> > Bourlatchkov > > < > > >>>> > > > > > > > > [email protected]> > > >>>> > > > > > > > > > > > > > > wrote: > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > Hi > > all, > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > Ayush and Prithvi recently contributed a > > couple > > >>>> of > > >>>> > > > > > > > > interesting > > >>>> > > > > > > PRs: > > >>>> > > > > > > > > > > > > > > > [4939], [5035]. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > It looks like people are starting to > encounter > > >>>> > > > > > consistency > > >>>> > > > > > > issues > > >>>> > > > > > > > > in > > > > >>>> > > > > > > > > > > > > JDBC persistence. > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > The PRs provide valuable insight > into > > the > > >>>> underlying > > >>>> > > > > issues. > > >>>> > > > > > > > They > > > > >>>> > > > > > > > > > offer > > >>>> > > > > > > > > > > > > > > incremental fixes that can work. However, I > > >>>> believe it > > >>>> > > > > > is > > >>>> > > > > > time > > >>>> > > > > > > for > > >>>> > > > > > > > > > > > > the > > >>>> > > > > > > > > > > > > Polaris community to review > and > > improve this > > >>>> area of > > >>>> > > the > > >>>> > > > > > > > > codebase > > >>>> > > > > > > > > > > > > holistically. > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > By this, I mean finding a > > solution that can > be > > >>>> applied > > >>>> > > to > > >>>> > > > > > > all > > >>>> > > > > > > > > > > > > persistence backends (in-memory, > JDBC, > > > NoSQL) > > >>>> and > > >>>> > > > addresses > > >>>> > > > > > > > these > > > > >>>> > > > > > > > > > > > > aspects: > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > * Supporting concurrent and consistent > > > changes > > >>>> where > > >>>> > > the > > >>>> > > > > > > service > > > > >>>> > > > > > > > > > reads > > >>>> > > > > > > > > > > > > and > validates > > current catalog state, then > > >>>> commits a > > >>>> > > > > change > > > > > >>>> > > > > > > (e.g. > > >>>> > > > > > > > > > > > > name clashes > during > > renames). > > >>>> > > > > > > > > > > > > * Supporting consistent but > > independent > > changes > > >>>> to RBAC > > >>>> > > > > > grants > > > >>>> > > > > > > > > and > > >>>> > > > > > > > > > > > > MetaStore entities. This > > independence is > > >>>> needed to > > >>>> > > > > support > > >>>> > > > > > > > > > > > > > > > external authorizers like OPA and Ranger. > > >>>> > > > > > > > > > > > > > > > * Supporting atomic changes across multiple > > >>>> > > similar > > >>>> > > > > entities. > > >>>> > > > > > > > > > > > > * > > Supporting authorization-based filtering > of > > >>>> list > > >>>> > > > > > > > operations > > >>>> > > > > > > (cf. > > >>>> > > > > > > > > > > > > > > [4831]). > > >>>> > > > > > > > > > > > > * Supporting credential-vending > > decisions > that > > >>>> are > > >>>> > > rooted > > >>>> > > > > in > > > > >>>> > > > > > > the > > >>>> > > > > > > > > > > > > exact state of the > > catalog. > > >>>> > > > > > > > > > > > > * Supporting server-side > retries > > for > transient > > >>>> > > > persistence > > >>>> > > > > > > failures > > > > > >>>> > > > > > > > > > > > > (e.g. RDBMS Tx serializability failures). > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Please > > share your comments and ideas. > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > [4831] > > >>>> > > https://github.com/apache/polaris/pull/4831 > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [4939] > > >>>> > > https://github.com/apache/polaris/pull/4939 > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > [5035] > > >>>> > > https://github.com/apache/polaris/pull/5035 > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > Thanks, > > >>>> > > > > > > > > > > > > > > > Dmitri > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > > > > >>>> > > > > >>>> > > >>> > > > > > > > >
