Hi Yeah fully agree and it was my point when I replied on the thread initially.
Regards JB Le jeu. 2 juil. 2026 à 23:32, Yufei Gu <[email protected]> a écrit : > If our goal is to make the default experience closer to production > while removing TreeMap, then JDBC with H2 seems like a more natural > choice. It exercises the same persistence layer, schema management, > and configuration that users will encounter in production, while still > keeping setup lightweight. > > Yufei > > Yufei > > > On Thu, Jul 2, 2026 at 3:46 AM Alexandre Dutra <[email protected]> wrote: > > > > Hi Russell, > > > > The existing default metastore shouldn't be viewed merely as a > > "relatively simple, self-contained implementation": it has unique > > quirks, such as returning result types that no other metastore does. > > > > Choosing the status quo simply to avoid making changes is imho the > > worst solution. > > > > Transitioning to the NoSQL metastore as the default remains a net > > benefit, even if it is only used by 30% of the user base. > > > > However, I agree with keeping the test suite on JDBC + H2 by default > > if that ensures a higher level of confidence in our tests ability to > > reproduce real production setups. > > > > Thanks, > > Alex > > > > On Wed, Jul 1, 2026 at 12:23 AM Russell Spitzer > > <[email protected]> wrote: > > > > > > I agree TreeMap and its associated transactional stack should be > deprecated. > > > It’s test-only, confusing, and not used by JDBC production paths. > > > > > > I’m less convinced that NoSQL + InMemory is a clearly superior > replacement > > > for TreeMap in tests. At the storage layer it’s essentially the same > class > > > of > > > thing: an in-memory, non-durable backend. The difference is that it > > > exercises > > > a completely different MetaStoreManager/persistence stack > > > (NoSqlMetaStoreManager > > > vs TransactionalMetaStoreManagerImpl), not the JDBC path most > deployments > > > use. It > > > replaces a relatively simple, self-contained implementation with > something > > > that is > > > part of a much more complicated persistence stack, while remaining > > > non-durable > > > and test-only at the storage layer. > > > > > > For the problems originally raised (test-grade defaults, dead > production > > > path, developer confusion), > > > JDBC + H2 seems like an actual fix. > > > > > > > > > On Mon, Jun 29, 2026 at 2:06 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi Alex, > > > > > > > > I tend to agree that the DataSource discussion related to H2 became > a bit > > > > convoluted, and it's a lot simpler to use the NoSQL in-memory > persistence > > > > instead. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Mon, Jun 29, 2026 at 1:57 PM Alexandre Dutra <[email protected]> > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Some updates on this: > > > > > > > > > > I recently started a separate discussion regarding dynamic > datasource > > > > > activation at runtime [1]. This was intended as a prerequisite for > > > > > adopting H2 as the default JDBC driver, as suggested on this > thread. > > > > > > > > > > However, that proposal may end up being rejected. > > > > > > > > > > Given this situation, we should reconsider our options. If the > > > > > PostgreSQL + H2 approach is no longer a viable replacement to the > > > > > TreeMapMetaStore, I propose an alternative: let's use the NoSQL > > > > > metastore with an InMemory backend. > > > > > > > > > > My reasoning is as follows: > > > > > > > > > > 1. It remains a production-ready metastore, so strictly superior to > > > > > TreeMapMetaStore. > > > > > > > > > > 2. The metastore state update logic is the same regardless of the > > > > > backend, so we're still close to a real production setup (same as > with > > > > > an in-memory JDBC driver). > > > > > > > > > > 3. It doesn't require any external dependencies (no bundled > driver). > > > > > > > > > > I'd also note that while the NoSQL metastore is already included > > > > > today, it is not actionable because no backend is available. This > goes > > > > > against the onboarding UX that we're trying to achieve, as users > > > > > interested in Polaris with the NoSQL metastore cannot test this > setup > > > > > with the official image. > > > > > > > > > > Therefore, I think NoSQL + InMemory would serve as a more > appropriate > > > > > default setup for the official Polaris image. > > > > > > > > > > From what I see, it would boil down to adding one line to > > > > > polaris-server or polaris-service: > > > > > > > > > > runtimeOnly(project(":polaris-persistence-nosql-db-inmemory")) > > > > > > > > > > What are your thoughts on this? > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > [1]: > https://lists.apache.org/thread/jy6wb186h94n9q86kv01shbn68ppr6gv > > > > > > > > > > > > > > > On Fri, Jun 19, 2026 at 12:07 PM Alexandre Dutra < > [email protected]> > > > > > wrote: > > > > > > > > > > > > Hi all, > > > > > > > > > > > > I wanted to give an update on this topic: I have started working > on > > > > > > this, but I would like to have [4812] merged first, since that's > a > > > > > > prerequisite. > > > > > > > > > > > > Thanks, > > > > > > Alex > > > > > > > > > > > > [4812]: https://github.com/apache/polaris/pull/4812 > > > > > > > > > > > > On Tue, Jun 16, 2026 at 5:15 PM Russell Spitzer > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > I believe the original idea of TreeMapMetastore was that it > looked > > > > very > > > > > > > similar to FoundationDB from an API perspective, so it served > well > > > > as > > > > > a > > > > > > > test system for the original backend being developed at SF. > > > > > > > > > > > > > > I agree that H2 + JDBC makes sense to me now for the project. > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 10:03 AM Alexandre Dutra < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > Thank you for your feedback. > > > > > > > > > > > > > > > > There seems to be a general agreement on the idea of > deprecating > > > > > > > > TreeMapMetaStore, coupled with the JDBC + H2 solution for > tests. > > > > > > > > > > > > > > > > In most modules, the tests migration won't pose any serious > > > > > > > > challenges, as replacing the metastore is generally just a > matter > > > > of > > > > > > > > changing the configuration, and making sure the realm is > properly > > > > > > > > bootstrapped. > > > > > > > > > > > > > > > > There will be, however, a few tricky situations in > polaris-core: a > > > > > few > > > > > > > > tests rely on the TreeMapMetaStore, mostly as a test > convenience; > > > > but > > > > > > > > there is no obvious replacement for it in that module. I am > however > > > > > > > > confident that we can find a solution for that, either based > on > > > > > mocks, > > > > > > > > or by bringing in a real metastore. > > > > > > > > > > > > > > > > If no objections are raised, I am going to prepare a PR for > this. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Alex > > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 3:28 PM Dmitri Bourlatchkov < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > > > > > > > Thanks for starting this thread. > > > > > > > > > > > > > > > > > > I also find the TransactionalMetaStoreManagerImpl and > related > > > > call > > > > > paths > > > > > > > > > conuising in Apache Polaris code. > > > > > > > > > > > > > > > > > > I support promoting H2 to the default (in memory) > Persistence > > > > > backend for > > > > > > > > > getting started cases. This should also resolve the old H2 > > > > > evolution > > > > > > > > thread > > > > > > > > > [1] by ensuring it is used regularly on the same code > paths as > > > > > > > > PostgreSQL. > > > > > > > > > > > > > > > > > > Deprecating TransactionalMetaStoreManagerImpl for removal > also > > > > > sounds > > > > > > > > > reasonable to me. Existing downstream users > > > > > > > > > of TransactionalMetaStoreManagerImpl will have time to > migrate, > > > > or > > > > > even > > > > > > > > > embed that code (per ASF license) into local builds, > during the > > > > > > > > deprecation > > > > > > > > > phase. > > > > > > > > > > > > > > > > > > [1] > > > > > https://lists.apache.org/thread/g1gg2w8hn9gvlmwrdh0x218whoh2wd39 > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2026 at 11:24 AM Alexandre Dutra < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > I am writing to ask the community whether it is OK to > deprecate > > > > > > > > > > TreeMapMetaStore, as well as all the in-memory metastore > > > > manager, > > > > > > > > > > metastore manager factory, and persistence types that > rely > > > > > solely on > > > > > > > > > > TreeMapMetaStore. > > > > > > > > > > > > > > > > > > > > As we know, these components are test-grade only, and not > > > > > suitable for > > > > > > > > > > production. They trigger a production readiness alert on > > > > Polaris > > > > > > > > > > startup. It's a considerable amount of code that is > virtually > > > > > dead in > > > > > > > > > > production. > > > > > > > > > > > > > > > > > > > > It's also confusing for developers. E.g. the > "transactional" > > > > > metastore > > > > > > > > > > is not transactional in the JDBC sense of the term, and > thus > > > > not > > > > > used > > > > > > > > > > by JDBC persistence. It also has its quirks: some return > > > > > statuses are > > > > > > > > > > only returned by that manager. > > > > > > > > > > > > > > > > > > > > However, these components are used in tests, and I agree > that > > > > > it's > > > > > > > > > > useful to have an in-memory version of the persistence > layer > > > > for > > > > > > > > > > tests. But we have today two alternatives that are imho > > > > superior > > > > > for > > > > > > > > > > tests in polaris-runtime-service: > > > > > > > > > > > > > > > > > > > > - JDBC persistence with H2 backend. There are already a > few > > > > tests > > > > > > > > > > using this setup. > > > > > > > > > > - NoSQL persistence with InMemory backend. > > > > > > > > > > > > > > > > > > > > Both alternatives test real production-grade persistence > code. > > > > > > > > > > > > > > > > > > > > And finally, TreeMapMetaStore is currently the default > runtime > > > > > > > > > > persistence in application.properties; and the Helm > chart also > > > > > > > > > > advertises it as the default. These are not sane > defaults, > > > > imho. > > > > > It's > > > > > > > > > > always tricky to provide a good default for datastores, > but > > > > > since JDBC > > > > > > > > > > persistence is included by default in the server image, > I think > > > > > that > > > > > > > > > > including the H2 driver by default could give us a saner > > > > default > > > > > while > > > > > > > > > > keeping the out-of-the-box experience intact (the > license is > > > > > Category > > > > > > > > > > A). > > > > > > > > > > > > > > > > > > > > Concretely: > > > > > > > > > > > > > > > > > > > > On the MetaStoreManagerFactory hierarchy, the following > > > > > > > > > > implementations are completely in-memory: > > > > > > > > > > > > > > > > > > > > - InMemoryPolarisMetaStoreManagerFactory: could be > removed > > > > > > > > > > > > > > > > > > > > - InMemoryAtomicOperationMetaStoreManagerFactory: could > be > > > > > removed > > > > > > > > > > > > > > > > > > > > - LocalPolarisMetaStoreManagerFactory: is the base class > of the > > > > > two > > > > > > > > > > above; imho it can be removed, but since it's an abstract > > > > class, > > > > > it > > > > > > > > > > may have been extended outside Polaris. But neither JDBC > nor > > > > > NoSQL use > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > On the PolarisMetaStoreManager hierarchy: > > > > > > > > > > > > > > > > > > > > - TransactionalMetaStoreManagerImpl: is only used by > > > > > > > > > > LocalPolarisMetaStoreManagerFactory. JDBC and NoSQL do > not use > > > > > it. > > > > > > > > > > Could be removed. > > > > > > > > > > > > > > > > > > > > - TransactionWorkspaceMetaStoreManager however is a > different > > > > > beast, > > > > > > > > > > in spite of the similar name. It is in use today on the > commit > > > > > path, > > > > > > > > > > so should not be removed. > > > > > > > > > > > > > > > > > > > > On the BasePersistence hierarchy: > > > > > > > > > > > > > > > > > > > > - TreeMapTransactionalPersistenceImpl and its > TreeMapMetaStore > > > > > are > > > > > > > > > > only used by InMemoryPolarisMetaStoreManagerFactory, and > could > > > > be > > > > > > > > > > removed; > > > > > > > > > > > > > > > > > > > > - TransactionalPersistence and > > > > AbstractTransactionalPersistence: > > > > > these > > > > > > > > > > are supertypes of TreeMapTransactionalPersistenceImpl > and thus > > > > > only > > > > > > > > > > used for in-memory. They imo can be removed, but they > might > > > > have > > > > > been > > > > > > > > > > extended or implemented outside Polaris. > > > > > > > > > > > > > > > > > > > > I think it's important to keep Polaris code tidy by > removing > > > > > unused, > > > > > > > > > > unimplementable, or test-grade only components. > > > > > > > > > > > > > > > > > > > > What are your thoughts? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > >
