That was also my point about the "split". We should group by "dependency unit". Events and metrics could be isolated.
Regards JB On Thu, Mar 12, 2026 at 6:06 PM Yufei Gu <[email protected]> wrote: > > the resolution scope of a datasource must stay consistent with the > maximum transaction scope (realm sounds okish), something finer grained > can lead to inconsistencies until you do setup XA which is not that trivial > with quarkus OOTB > > Exactly. I'm less worried about events and metrics related to the > transaction scope though. Besides the transaction scope, we must be careful > about whether a join is needed across tables, which is sometimes impossible > if those tables are in different data sources. For example, the current > metrics tables only have the table entity id, which has to be joined with > the entity table to make sense of it. > > Yufei > > > On Thu, Mar 12, 2026 at 12:39 AM Romain Manni-Bucau <[email protected] > > > wrote: > > > Hi, > > > > two small notes on the PR from an end user standpoint: > > > > 1. to make it usable by the community (vs vendors who can always mutate > > beans at build time) it should have a default implementation which is > > configurable - vs doesn't add a feature compared to today - with MP > config > > (in the spirit same capability than > > > > > https://tomee.apache.org/tomee-10.1/examples/dynamic-datasource-routing.html > > but using another config mechanism) > > 2. the resolution scope of a datasource must stay consistent with the > > maximum transaction scope (realm sounds okish), something finer grained > can > > lead to inconsistencies until you do setup XA which is not that trivial > > with quarkus OOTB > > > > hope it helps > > > > Romain Manni-Bucau > > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > > <https://dotnetbirdie.github.io/> | Blog <https://rmannibucau.github.io/ > > > > | Old > > Blog <http://rmannibucau.wordpress.com> | Github > > <https://github.com/rmannibucau> | LinkedIn > > <https://www.linkedin.com/in/rmannibucau> | Book > > < > > > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064 > > > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > > > > Le jeu. 12 mars 2026 à 07:29, EJ Wang <[email protected]> a > > écrit : > > > > > Hi Subham, Dmitri, Yufei, JB, > > > > > > I’m generally aligned with the direction here, and I left a few more > > > detailed comments on the PR. > > > At a high level, my main concern is that the current proposal may still > > be > > > a bit ahead of the current story/scope. I’d lean toward keeping the > first > > > step narrower, preferring purpose-based routing over per-realm routing > > for > > > v1, and making the supported model/config+migration story more explicit > > > before the broader contract hardens. > > > > > > -ej > > > > > > On Wed, Mar 11, 2026 at 12:37 PM Jean-Baptiste Onofré <[email protected] > > > > > wrote: > > > > > > > Hi Subham, > > > > > > > > Thanks for this contribution. It's an interesting feature. > > > > > > > > As mentioned in the GitHub issues, I am fine with moving forward > with a > > > PR > > > > as long as it remains a Draft PR to help drive the discussion. I > > suggest > > > > linking a GitHub Discussion or a Design Doc within the PR to help > build > > > > consensus. > > > > > > > > That said, I have a few initial comments: > > > > > > > > 1. I like the SPI approach used in the PR. This should become a > > standard > > > in > > > > Polaris to facilitate custom implementations. > > > > 2. I agree that having a data source "per purpose" is a good idea. > The > > > main > > > > question is how we should handle the split: > > > > - Very granular (per entity) > > > > - By table "meaning" > > > > - By realm (this may not be granular enough) > > > > > > > > From a user standpoint, I believe we should keep it simple. It would > > be a > > > > first great step forward. For example, in the configuration (e.g., > > > > application.properties), it could look like: > > > > - polaris.datasource.entities= > > > > - polaris.datasource.events= > > > > - polaris.datasource.grants= > > > > > > > > Regards, > > > > JB > > > > > > > > On Tue, Mar 10, 2026 at 11:19 PM Yufei Gu <[email protected]> > > wrote: > > > > > > > > > Hi Subham, > > > > > > > > > > Thanks for working on this. Given the complexity and long term > > > > implications > > > > > discussed in https://github.com/apache/polaris/issues/3890, I > think > > a > > > > > short > > > > > design doc could still be helpful to capture the intended > > architecture > > > > and > > > > > future evolution. Here are a few questions listed in the issue. I > > > believe > > > > > these should be answered before jumping to an implementation. > > > > > > > > > > > > > > > 1. Should we split each potential noisy table into its own > > dedicated > > > > > data source. For example, one data source for events, one for > > > metrics, > > > > > and > > > > > one for idempotency. > > > > > 2. Should we allow flexible grouping. For example, events and > > > > > idempotency tables sharing one data source, while metrics uses > > > > another. > > > > > 3. Should we consider different DS per realm instead of > > table-level > > > > > spliting? > > > > > 4. How should schema version information be managed. If tables > > live > > > in > > > > > different data sources, how do we track and coordinate schema > > > > evolution. > > > > > 5. Should different data sources be allowed to point to > different > > > > > schemas or databases. This likely aligns with the isolation > goal, > > > but > > > > it > > > > > implies that cross table joins become difficult or impossible at > > the > > > > > database level, leaving only in memory joins as an option. > > > > > 6. Should different data sources be allowed to point to the same > > > > schema. > > > > > If not, we need validation logic to detect and prevent > > > > misconfiguration. > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Tue, Mar 10, 2026 at 7:33 AM Dmitri Bourlatchkov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Subham, > > > > > > > > > > > > Thanks again for your contribution! > > > > > > > > > > > > I believe PR 3960 moves in the right direction by establishing an > > SPI > > > > to > > > > > > delegate DataSource resolution logic to the runtime environment. > > > > > > > > > > > > It immediately allows custom implementations in downstream > projects > > > (if > > > > > > people wish to do that) and opens a way for supporting multiple > > > > > DataSources > > > > > > in Apache Polaris (in follow-up PRs), > > > > > > > > > > > > I think the PR is pretty clear in itself and does not require any > > > extra > > > > > > design docs. Let's review it in GH and merge when we have > > consensus. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Tue, Mar 10, 2026 at 8:27 AM Subham Sangwan < > > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Polaris Dev Team I have opened PR #3960 [1] to introduce the > > > > > > > foundational groundwork for multi-datasource support in JDBC > > > > > persistence, > > > > > > > addressing Issue #3890 [2].The goal is to enable physical > > isolation > > > > of > > > > > > > different persistence workloads (METASTORE, METRICS, EVENTS) > into > > > > > > dedicated > > > > > > > connection pools or databases. This will allow Polaris to > better > > > > handle > > > > > > > high-traffic environments by preventing "noisy neighbor" > effects > > on > > > > the > > > > > > > core entity tables. > > > > > > > > > > > > > > Key Highlights: > > > > > > > > > > > > > > - DataSourceResolver: A new pluggable interface for routing > > JDBC > > > > > > > connections based on RealmContext and StoreType. > > > > > > > - Modular Design: Decoupled the resolution implementation > into > > > the > > > > > > > runtime-common module. > > > > > > > - Consistency: Utilizes a type-safe StoreType enum and > aligns > > > with > > > > > > > existing RealmContext patterns. > > > > > > > > > > > > > > The PR has been refined with feedback from @dimas-b and is now > > > ready > > > > > for > > > > > > > community review. I'd appreciate any feedback on the overall > > > > approach. > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Subham Sangwan > > > > > > > GitHub: Subham-KRLX > > > > > > > > > > > > > > [1] https://github.com/apache/polaris/pull/3960 > > > > > > > [2] https://github.com/apache/polaris/issues/3890 > > > > > > > > > > > > > > > > > > > > > > > > > > > >
