Hi Prashant, Re: separate (events / metrics) schema - in general I'd prefer to be able to evolve events tables separately from the "metastore" schema too.
My rationale is that metastore code does not depend on events (or metrics) persistence, therefore, their schemas could be evolved independently, which aids developing events/metrics enhancements without affecting the main metastore call paths. Once does not have to bump the metastore schema in case events / metrics DDL changes, but the user elected not to persist them. The metrics feature paves the way in that direction, but if the community agrees on this approach in general, I suppose we could refactor events to that effect later. Re: conditional bootstrap - it would certainly make sense to make a ProductionReadinessCheck for verifying (on startup) that if metrics persistence is enabled, we actually have tables in place. Cheers, Dmitri. On Thu, Feb 12, 2026 at 12:14 PM Prashant Singh via dev < [email protected]> wrote: > Thank you for the Proposal Anand ! > The only feedback I had was if we are doing a conditional bootstrap of > metrics table we need to make sure when someone enables the feature > get a clear message that hey table was not bootstrapped we should bootstrap > it first before enabling it ? > In events I am not sure we discussed this but the events table gets created > along with other required tables even when the config is off so we should > make sure we do that. > > Regarding having a connection pool, i agree we should have named datasource > for this to not hog the connection pool which is used by polaris for > entities and ops > > Let me know how it sounds ! > > Best, > Prashant Singh > > On Mon, Feb 9, 2026 at 11:08 AM Anand Kumar Sankaran via dev < > [email protected]> wrote: > > > Dmitri and others, > > > > Thanks for all the review feedback and being patient while we iterated. > My > > initial proposal was to persist the table metrics as events (least amount > > of work 😊), we have come far from that proposal. > > > > To detach the SPI fully, there are a few things to do > (loadSchemaVersion() > > and shared DatasourceOperations). We can add a separate datasource config > > to specify a different connection pool, perhaps. Can be done as a follow > up > > PR. > > > > — > > Anand > > > > From: Dmitri Bourlatchkov <[email protected]> > > Date: Monday, February 9, 2026 at 10:52 AM > > To: [email protected] <[email protected]> > > Cc: Anand Kumar Sankaran <[email protected]> > > Subject: Re: Polaris Telemetry and Audit Trail > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Report Suspicious< > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$ > > > > > > > Hi Anand and all, > > > > Thanks for pushing this feature forward! From my POV the SPI [3616] and > > the schema [3523] PRs are good to merge as the initial implementation of > > this feature (annotated as "beta"). The way is open to the JDBC impl. PR > > [3385] (which will probably need a rebase after merging the SPI and > schema > > code). > > > > I'm sending this email to highlight the direction taken with respect to > > the SQL schema in this feature, as it might have been visible enough to > the > > broader community during the PR reviews. > > > > * The scan / commit metrics schema is managed / versioned separately from > > the JDBC MetaStore schema > > > > * The Metrics Persistence SPI is detached (mostly at this time, but > > supposedly fully detached later) from the JDBC MetaStore > > > > * The JDBC DataSource for metrics may be the same or different from the > > MetaStore datasource. > > > > * Metrics Persistence may have non-JDBC implementation in the future (not > > implemented ATM) > > > > * JDBC Metrics Persistence may co-exist with other MetaStore > > implementations. > > > > I believe this design allows great flexibility for end users to try this > > feature out and keeps options open for future enhancements in Scan / > Commit > > Metrics in Polaris. > > > > [3616] https://github.com/apache/polaris/pull/3616< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$ > > > > > > > [3523] https://github.com/apache/polaris/pull/3523< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$ > > > > > > > [3385] https://github.com/apache/polaris/pull/3385< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$ > > > > > > > Thoughts / comments? > > > > Cheers, > > Dmitri. > > > > > > On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev < > > [email protected]<mailto:[email protected]>> wrote: > > Hi Dmitri, > > > > Following up on your feedback regarding the metrics schema separation > > approach. I've implemented the simplified design you suggested in > > https://github.com/apache/polaris/pull/3523< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$ > > >: > > > > > > The metrics schema can now evolve independently from the entity schema. > > Let me know if you'd like any changes to this approach. > > > > Thanks, > > Anand > > > > > > > > From: Dmitri Bourlatchkov <[email protected]<mailto:[email protected]>> > > Date: Thursday, February 5, 2026 at 2:57 PM > > To: [email protected]<mailto:[email protected]> < > > [email protected]<mailto:[email protected]>> > > Cc: Anand Kumar Sankaran <[email protected]<mailto: > > [email protected]>> > > Subject: Re: Polaris Telemetry and Audit Trail > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Report Suspicious< > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$ > > > > > > > Hi Anand, > > > > Re: MetricsPersistence Factory - TBH I do not think it is required. If we > > go with an MetricsPersistence impl. independent of the JDBC MetaStore, it > > could be injected into PersistingMetricsReporter directly by CDI. > > > > Re: DataSource - for a start we could probably reuse the same DataSource > > as the JDBS MetaStore uses. In principle, we could introduce a separate > > DataSource, but I do not think we need to complicate the initial > > implementation with that. It can be added later, if required. > > > > Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql > > file for metrics DDL. Then we could add an option to the bootstrap > command > > in the Admin Tool, or even a new command for creating / updating the > > metrics schema. This might require some refactoring, but I believe the > > bootstrap code needs some cleanup anyway. > > > > The general idea is to allow the metrics schema to evolve independently. > > It's a new and actively developed feature, so having flexibility is going > > to be good for making progress / improvements, I hope. > > > > Does that sound reasonable? > > > > Thanks, > > Dmitri. > > > > On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev < > > [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>>> wrote: > > Hi Dmitri > > > > The SPI is designed to be pluggable via > > MetaStoreManagerFactory.getOrCreateMetricsPersistence()< > > > https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51 > > < > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$ > > >< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$ > >>. > > The default returns MetricsPersistence.NOOP, and only JDBC with schema > v4+ > > returns a functional implementation. > > > > Your proposal makes sense to me. To allow NoSQL Entity Persistence + > JDBC > > Metrics Persistence would require: > > > > 1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that > > can be bootstrapped independently > > 2. Separate datasource configuration for metrics (e.g., > > polaris.metrics.jdbc.*) > > 3. A dedicated `MetricsPersistenceFactory` interface (separate from > > MetaStoreManagerFactory) > > 4. Updates to the Admin Tool bootstrap workflow to handle both > schemas > > > > Does that sound reasonable? If we do this, all this work should go into > > https://github.com/apache/polaris/pull/3523< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$ > > >< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$ > > > > as one atomic commit, correct? > > > > — > > Anand > > > > > > > > From: Dmitri Bourlatchkov <[email protected]<mailto:[email protected] > > ><mailto:[email protected]<mailto:[email protected]>>> > > Date: Thursday, February 5, 2026 at 8:28 AM > > To: Anand Kumar Sankaran <[email protected]<mailto: > > [email protected]><mailto:[email protected]<mailto: > > [email protected]>>> > > Cc: [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>> < > > [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>>> > > Subject: Re: Polaris Telemetry and Audit Trail > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Report Suspicious< > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$ > > > > > > > > > Hi Anand, > > > > Thanks for driving the SPI definition for scan/commits metrics. > > > > I did not think from this angle before, but now I suppose it might be > worth > > introducing a separate SQL schema for metrics, detached from the main > JDBC > > Persistence schema. > > > > My rationale is that the SPI may be exercised independently of the JDBC > > Entity Persistence. I can imagine people using NoSQL Entity Persistence > > plus JDBC Scan Metrics Persistence in the same deployment. > > > > Using a separate schema will add further work on the Bootstrap workflow > > (Admin Tool), but I hope the end result is going to be more convenient > and > > versatile for end users. > > > > WDYT? > > > > Thanks, > > Dmitri. > > > > On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran < > > [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>>> wrote: > > > > > Hi Dmitri, > > > > > > Thank you for taking the time to review. I had pulled out the schema in > > > its own PR (based on comments) here: > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$ > > > > > > I will update the document with the SPI and carve out another PR that > > > focuses on the SPI. > > > > > > — > > > Anand > > > > > > *From: *Dmitri Bourlatchkov <[email protected]<mailto: > > [email protected]><mailto:[email protected] > > <mailto:[email protected]>>> > > > *Date: *Thursday, January 29, 2026 at 10:43 AM > > > *To: *[email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>> < > > [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>>> > > > *Cc: *Anand Kumar Sankaran <[email protected]<mailto: > > [email protected]><mailto:[email protected]<mailto: > > [email protected]>>> > > > *Subject: *Re: Polaris Telemetry and Audit Trail > > > > > > This Message Is From an External Sender > > > This message came from outside your organization. > > > Report Suspicious > > > < > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ > > > > > > > > > Hi Anand, > > > > > > Thanks for making a proposal doc and starting this thread. Apologies > for > > > late comments. > > > > > > As you know I already reviewed a couple of related PRs, which had a > > > smaller impact on the codebase, but now, I believe, we're coming to the > > > core of this feature and it might need a deeper discussion. > > > > > > For the sake of awareness of other project members, I'd like to > > > highlight some aspects by email. > > > > > > * Storing scan metrics in RDBMS in general looks like a > > > reasonable approach to me. However, Polaris Persistence is an > extensible > > > mechanism. Other backends are possible, including privately developed > > > backends. > > > > > > From this POV, I believe it might be preferable to start not with an > > RDBMS > > > schema, but with a java SPI outlining expected read/write operations > and > > > the data model. This will make it easier to assess impact on other > > > Persistence implementations. Naturally, Polaris service code will have > to > > > be able to work seamlessly regardless of the backend impl. (including > the > > > do-nothing impl.). > > > > > > I see that some connection between the SQL schema and java code is made > > in > > > PR [3385], but it does not appear to offer a clean SPI that could be > > > implemented by different Persistence backends. Would it be ok from your > > > perspective to make another PR with just SPI code? > > > > > > * Regarding ad-hoc SQL queries - it is certainly a valid use case for > > > custom code to query the Polaris database directly. However, those > ad-hoc > > > queries may require certain indexes, which may not have a relevant use > > case > > > in OSS code. In that case, I believe it would be preferable to manage > the > > > extra indexes in custom code (not in Polaris). Whether this is the case > > or > > > not currently, I cannot say with certainty, so apologies if this is a > > false > > > alarm... Having the java SPI available would be a huge aid to reasoning > > > about the proposed SQL schema, I think. > > > > > > That said, I do not mean to complicate the development of this feature, > > > just trying to structure it in a way that is hopefully easy to > understand > > > (including myself), maintain and expand later. If I missed something, > > > please feel free to point it out. > > > > > > [3385] > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ > > > < > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$ > > > > > > > > >Thanks, > > > Dmitri. > > > > > > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev < > > > [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>>> wrote: > > > > > > Hi Yufei, > > > > > > Gave commenting privileges to all. > > > > > > Thanks. > > > > > > — > > > Anand > > > > > > From: Yufei Gu <[email protected]<mailto:[email protected] > > ><mailto:[email protected]<mailto:[email protected]>>> > > > Date: Wednesday, January 14, 2026 at 6:13 PM > > > To: [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>> < > > [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>>> > > > Cc: Anand Kumar Sankaran <[email protected]<mailto: > > [email protected]><mailto:[email protected]<mailto: > > [email protected]>>> > > > Subject: Re: Polaris Telemetry and Audit Trail > > > > > > This Message Is From an External Sender > > > This message came from outside your organization. > > > Report Suspicious< > > > > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$ > > > > > > > > > > Hi Anand, thanks for the design doc and PR. Can you allow comments in > the > > > doc so that people can chime in? Thanks! > > > > > > Yufei > > > > > > > > > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev < > > > [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>><mailto: > > [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>>>> wrote: > > > Hi all > > > > > > My first PR for adding AWS STS Session Tags support for credential > > vending > > > was merged. > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$ > > > < > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$ > > > > > >< > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$ > > >> > > > > > > I have been working with Prashant on an end-to-end telemetry and audit > > > trail tracking for Polaris. It is documented here. > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$ > > > < > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$ > > > > > >< > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$ > > >> > > > > > > Based on Prashant’s initial feedback (parity with Apache Gravitino for > > > metrics reports), I have an initial PR here. > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ > > > < > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$ > > > > > >< > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$ > > >> > > > > > > I will be reaching out to various folks for review. I am open to > breaking > > > down the PR into smaller logical groups of PR if it helps. > > > > > > I request you to look at the Google doc and let me know what you think. > > > > > > — > > > Anand > > > > > > > > > > > > -- > > > Dmitri Bourlatchkov > > > Senior Staff Software Engineer, Dremio > > > Dremio.com > > > < > > > https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$ > > >/ > > > Follow Us on LinkedIn > > > < > > > https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$ > > >/ > > > Get Started > > > < > > > https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$ > > > > > > > > > > > >The Agentic Lakehouse > > > *The only lakehouse built for agents, managed by agents* > > > > > > > > > > > > > >
