Hi Anand, Re: MetricsPersistence Factory - TBH I do not think it is required. If we go with an MetricsPersistence impl. independent of the JDBC MetaStore, it could be injected into PersistingMetricsReporter directly by CDI.
Re: DataSource - for a start we could probably reuse the same DataSource as the JDBS MetaStore uses. In principle, we could introduce a separate DataSource, but I do not think we need to complicate the initial implementation with that. It can be added later, if required. Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql file for metrics DDL. Then we could add an option to the bootstrap command in the Admin Tool, or even a new command for creating / updating the metrics schema. This might require some refactoring, but I believe the bootstrap code needs some cleanup anyway. The general idea is to allow the metrics schema to evolve independently. It's a new and actively developed feature, so having flexibility is going to be good for making progress / improvements, I hope. Does that sound reasonable? Thanks, Dmitri. On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev < [email protected]> wrote: > Hi Dmitri > > The SPI is designed to be pluggable via > MetaStoreManagerFactory.getOrCreateMetricsPersistence()< > https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51>. > The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+ > returns a functional implementation. > > Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC > Metrics Persistence would require: > > 1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that > can be bootstrapped independently > 2. Separate datasource configuration for metrics (e.g., > polaris.metrics.jdbc.*) > 3. A dedicated `MetricsPersistenceFactory` interface (separate from > MetaStoreManagerFactory) > 4. Updates to the Admin Tool bootstrap workflow to handle both schemas > > Does that sound reasonable? If we do this, all this work should go into > https://github.com/apache/polaris/pull/3523 as one atomic commit, correct? > > — > Anand > > > > From: Dmitri Bourlatchkov <[email protected]> > Date: Thursday, February 5, 2026 at 8:28 AM > To: Anand Kumar Sankaran <[email protected]> > Cc: [email protected] <[email protected]> > Subject: Re: Polaris Telemetry and Audit Trail > > This Message Is From an External Sender > This message came from outside your organization. > Report Suspicious< > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$ > > > > > Hi Anand, > > Thanks for driving the SPI definition for scan/commits metrics. > > I did not think from this angle before, but now I suppose it might be worth > introducing a separate SQL schema for metrics, detached from the main JDBC > Persistence schema. > > My rationale is that the SPI may be exercised independently of the JDBC > Entity Persistence. I can imagine people using NoSQL Entity Persistence > plus JDBC Scan Metrics Persistence in the same deployment. > > Using a separate schema will add further work on the Bootstrap workflow > (Admin Tool), but I hope the end result is going to be more convenient and > versatile for end users. > > WDYT? > > Thanks, > Dmitri. > > On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran < > [email protected]> wrote: > > > Hi Dmitri, > > > > Thank you for taking the time to review. I had pulled out the schema in > > its own PR (based on comments) here: > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$ > > > > I will update the document with the SPI and carve out another PR that > > focuses on the SPI. > > > > — > > Anand > > > > *From: *Dmitri Bourlatchkov <[email protected]> > > *Date: *Thursday, January 29, 2026 at 10:43 AM > > *To: *[email protected] <[email protected]> > > *Cc: *Anand Kumar Sankaran <[email protected]> > > *Subject: *Re: Polaris Telemetry and Audit Trail > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Report Suspicious > > < > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ > > > > > > Hi Anand, > > > > Thanks for making a proposal doc and starting this thread. Apologies for > > late comments. > > > > As you know I already reviewed a couple of related PRs, which had a > > smaller impact on the codebase, but now, I believe, we're coming to the > > core of this feature and it might need a deeper discussion. > > > > For the sake of awareness of other project members, I'd like to > > highlight some aspects by email. > > > > * Storing scan metrics in RDBMS in general looks like a > > reasonable approach to me. However, Polaris Persistence is an extensible > > mechanism. Other backends are possible, including privately developed > > backends. > > > > From this POV, I believe it might be preferable to start not with an > RDBMS > > schema, but with a java SPI outlining expected read/write operations and > > the data model. This will make it easier to assess impact on other > > Persistence implementations. Naturally, Polaris service code will have to > > be able to work seamlessly regardless of the backend impl. (including the > > do-nothing impl.). > > > > I see that some connection between the SQL schema and java code is made > in > > PR [3385], but it does not appear to offer a clean SPI that could be > > implemented by different Persistence backends. Would it be ok from your > > perspective to make another PR with just SPI code? > > > > * Regarding ad-hoc SQL queries - it is certainly a valid use case for > > custom code to query the Polaris database directly. However, those ad-hoc > > queries may require certain indexes, which may not have a relevant use > case > > in OSS code. In that case, I believe it would be preferable to manage the > > extra indexes in custom code (not in Polaris). Whether this is the case > or > > not currently, I cannot say with certainty, so apologies if this is a > false > > alarm... Having the java SPI available would be a huge aid to reasoning > > about the proposed SQL schema, I think. > > > > That said, I do not mean to complicate the development of this feature, > > just trying to structure it in a way that is hopefully easy to understand > > (including myself), maintain and expand later. If I missed something, > > please feel free to point it out. > > > > [3385] > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ > > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$ > > > > > >Thanks, > > Dmitri. > > > > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev < > > [email protected]> wrote: > > > > Hi Yufei, > > > > Gave commenting privileges to all. > > > > Thanks. > > > > — > > Anand > > > > From: Yufei Gu <[email protected]> > > Date: Wednesday, January 14, 2026 at 6:13 PM > > To: [email protected] <[email protected]> > > Cc: Anand Kumar Sankaran <[email protected]> > > Subject: Re: Polaris Telemetry and Audit Trail > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Report Suspicious< > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$ > > > > > > > Hi Anand, thanks for the design doc and PR. Can you allow comments in the > > doc so that people can chime in? Thanks! > > > > Yufei > > > > > > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev < > > [email protected]<mailto:[email protected]>> wrote: > > Hi all > > > > My first PR for adding AWS STS Session Tags support for credential > vending > > was merged. > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$ > > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$ > > > >< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$ > >> > > > > I have been working with Prashant on an end-to-end telemetry and audit > > trail tracking for Polaris. It is documented here. > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$ > > < > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$ > > > >< > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$ > >> > > > > Based on Prashant’s initial feedback (parity with Apache Gravitino for > > metrics reports), I have an initial PR here. > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ > > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$ > > > >< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$ > >> > > > > I will be reaching out to various folks for review. I am open to breaking > > down the PR into smaller logical groups of PR if it helps. > > > > I request you to look at the Google doc and let me know what you think. > > > > — > > Anand > > > > > > > > -- > > Dmitri Bourlatchkov > > Senior Staff Software Engineer, Dremio > > Dremio.com > > < > https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$ > >/ > > Follow Us on LinkedIn > > < > https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$ > >/ > > Get Started > > < > https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$ > > > > > > > >The Agentic Lakehouse > > *The only lakehouse built for agents, managed by agents* > > > > > > > >
