Hi Dmitri Thanks for all your support again.
I talked to Yufei and I have addressed his review feedback in this commit. https://github.com/apache/polaris/pull/3385/changes/9e3d4d0f409720ab4005db176190ebde29a9d2b6 Yufei and Prashant - can you please review these changes? Dmitri has already reviewed the PR and approved. - Anand From: Dmitri Bourlatchkov <[email protected]> Date: Tuesday, February 24, 2026 at 10:38 AM To: [email protected] <[email protected]> Cc: Anand Kumar Sankaran <[email protected]> Subject: Re: Polaris Telemetry and Audit Trail This Message Is From an External Sender This message came from outside your organization. Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1xO1e2JFuKSh-PdIU2YhFBQOvhapv4xXZ0ucXUR7TPwnqpAVpb05t786Kda3YPcEe-qOyYlMkJM8hj02m4Yv48r-c47jSruOAIe3mbeFZ5dBQBrfOUk$> Hi Yufei, I see that you propose a different approach to schema evolution for metrics in GH [1] Please share your view of this in light of the previous discussion regarding combining the metrics persistence with different metastore persistence options. It would be good to sync up on this email thread since GH comments tend to be hidden on long PRs and not all readers may be able to easily find them. [1] https://github.com/apache/polaris/pull/3385#issuecomment-3953906091<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385*issuecomment-3953906091__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKT8_s2eg$> Thanks, Dmitri. On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev <[email protected]<mailto:[email protected]>> wrote: Hi Dmitri The SPI is designed to be pluggable via MetaStoreManagerFactory.getOrCreateMetricsPersistence()<https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKXRU26ZA$>>. The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+ returns a functional implementation. Your proposal makes sense to me. To allow NoSQL Entity Persistence + JDBC Metrics Persistence would require: 1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that can be bootstrapped independently 2. Separate datasource configuration for metrics (e.g., polaris.metrics.jdbc.*) 3. A dedicated `MetricsPersistenceFactory` interface (separate from MetaStoreManagerFactory) 4. Updates to the Admin Tool bootstrap workflow to handle both schemas Does that sound reasonable? If we do this, all this work should go into https://github.com/apache/polaris/pull/3523<https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLIg5ApDFA$> as one atomic commit, correct? — Anand From: Dmitri Bourlatchkov <[email protected]<mailto:[email protected]>> Date: Thursday, February 5, 2026 at 8:28 AM To: Anand Kumar Sankaran <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: Polaris Telemetry and Audit Trail This Message Is From an External Sender This message came from outside your organization. Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$> Hi Anand, Thanks for driving the SPI definition for scan/commits metrics. I did not think from this angle before, but now I suppose it might be worth introducing a separate SQL schema for metrics, detached from the main JDBC Persistence schema. My rationale is that the SPI may be exercised independently of the JDBC Entity Persistence. I can imagine people using NoSQL Entity Persistence plus JDBC Scan Metrics Persistence in the same deployment. Using a separate schema will add further work on the Bootstrap workflow (Admin Tool), but I hope the end result is going to be more convenient and versatile for end users. WDYT? Thanks, Dmitri. On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran < [email protected]<mailto:[email protected]>> wrote: > Hi Dmitri, > > Thank you for taking the time to review. I had pulled out the schema in > its own PR (based on comments) here: > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$ > > I will update the document with the SPI and carve out another PR that > focuses on the SPI. > > — > Anand > > *From: *Dmitri Bourlatchkov > <[email protected]<mailto:[email protected]>> > *Date: *Thursday, January 29, 2026 at 10:43 AM > *To: *[email protected]<mailto:[email protected]> > <[email protected]<mailto:[email protected]>> > *Cc: *Anand Kumar Sankaran > <[email protected]<mailto:[email protected]>> > *Subject: *Re: Polaris Telemetry and Audit Trail > > This Message Is From an External Sender > This message came from outside your organization. > Report Suspicious > <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ > > > > Hi Anand, > > Thanks for making a proposal doc and starting this thread. Apologies for > late comments. > > As you know I already reviewed a couple of related PRs, which had a > smaller impact on the codebase, but now, I believe, we're coming to the > core of this feature and it might need a deeper discussion. > > For the sake of awareness of other project members, I'd like to > highlight some aspects by email. > > * Storing scan metrics in RDBMS in general looks like a > reasonable approach to me. However, Polaris Persistence is an extensible > mechanism. Other backends are possible, including privately developed > backends. > > From this POV, I believe it might be preferable to start not with an RDBMS > schema, but with a java SPI outlining expected read/write operations and > the data model. This will make it easier to assess impact on other > Persistence implementations. Naturally, Polaris service code will have to > be able to work seamlessly regardless of the backend impl. (including the > do-nothing impl.). > > I see that some connection between the SQL schema and java code is made in > PR [3385], but it does not appear to offer a clean SPI that could be > implemented by different Persistence backends. Would it be ok from your > perspective to make another PR with just SPI code? > > * Regarding ad-hoc SQL queries - it is certainly a valid use case for > custom code to query the Polaris database directly. However, those ad-hoc > queries may require certain indexes, which may not have a relevant use case > in OSS code. In that case, I believe it would be preferable to manage the > extra indexes in custom code (not in Polaris). Whether this is the case or > not currently, I cannot say with certainty, so apologies if this is a false > alarm... Having the java SPI available would be a huge aid to reasoning > about the proposed SQL schema, I think. > > That said, I do not mean to complicate the development of this feature, > just trying to structure it in a way that is hopefully easy to understand > (including myself), maintain and expand later. If I missed something, > please feel free to point it out. > > [3385] > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$> > >Thanks, > Dmitri. > > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev < > [email protected]<mailto:[email protected]>> wrote: > > Hi Yufei, > > Gave commenting privileges to all. > > Thanks. > > — > Anand > > From: Yufei Gu <[email protected]<mailto:[email protected]>> > Date: Wednesday, January 14, 2026 at 6:13 PM > To: [email protected]<mailto:[email protected]> > <[email protected]<mailto:[email protected]>> > Cc: Anand Kumar Sankaran > <[email protected]<mailto:[email protected]>> > Subject: Re: Polaris Telemetry and Audit Trail > > This Message Is From an External Sender > This message came from outside your organization. > Report Suspicious< > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$ > > > > Hi Anand, thanks for the design doc and PR. Can you allow comments in the > doc so that people can chime in? Thanks! > > Yufei > > > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev < > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > wrote: > Hi all > > My first PR for adding AWS STS Session Tags support for credential vending > was merged. > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$ > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$> >< > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$ >> > > I have been working with Prashant on an end-to-end telemetry and audit > trail tracking for Polaris. It is documented here. > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$ > <https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$> >< > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$ >> > > Based on Prashant’s initial feedback (parity with Apache Gravitino for > metrics reports), I have an initial PR here. > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$> >< > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$ >> > > I will be reaching out to various folks for review. I am open to breaking > down the PR into smaller logical groups of PR if it helps. > > I request you to look at the Google doc and let me know what you think. > > — > Anand > > > > -- > Dmitri Bourlatchkov > Senior Staff Software Engineer, Dremio > Dremio.com > <https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$>/ > Follow Us on LinkedIn > <https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$>/ > Get Started > <https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$> > > >The Agentic Lakehouse > *The only lakehouse built for agents, managed by agents* > > >
