Hi all, Thanks for driving this work forward. Building this as a POC makes perfect sense if the goal is to validate the idea and gather feedback. Should we merge a POC into the main branch before agreeing on the overall direction? To me, that feels like a separate discussion from whether the implementation itself looks reasonable.
One concern that I don't think has been fully addressed is whether we need to expose metrics consumption through a Polaris REST endpoint at all. My concern is less about the API itself and more about the boundary Polaris should own. The proposed REST API is tied to a particular persistence schema, but that schema is just one possible implementation. While the current implementation stores metrics in the metastore, other deployments may choose a KV store, a dedicated metrics warehouse, or another backend entirely. In those cases, exposing a Polaris REST API over a specific schema feels like coupling the API to an implementation detail. Polaris isn't intended to become a full metrics system. I'd expect it to focus on producing metrics and let deployments choose the most appropriate storage and consumption model for their environment. I'm not opposed to exploring this direction, but we should first align on whether exposing a REST API through Polaris is the right long term approach before merging it into the main branch. Thanks, Yufei Yufei On Mon, Jun 29, 2026 at 9:27 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > It looks like Anand addressed all action items in [4115]. > > Please review the latest state of the code. The PR LGTM and I approved in > GH from my side. > > [4115] https://github.com/apache/polaris/pull/4115 > > Thanks, > Dmitri. > > On Thu, Jun 25, 2026 at 1:56 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi All, > > > > Recap of the discussion in today's Community Sync: > > > > * The modular approach for Metrics SPI(s) and implementation seems to > have > > community consensus (please comment if you still have concerns) > > > > * What modules / feature flags are enabled in Apache Polaris images by > > default will be discussed later (when the feature is close to > completion). > > > > In any case, users will be able to turn the full feature on/off at their > > discretion, regardless of the default flag values. > > > > * It is generally preferable to have smaller PRs and introduce new > > features gradually to simplify reviews. > > > > What this means for the Metrics API, IMHO is: > > > > a) The scope of PR 4115 is probably ok as it stands right now. If someone > > wants to propose splitting it further, please comment. > > > > b) After 4115 is merged, Metrics Persistence PRs will follow, with > smaller > > scope per PR as appropriate for the nature of the code changes. > > > > * The major concern was about exposing the new Metrics REST API to end > > users. The nature of this concern was about avoiding the (false) > impression > > that the new API is a firm and adopted Polaris specification. > > > > I think this concern is reasonable. > > > > I propose that Anand update the documentation sections of PR 4115 to > > emphasise that the API is currently an experimental / POC API at this > time > > and is subject to change (including breaking changes). I believe it > already > > has the "beta" label. So this request is mostly for doc changes to > clarify > > user expectations, since the "beta" label alone may not be interpreted > the > > same way by all users. > > > > I hope this update will resolve all remaining concerns with 4115 and > allow > > for it to be merged so that the community will be able to proceed to > > follow-up PRs (as noted above). > > > > Please add/correct me if I missed anything relevant. > > > > Thanks, > > Dmitri. > > > > On Wed, Jun 24, 2026 at 8:11 PM Yufei Gu <[email protected]> wrote: > > > >> I do not think we have agreement on the query REST APIs yet. > >> > >> In particular, the current query API shape seems highly coupled to the > >> *example[1]* metrics in the Iceberg REST spec. I do not think that is a > >> good way to design a REST API contract. For example, fields like > >> resultDataFiles, resultDeleteFiles, totalFileSizeBytes, > >> totalDataManifests, > >> totalDeleteManifests, and scannedDataManifests are very specific to one > >> style of scan metric. That makes the API look more like a direct > >> projection > >> of the current example payload than a general query model. > >> > >> My preference is that Polaris should not define or strongly opine on the > >> Iceberg scan or commit metrics query REST API at this point. Polaris can > >> resolve and authorize the table, accept the metrics report, and delegate > >> it > >> to the configured reporting path. Durable storage, filtering, > dashboards, > >> and query APIs should remain downstream impl. choices unless and until > the > >> Iceberg community standardizes a query contract, which I doubt will > >> happen. > >> > >> So I do not think the intake path has to be blocked, but I would prefer > >> not > >> to merge a Polaris specific query API that may be hard to evolve later. > >> > >> 1. > >> > >> > https://github.com/apache/iceberg/blob/2a6c556c0c883c04c6d3fbd68e6aeda36b91e0aa/open-api/rest-catalog-open-api.yaml#L4152 > >> > >> Yufei > >> > >> > >> On Tue, Jun 23, 2026 at 9:30 AM Dmitri Bourlatchkov <[email protected]> > >> wrote: > >> > >> > Hi All, > >> > > >> > So, what are the blockers for merging [4115], if any? > >> > > >> > [4115] https://github.com/apache/polaris/pull/4115 > >> > > >> > Thanks, > >> > Dmitri. > >> > > >> > On Wed, Jun 17, 2026 at 5:28 PM EJ Wang < > [email protected] > >> > > >> > wrote: > >> > > >> > > Hi folks, > >> > > > >> > > Thanks again for the discussion today. I updated the sync doc with > >> notes > >> > > from today's metrics sync: > >> > > > >> > > > >> > > > >> > > >> > https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.ezk0rgdx0c6m > >> > > > >> > > A few highlights: > >> > > > >> > > - No objection to keeping the default metrics behavior small: > >> > > no-op/log-only is enough as the built-in default. > >> > > - Durable storage, event forwarding, external queues, dashboards, > >> and > >> > > custom filtering should be implementation choices behind the > >> metrics > >> > > reporting path. > >> > > - For PR #4115, the REST layer should resolve/authz/accept the > >> report, > >> > > then delegate to the selected metrics implementation. It should > not > >> > own > >> > > durable storage semantics. > >> > > - The rough boundary we discussed is: spec owns the wire > contract, > >> > > runtime owns framework wiring, API/contract modules own > >> > provider-facing > >> > > contracts, and extensions own replaceable implementations. > >> > > - OpenLineage has a similar path-naming question. We should avoid > >> > > occupying a generic lineage path too early if we want room for > >> other > >> > > lineage systems later. > >> > > > >> > > I believe the boundary rules previewed in sync meeting are useful > for > >> > > reviewing PR #4115, but it is not > >> > > a final module-layout proposal. > >> > > > >> > > For metrics specifically, I think the target remains: > >> > > runtime > >> > > -> metrics reporting/emitting contract > >> > > -> default implementation > >> > > -> lower implementation details, if needed > >> > > > >> > > That keeps the default simple, while allowing durable JDBC, > >> event-backed, > >> > > or external-queue implementations to be added separately. > >> > > > >> > > We also touched on the split between Iceberg metrics emitting and > >> > > Polaris-owned metrics querying. The querying side can keep evolving, > >> > > especially for dashboard or generic-table use cases, but I do not > >> think > >> > > that needs to block the intake path in this PR. > >> > > > >> > > Thanks, > >> > > -ej > >> > > > >> > > On Tue, Jun 16, 2026 at 2:17 PM Yufei Gu <[email protected]> > >> wrote: > >> > > > >> > > > Thanks for chiming in, EJ. Agreed that the default battery should > >> stay > >> > > > small. I'm leaning toward not including metrics persistence in the > >> > > default > >> > > > battery. > >> > > > > >> > > > Yufei > >> > > > > >> > > > > >> > > > On Thu, Jun 11, 2026 at 8:30 PM EJ Wang < > >> > [email protected]> > >> > > > wrote: > >> > > > > >> > > > > Hi Yufei, > >> > > > > > >> > > > > Thanks for connecting this back to the REST endpoint proposal. I > >> > > replied > >> > > > > with the fuller version on the event-forwarding thread, but > >> wanted to > >> > > add > >> > > > > the shorter version here too. > >> > > > > > >> > > > > I agree that metrics reporting/emitting is the right conceptual > >> > > boundary, > >> > > > > and I also think the event/listener path is a good > implementation > >> > > > direction > >> > > > > to explore. The distinction I want to keep clear is default > >> battery > >> > vs > >> > > > > extension implementation. > >> > > > > > >> > > > > For the REST/API side, I would keep the semantics narrow: > Polaris > >> > > > resolves > >> > > > > and authorizes the table, accepts the Iceberg scan/commit > metrics > >> > > report > >> > > > > into the configured reporting/emitting path, and returns 204. > That > >> > 204 > >> > > > > should mean Polaris accepted the report into the ingestion path; > >> it > >> > > > should > >> > > > > not imply durable storage. > >> > > > > > >> > > > > Durable storage, event forwarding, external telemetry routing, > >> > > filtering, > >> > > > > and retention should sit behind that reporting/emitting boundary > >> as > >> > > > > implementation-layer behavior. A durable JDBC path can be one > >> named > >> > > > > extension implementation. An event/listener forwarding path can > be > >> > > > another > >> > > > > named extension implementation. > >> > > > > > >> > > > > So I am +1 on the event-forwarding idea as a non-default > extension > >> > > > > implementation. I just would not make it the default battery or > >> the > >> > > core > >> > > > > API shape. The default battery can stay small and safe, while > >> > > deployments > >> > > > > that need forwarding, dashboards, or durable storage can opt > into > >> the > >> > > > > implementation that matches their operational model. > >> > > > > > >> > > > > Thanks, > >> > > > > -ej > >> > > > > > >> > > > > On Wed, Jun 10, 2026 at 11:43 AM Yufei Gu <[email protected] > > > >> > > wrote: > >> > > > > > >> > > > > > Thanks EJ. I agree that the reporting/emitting boundary feels > >> like > >> > > the > >> > > > > > right SPI boundary, but I wonder if we can simplify this even > >> > > further. > >> > > > > > > >> > > > > > Iceberg scan and commit metrics look very similar to events to > >> me. > >> > > They > >> > > > > are > >> > > > > > append only, asynchronously consumed, often forwarded to > >> external > >> > > > > systems, > >> > > > > > and have similar retention and cleanup requirements. More > >> details > >> > can > >> > > > be > >> > > > > > found in this thread [1]. In that case, Polaris may only need > to > >> > > accept > >> > > > > the > >> > > > > > report and emit an event through the existing event framework. > >> > > > Filtering, > >> > > > > > async delivery, custom sinks, and retention mechanisms could > >> then > >> > be > >> > > > > shared > >> > > > > > instead of introducing a separate metrics specific extension > >> layer > >> > > and > >> > > > > SPI. > >> > > > > > > >> > > > > > The main requirement I still see is querying metrics through > >> > Polaris. > >> > > > If > >> > > > > we > >> > > > > > want that battery included experience, we could work on a > >> > persistence > >> > > > > > listener implementation that selectively saves metrics and > >> expose > >> > it > >> > > > via > >> > > > > > the IRC event endpoint which seems pretty close in Iceberg > >> > > > > > community. Another option I like more is to allow users to > >> develop > >> > > > their > >> > > > > > own listener, so they can persist scan/commit metrics to any > >> > systems > >> > > > they > >> > > > > > prefer. In general, that feels like an implementation choice > on > >> top > >> > > of > >> > > > > the > >> > > > > > event framework rather than something that needs to shape the > >> core > >> > > > > > architecture. > >> > > > > > > >> > > > > > 1. > >> > https://lists.apache.org/thread/x9j8nscvy8hq61tyn01mj8yp6n9of0kp > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Yufei > >> > > > > > > >> > > > > > > >> > > > > > On Thu, Jun 4, 2026 at 5:14 PM EJ Wang < > >> > > [email protected] > >> > > > > > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Good point. I agree the existing PolarisMetricsReporter is > >> > already > >> > > > very > >> > > > > > > close to the right conceptual boundary. I am not proposing a > >> > second > >> > > > > > > parallel reporter concept. The distinction I am trying to > >> make is > >> > > > > > narrower: > >> > > > > > > > >> > > > > > > Current state: > >> > > > > > > > >> > > > > > > - PolarisMetricsReporter is the existing metrics > reporting > >> > hook. > >> > > > > > > - But it currently lives under runtime/service. > >> > > > > > > - Its contract is documented around Quarkus/CDI discovery > >> and > >> > > > > > > @Identifier > >> > > > > > > selection. > >> > > > > > > - The default implementation is log-only. > >> > > > > > > - The durable implementation writes through > >> > > PolarisMetricsManager > >> > > > > and > >> > > > > > > MetricsPersistence. > >> > > > > > > - MetricsPersistence is currently inherited by > >> > BasePersistence. > >> > > > > > > > >> > > > > > > My proposal: > >> > > > > > > > >> > > > > > > - Keep the reporting/emitting boundary as the SPI. > >> > > > > > > - Revise that contract in a framework-agnostic optional > >> > Iceberg > >> > > > > > *metrics > >> > > > > > > extension API layer*. > >> > > > > > > - Keep runtime/service responsible for REST ingestion, > >> > > table/authz > >> > > > > > > validation, and runtime wiring. > >> > > > > > > - Keep the battery implementation log-only in the metrics > >> > > > extension > >> > > > > > API > >> > > > > > > layer, *not under runtime/service*. > >> > > > > > > - Treat durable JDBC metrics as one implementation of the > >> > > > reporting > >> > > > > > SPI. > >> > > > > > > - Decompose the MetricsPersistence and BasePersistence > >> > coupling > >> > > as > >> > > > > > > durable implementation detail. > >> > > > > > > > >> > > > > > > So the difference is not “new reporter concept vs old > reporter > >> > > > > concept.” > >> > > > > > > but: > >> > > > > > > > >> > > > > > > > >> > > > > > > - PolarisMetricsReporter today = runtime/service > CDI-shaped > >> > > hook. > >> > > > > > > - Target reporting SPI = same conceptual boundary, but > >> placed > >> > in > >> > > > the > >> > > > > > > right extension API layer and kept framework-agnostic. > >> > > > > > > > >> > > > > > > For the current metrics PR, I think the minimal > architectural > >> > > cleanup > >> > > > > > > before merge is to *put the reporting SPI contract and the > >> > battery > >> > > > > > default > >> > > > > > > in the right place*. The durable JDBC implementation, > >> > event-backed > >> > > > > async > >> > > > > > > implementation, external queue support, and deeper > persistence > >> > > > cleanup > >> > > > > > can > >> > > > > > > continue as follow-ups, as long as we do not lock the SPI > >> > boundary > >> > > to > >> > > > > > > BasePersistence or to runtime/service wiring. > >> > > > > > > > >> > > > > > > -ej > >> > > > > > > > >> > > > > > > On Wed, Jun 3, 2026 at 3:00 PM Dmitri Bourlatchkov < > >> > > [email protected] > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Hi EJ, > >> > > > > > > > > >> > > > > > > > Thanks for the recap / summary! > >> > > > > > > > > >> > > > > > > > > Do folks agree that the stable SPI boundary should be > >> > > > > > > > metrics reporting/emitting, not metrics persistence? > >> > > > > > > > > >> > > > > > > > SGTM. > >> > > > > > > > > >> > > > > > > > > Does an optional Iceberg metrics extension API layer > sound > >> > like > >> > > > the > >> > > > > > > right > >> > > > > > > > home for this SPI? > >> > > > > > > > > >> > > > > > > > How is that different from current PolarisMetricsReporter? > >> > > > > > > > > >> > > > > > > > > Should the current durable metrics work be reframed as a > >> > > durable > >> > > > > > > > JDBC reference implementation of that SPI? > >> > > > > > > > > >> > > > > > > > SGTM. > >> > > > > > > > > >> > > > > > > > > What is the smallest PR sequence to get there without > >> > blocking > >> > > > > > > > the current metrics work unnecessarily? > >> > > > > > > > > >> > > > > > > > Let's get https://github.com/apache/polaris/pull/4397 > >> merged > >> > > > first. > >> > > > > > > > > >> > > > > > > > Then, I'd propose isolating the Metrics schema from the > >> > MetaStore > >> > > > > > schema. > >> > > > > > > > This will probably have some ripple effect into the > >> bootstrap > >> > > > > > workflows, > >> > > > > > > so > >> > > > > > > > it's not a trivial change. > >> > > > > > > > > >> > > > > > > > Then, let's reassess. > >> > > > > > > > > >> > > > > > > > Cheers, > >> > > > > > > > Dmitri. > >> > > > > > > > > >> > > > > > > > On Wed, Jun 3, 2026 at 5:40 PM EJ Wang < > >> > > > > [email protected] > >> > > > > > > > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Hi folks, > >> > > > > > > > > > >> > > > > > > > > Thanks again for the discussion today. I updated the > sync > >> doc > >> > > > with > >> > > > > > > notes > >> > > > > > > > > from the third metrics architecture sync: > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.k96s2xyqr5u1 > >> > > > > > > > > > >> > > > > > > > > A few highlights from today’s discussion: > >> > > > > > > > > > >> > > > > > > > > - We clarified that Iceberg metrics reporting can be > >> > > > interpreted > >> > > > > > > > either > >> > > > > > > > > as sync or async handling from Polaris’s perspective, > >> so > >> > > > Polaris > >> > > > > > > > should > >> > > > > > > > > stay flexible as a platform instead of baking one > >> handling > >> > > > model > >> > > > > > > into > >> > > > > > > > > the > >> > > > > > > > > REST API behavior. > >> > > > > > > > > - We aligned that the built-in (battery) behavior for > >> > > metrics > >> > > > > > > emitting > >> > > > > > > > > can stay simple: no-op/log-only is enough as the > >> default. > >> > > > > > > > > - Durable metrics persistence should be treated as an > >> > > > > > implementation > >> > > > > > > > of > >> > > > > > > > > the metrics reporting path, not as the core SPI > >> boundary. > >> > > > > > > > > - The existing durable metrics work can be reviewed > as > >> a > >> > > > > reference > >> > > > > > > > > implementation of the reporting SPI, with > >> > > persistence-related > >> > > > > > logic > >> > > > > > > > kept > >> > > > > > > > > self-contained under a metrics durable implementation > >> > module > >> > > > > > rather > >> > > > > > > > than > >> > > > > > > > > scattered through core entity persistence. > >> > > > > > > > > - Dashboard/insights remains a real use case, but we > >> > agreed > >> > > to > >> > > > > > keep > >> > > > > > > it > >> > > > > > > > > separate from the core metrics intake discussion for > >> now. > >> > > > > > > > > > >> > > > > > > > > I also did a quick source check after the meeting to > make > >> > sure > >> > > we > >> > > > > are > >> > > > > > > > > describing the current state accurately. > >> > > > > > > > > > >> > > > > > > > > Current state: > >> > > > > > > > > > >> > > > > > > > > - Polaris already has a metrics reporting hook: > >> > > > > > > > PolarisMetricsReporter. > >> > > > > > > > > - The default implementation is > DefaultMetricsReporter, > >> > > > selected > >> > > > > > by > >> > > > > > > > > polaris.iceberg-metrics.reporting.type=default. It is > >> > > > log-only, > >> > > > > > > > > effectively quiet unless metrics logging is enabled. > >> > > > > > > > > - There is also a PersistingMetricsReporter, selected > >> by > >> > > > > > > > > polaris.iceberg-metrics.reporting.type=persisting, > >> which > >> > > > > converts > >> > > > > > > > > Iceberg scan/commit reports into Polaris metrics > >> records > >> > and > >> > > > > > writes > >> > > > > > > > > through > >> > > > > > > > > PolarisMetricsManager -> MetricsPersistence. > >> > > > > > > > > - MetricsPersistence currently exists in the > >> persistence > >> > > layer > >> > > > > and > >> > > > > > > is > >> > > > > > > > > inherited by BasePersistence. > >> > > > > > > > > > >> > > > > > > > > My read from the discussion is that the target boundary > >> > should > >> > > > not > >> > > > > be > >> > > > > > > > > MetricsPersistence as inherited by BasePersistence. That > >> path > >> > > > > should > >> > > > > > be > >> > > > > > > > > decomposed as durable implementation detail (taken care > >> of by > >> > > > > > > > > https://github.com/apache/polaris/pull/4397). The > stable > >> > > > extension > >> > > > > > > point > >> > > > > > > > > should instead be the metrics reporting/emitting > boundary. > >> > > > > > > > > > >> > > > > > > > > Concretely, I think the next proposal should be shaped > >> like > >> > > this: > >> > > > > > > > > > >> > > > > > > > > 1. > >> > > > > > > > > > >> > > > > > > > > Define the proper metrics reporting SPI at an > optional > >> > > Iceberg > >> > > > > > > metrics > >> > > > > > > > > extension API layer. > >> > > > > > > > > > >> > > > > > > > > Example direction: > >> > > > > > > > > > >> > > > > > > > > public interface IcebergMetricsReporter { > >> > > > > > > > > void reportMetric(IcebergMetricsReportContext > >> context, > >> > > > > > > > > MetricsReport report); > >> > > > > > > > > } > >> > > > > > > > > > >> > > > > > > > > The context would carry the small Polaris-resolved > >> > envelope: > >> > > > > > > > > catalog/table identity, report type, received > >> timestamp, > >> > and > >> > > > > > > > > request/trace > >> > > > > > > > > context if available. The raw Iceberg MetricsReport > >> > remains > >> > > > the > >> > > > > > > > payload. > >> > > > > > > > > 2. > >> > > > > > > > > > >> > > > > > > > > Keep runtime/service as the REST ingestion and wiring > >> > layer. > >> > > > > > > > > > >> > > > > > > > > The REST handler still resolves the table and > performs > >> > authz > >> > > > > > before > >> > > > > > > > > accepting the report. After that, it calls the > selected > >> > > > > reporting > >> > > > > > > > > implementation. > >> > > > > > > > > 3. > >> > > > > > > > > > >> > > > > > > > > Keep the battery default no-op/log-only. > >> > > > > > > > > > >> > > > > > > > > This preserves an out-of-box safe default and avoids > >> > > > requiring a > >> > > > > > > > durable > >> > > > > > > > > metrics store for every Polaris deployment. > >> > > > > > > > > 4. > >> > > > > > > > > > >> > > > > > > > > Move durable JDBC metrics into a self-contained > >> > > implementation > >> > > > > of > >> > > > > > > the > >> > > > > > > > > reporting SPI. > >> > > > > > > > > > >> > > > > > > > > That implementation can own its schema, bootstrap, > >> > > retention, > >> > > > > and > >> > > > > > > read > >> > > > > > > > > API support. It should not define the core reporting > >> SPI > >> > > > > boundary, > >> > > > > > > and > >> > > > > > > > > it > >> > > > > > > > > should not require metrics persistence to remain > >> inherited > >> > > > from > >> > > > > > > > > BasePersistence. > >> > > > > > > > > 5. > >> > > > > > > > > > >> > > > > > > > > Treat async/event-backed handling as another > >> > implementation > >> > > of > >> > > > > the > >> > > > > > > > same > >> > > > > > > > > reporting SPI. > >> > > > > > > > > > >> > > > > > > > > For example, an event-backed reporter could enqueue a > >> > > metrics > >> > > > > > event > >> > > > > > > > and > >> > > > > > > > > let listeners handle durable storage or other sinks. > >> If we > >> > > > later > >> > > > > > > need > >> > > > > > > > a > >> > > > > > > > > replaceable queue engine, that seems like a shared > >> > > > event/metrics > >> > > > > > > > > substrate > >> > > > > > > > > topic rather than a metrics-only requirement. > >> > > > > > > > > > >> > > > > > > > > This framing lets us keep the REST metrics endpoint > >> simple, > >> > > > > preserve > >> > > > > > > the > >> > > > > > > > > current default behavior, support durable metrics users, > >> and > >> > > > still > >> > > > > > > leave > >> > > > > > > > > room for async/event-backed or external-queue-based > >> > > > > implementations. > >> > > > > > > > > > >> > > > > > > > > I think the main follow-up questions are: > >> > > > > > > > > > >> > > > > > > > > - Do folks agree that the stable SPI boundary should > be > >> > > > metrics > >> > > > > > > > > reporting/emitting, not metrics persistence? > >> > > > > > > > > - Does an optional Iceberg metrics extension API > layer > >> > sound > >> > > > > like > >> > > > > > > the > >> > > > > > > > > right home for this SPI? > >> > > > > > > > > - Should the current durable metrics work be reframed > >> as a > >> > > > > durable > >> > > > > > > > JDBC > >> > > > > > > > > reference implementation of that SPI? > >> > > > > > > > > - What is the smallest PR sequence to get there > without > >> > > > blocking > >> > > > > > the > >> > > > > > > > > current metrics work unnecessarily? > >> > > > > > > > > > >> > > > > > > > > Thanks, > >> > > > > > > > > -ej > >> > > > > > > > > > >> > > > > > > > > On Fri, May 15, 2026 at 5:16 PM Dmitri Bourlatchkov < > >> > > > > > [email protected]> > >> > > > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Hi JB, > >> > > > > > > > > > > >> > > > > > > > > > Could you set up another meeting, please? Same time on > >> > > > Wednesday > >> > > > > as > >> > > > > > > > last > >> > > > > > > > > > time... I hope it works for everyone. > >> > > > > > > > > > > >> > > > > > > > > > Cheers, > >> > > > > > > > > > Dmitri. > >> > > > > > > > > > > >> > > > > > > > > > On Fri, May 15, 2026 at 8:06 PM Yufei Gu < > >> > > [email protected] > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > +1 on another sync call next week. > >> > > > > > > > > > > > >> > > > > > > > > > > Yufei > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > On Fri, May 15, 2026 at 4:52 PM Dmitri Bourlatchkov > < > >> > > > > > > > [email protected]> > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > >> > > > > > > > > > > > Hi All, > >> > > > > > > > > > > > > >> > > > > > > > > > > > WDYT about another sync call next week? > >> > > > > > > > > > > > > >> > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > Dmitri. > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Wed, May 6, 2026 at 5:29 PM Dmitri > Bourlatchkov < > >> > > > > > > > [email protected] > >> > > > > > > > > > > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > > Hi EJ, > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks for the summary! It covers what we > >> discussed > >> > in > >> > > > the > >> > > > > > > > meeting > >> > > > > > > > > > very > >> > > > > > > > > > > > > well, IMHO. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Looking forward to concrete PRs :) > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Cheers, > >> > > > > > > > > > > > > Dmitri. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Wed, May 6, 2026 at 5:08 PM EJ Wang < > >> > > > > > > > > > [email protected] > >> > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> Hi folks, > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> We had a community sync earlier, thanks JB for > >> > > > scheduling > >> > > > > > it. > >> > > > > > > > > Notes > >> > > > > > > > > > > from > >> > > > > > > > > > > > >> the first metrics architecture sync (May 6, > >> 10-11am > >> > > PT). > >> > > > > > > > > Discussion > >> > > > > > > > > > > doc > >> > > > > > > > > > > > >> with per-section status: > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/100h7c4damrUzVuquYbBHM0EvA4LSWuW2IT2dN_7nYVA/edit?tab=t.0 > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> *The meeting covered both topics from the doc. > >> > > > > > Direction-level > >> > > > > > > > > > > alignment > >> > > > > > > > > > > > >> was reached on the headline pieces; details > >> remain > >> > for > >> > > > PR > >> > > > > > > review > >> > > > > > > > > or > >> > > > > > > > > > > > >> follow-up sessions.* > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> *Topic 1 — Persistence schema redesign* > >> > > > > > > > > > > > >> Idea-level alignment on consolidating per-type > >> > tables > >> > > > > > > > > > > > >> (scan_metrics_report, > >> > > > > > > > > > > > >> commit_metrics_report) into a single > >> metrics_report > >> > > > table. > >> > > > > > The > >> > > > > > > > > > > > motivating > >> > > > > > > > > > > > >> cost is the surface area added by every new > >> metric > >> > > type > >> > > > > > today: > >> > > > > > > > new > >> > > > > > > > > > > > table, > >> > > > > > > > > > > > >> SPI method, record class, model, converter, > >> schema > >> > > > > > migration. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> Most schema details are deferred to the schema > >> PR. A > >> > > few > >> > > > > > > > specific > >> > > > > > > > > > > points > >> > > > > > > > > > > > >> came up: > >> > > > > > > > > > > > >> • metric_schema_version: Yufei prefers > dropping > >> > it, > >> > > > > since > >> > > > > > > > there > >> > > > > > > > > is > >> > > > > > > > > > > no > >> > > > > > > > > > > > >> spec-level concept of metrics versioning today > >> and > >> > it > >> > > is > >> > > > > > hard > >> > > > > > > to > >> > > > > > > > > > > define > >> > > > > > > > > > > > >> unilaterally. Robert prefers keeping it, given > >> IRC > >> > v2 > >> > > is > >> > > > > > > coming > >> > > > > > > > > and > >> > > > > > > > > > > the > >> > > > > > > > > > > > >> schema should be considered against its likely > >> > shape; > >> > > > > Robert > >> > > > > > > > also > >> > > > > > > > > > > raised > >> > > > > > > > > > > > >> how to differentiate various payload formats if > >> any. > >> > > > EJ's > >> > > > > > read > >> > > > > > > > is > >> > > > > > > > > > that > >> > > > > > > > > > > > >> this > >> > > > > > > > > > > > >> is a two-way-door decision. We can start > without > >> the > >> > > > > field, > >> > > > > > > and > >> > > > > > > > if > >> > > > > > > > > > IRC > >> > > > > > > > > > > > v2 > >> > > > > > > > > > > > >> changes the shape we would likely roll a > >> > corresponding > >> > > > new > >> > > > > > > > schema > >> > > > > > > > > > > > anyway, > >> > > > > > > > > > > > >> which is not particularly costly. > >> > > > > > > > > > > > >> • Payload format: Robert pointed out that > >> future > >> > > > formats > >> > > > > > > > beyond > >> > > > > > > > > > JSON > >> > > > > > > > > > > > may > >> > > > > > > > > > > > >> be worth supporting. The exact shape is > deferred > >> to > >> > > the > >> > > > > > schema > >> > > > > > > > > > > > discussion. > >> > > > > > > > > > > > >> • Partition strategy: Anand suggested monthly > >> > > > > partitioning > >> > > > > > > > based > >> > > > > > > > > > on > >> > > > > > > > > > > > his > >> > > > > > > > > > > > >> experience as potentially helpful at scale. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> *Topic 2 — Where metrics ingestion and storage > >> > belong* > >> > > > > > > > > > > > >> Idea-level alignment that metrics should be a > >> > > separated > >> > > > > SPI > >> > > > > > > from > >> > > > > > > > > the > >> > > > > > > > > > > > >> entity > >> > > > > > > > > > > > >> persistence stack. Two reasons surfaced: (a) > >> > workloads > >> > > > and > >> > > > > > > > > > capability > >> > > > > > > > > > > > >> requirements diverge enough that coupling them > >> > creates > >> > > > > > > > artificial > >> > > > > > > > > > > > >> constraints, and (b) admin experience improves > >> when > >> > > > > metrics > >> > > > > > > has > >> > > > > > > > > its > >> > > > > > > > > > > own > >> > > > > > > > > > > > >> bootstrap, retention, and lifecycle. Dmitri > noted > >> > > > Polaris > >> > > > > > > being > >> > > > > > > > a > >> > > > > > > > > > > > platform > >> > > > > > > > > > > > >> should have the flexibility to support > different > >> > > > > persistence > >> > > > > > > > > > backends > >> > > > > > > > > > > > per > >> > > > > > > > > > > > >> concern, and pointed to a concrete next step of > >> > > > separating > >> > > > > > the > >> > > > > > > > > JDBC > >> > > > > > > > > > > > >> bootstrap for metrics from the metastore > >> bootstrap. > >> > > > Robert > >> > > > > > > > > proposed > >> > > > > > > > > > an > >> > > > > > > > > > > > >> additional UX extension: detect an > unbootstrapped > >> > > > metrics > >> > > > > > > store > >> > > > > > > > on > >> > > > > > > > > > > first > >> > > > > > > > > > > > >> use and auto-bootstrap rather than requiring an > >> > > explicit > >> > > > > > > manual > >> > > > > > > > > > > > bootstrap > >> > > > > > > > > > > > >> step. > >> > > > > > > > > > > > >> The meeting also confirmed that Polaris metrics > >> can > >> > > > start > >> > > > > > > small > >> > > > > > > > > and > >> > > > > > > > > > > stay > >> > > > > > > > > > > > >> Iceberg-focused. Naming and persistence schema > >> can > >> > > lean > >> > > > > > > > > > > > Iceberg-specific. > >> > > > > > > > > > > > >> If a future expansion to generic-table metrics > or > >> > > > > > operational > >> > > > > > > > > > metrics > >> > > > > > > > > > > > >> arrives, an abstraction layer can be built on > >> top of > >> > > the > >> > > > > > > Iceberg > >> > > > > > > > > > > metrics > >> > > > > > > > > > > > >> reporter at that point. Robert remains on the > >> fence > >> > > and > >> > > > > > would > >> > > > > > > > > prefer > >> > > > > > > > > > > > >> something more generic but did not block the > >> > > direction; > >> > > > > > > Dmitri's > >> > > > > > > > > > read > >> > > > > > > > > > > > was > >> > > > > > > > > > > > >> that the proposed framework already has enough > >> > > > flexibility > >> > > > > > to > >> > > > > > > > > absorb > >> > > > > > > > > > > > >> future > >> > > > > > > > > > > > >> expansion. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> The Trade-offs and Proposed structure sections > in > >> > the > >> > > > doc > >> > > > > > were > >> > > > > > > > not > >> > > > > > > > > > > > >> reviewed > >> > > > > > > > > > > > >> in detail. They remain open for either the next > >> sync > >> > > or > >> > > > PR > >> > > > > > > > review. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> *Cross-cutting alignment — battery-included > plus > >> > > > > pluggable* > >> > > > > > > > > > > > >> A common philosophy emerged from the > discussion. > >> EJ > >> > > > > > summarized > >> > > > > > > > it > >> > > > > > > > > > as: > >> > > > > > > > > > > > >> Polaris should provide a battery-included UX > for > >> > > > beginners > >> > > > > > and > >> > > > > > > > the > >> > > > > > > > > > > > >> flexibility for advanced users to swap the > >> included > >> > > > > battery > >> > > > > > > for > >> > > > > > > > > > > > something > >> > > > > > > > > > > > >> more powerful or tailored to their use case. > The > >> SPI > >> > > > > design > >> > > > > > > > needs > >> > > > > > > > > to > >> > > > > > > > > > > > >> enable > >> > > > > > > > > > > > >> both. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> The inputs that shaped this framing: > >> > > > > > > > > > > > >> • Anand described how his team uses the > current > >> > > > metrics > >> > > > > > > > > > persistence > >> > > > > > > > > > > > >> (three metrics consumers in v1.4). > >> > > > > > > > > > > > >> • Yufei raised Grafana and dashboard > >> integrations > >> > > as a > >> > > > > > > > > destination > >> > > > > > > > > > > use > >> > > > > > > > > > > > >> case beyond the default. > >> > > > > > > > > > > > >> • Robert called out that the current design > is > >> > more > >> > > > > > > > > JDBC-focused. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> Two concrete instances: > >> > > > > > > > > > > > >> • Async metrics intake: Yufei's initial > >> position > >> > was > >> > > > > that > >> > > > > > > > async > >> > > > > > > > > > > should > >> > > > > > > > > > > > >> largely live on the producer side and there is > >> not > >> > > much > >> > > > > > > Polaris > >> > > > > > > > > can > >> > > > > > > > > > > do. > >> > > > > > > > > > > > >> Robert suggested a Polaris-side default is > doable > >> > via > >> > > > > > Vert.x. > >> > > > > > > > > Dmitri > >> > > > > > > > > > > > >> agreed > >> > > > > > > > > > > > >> the direction is worth exploring. The meeting > >> > > converged > >> > > > > on a > >> > > > > > > > > > > > >> battery-included default (likely Vert.x-backed) > >> with > >> > > an > >> > > > > SPI > >> > > > > > > > shape > >> > > > > > > > > > that > >> > > > > > > > > > > > >> lets > >> > > > > > > > > > > > >> power users route to a more scalable backend > >> > > (k8s-hosted > >> > > > > > > queue, > >> > > > > > > > > AWS > >> > > > > > > > > > > SQS, > >> > > > > > > > > > > > >> etc.). > >> > > > > > > > > > > > >> • Pluggable destinations: combining Yufei's > >> > > dashboard > >> > > > > use > >> > > > > > > case > >> > > > > > > > > > with > >> > > > > > > > > > > > >> Robert's JDBC-focused call-out, the meeting > >> agreed > >> > the > >> > > > SPI > >> > > > > > > > should > >> > > > > > > > > be > >> > > > > > > > > > > > >> structured for multiple sinks so integrations > >> become > >> > > > impl > >> > > > > > > > choices > >> > > > > > > > > > > rather > >> > > > > > > > > > > > >> than architectural changes. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> The battery-included default is most likely to > >> use > >> > the > >> > > > > > > existing > >> > > > > > > > > > > > >> JDBC-backed > >> > > > > > > > > > > > >> approach. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> *Direction (idea-level alignment)* > >> > > > > > > > > > > > >> • Single metrics_report table consolidating > >> > per-type > >> > > > > > > metrics, > >> > > > > > > > > > > > replacing > >> > > > > > > > > > > > >> scan_metrics_report and commit_metrics_report > >> > > > > > > > > > > > >> • Iceberg-focused naming and schema for now, > >> > revisit > >> > > > if > >> > > > > > > > > > > generic-table > >> > > > > > > > > > > > or > >> > > > > > > > > > > > >> operational metrics arrive > >> > > > > > > > > > > > >> • Metrics persistence as a separated SPI, not > >> on > >> > > > > > > > BasePersistence > >> > > > > > > > > > > > >> • Bootstrap path separated for metrics, > >> > independent > >> > > of > >> > > > > > > > metastore > >> > > > > > > > > > > > >> bootstrap > >> > > > > > > > > > > > >> • "Battery-included plus pluggable" as the > SPI > >> > > design > >> > > > > > > > philosophy > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> *Open items* > >> > > > > > > > > > > > >> • Schema details: metric_schema_version, > >> payload > >> > > > format, > >> > > > > > IRC > >> > > > > > > > v2 > >> > > > > > > > > > > > >> forward-compat shape > >> > > > > > > > > > > > >> • SPI design details — full review either in > >> the > >> > > next > >> > > > > sync > >> > > > > > > or > >> > > > > > > > in > >> > > > > > > > > > the > >> > > > > > > > > > > > >> corresponding PR > >> > > > > > > > > > > > >> • Schema refactor PR ownership > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> *Action items* > >> > > > > > > > > > > > >> • EJ to take a first stab at the SPI design > and > >> > > > > > potentially > >> > > > > > > > > > partner > >> > > > > > > > > > > > with > >> > > > > > > > > > > > >> Anand to incorporate the lessons learned from > the > >> > > > existing > >> > > > > > > > > reporter > >> > > > > > > > > > > and > >> > > > > > > > > > > > >> persistence work. > >> > > > > > > > > > > > >> • Schema refactor PR ownership is not yet > >> decided. > >> > > If > >> > > > > > anyone > >> > > > > > > > is > >> > > > > > > > > > > > >> interested in driving it, reply on this thread. > >> > > > > > > > > > > > >> • JB to schedule the next sync, tentatively > in > >> two > >> > > > > weeks. > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> -ej > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> On Mon, Apr 27, 2026 at 3:07 PM EJ Wang < > >> > > > > > > > > > > [email protected] > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> wrote: > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > Thanks Yufei for the +1. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > JB, could you help add a biweekly metrics > >> > > architecture > >> > > > > > sync > >> > > > > > > to > >> > > > > > > > > the > >> > > > > > > > > > > > >> Polaris > >> > > > > > > > > > > > >> > community calendar? I'm thinking Thursdays at > >> > 9-10am > >> > > > PT, > >> > > > > > on > >> > > > > > > > the > >> > > > > > > > > > > > >> off-weeks > >> > > > > > > > > > > > >> > from the community meeting (starting May 7), > 60 > >> > > > minutes. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > Here's a rough agenda to work through over > the > >> > first > >> > > > few > >> > > > > > > > > sessions, > >> > > > > > > > > > > > >> grouped > >> > > > > > > > > > > > >> > by priority: > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > *First: foundational direction* > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > 1. MetricsPersistence: public SPI or > internal > >> > > > > > > implementation > >> > > > > > > > > > > detail? > >> > > > > > > > > > > > >> > • Marked @Beta, javadoc calls it a > >> "Service > >> > > > > Provider > >> > > > > > > > > > > Interface", > >> > > > > > > > > > > > >> but > >> > > > > > > > > > > > >> > only one consumer (JdbcBasePersistenceImpl), > >> lives > >> > > on > >> > > > > > > > > > > BasePersistence. > >> > > > > > > > > > > > >> If > >> > > > > > > > > > > > >> > demoted to a private helper inside a > persisting > >> > > > reporter > >> > > > > > > impl, > >> > > > > > > > > > most > >> > > > > > > > > > > > >> > downstream design decisions become > >> implementation > >> > > > > details > >> > > > > > > > rather > >> > > > > > > > > > > than > >> > > > > > > > > > > > >> > contract questions. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > 2. Persistence schema redesign > >> > > > > > > > > > > > >> > • Current two-table layout > >> > > (scan_metrics_report, > >> > > > > > > > > > > > >> > commit_metrics_report) with ~25 flattened > >> columns > >> > > > each. > >> > > > > > > Every > >> > > > > > > > > new > >> > > > > > > > > > > > metric > >> > > > > > > > > > > > >> > type requires a new table, SPI method, record > >> > class, > >> > > > > > model, > >> > > > > > > > > > > converter, > >> > > > > > > > > > > > >> and > >> > > > > > > > > > > > >> > schema migration. Direction to explore: > single > >> > table > >> > > > > with > >> > > > > > > > > > > metric_type > >> > > > > > > > > > > > >> enum, > >> > > > > > > > > > > > >> > schema_version, and JSON payload column. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > *Second: design details once direction is > set* > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > 3. Partition key strategy > >> > > > > > > > > > > > >> > • Single-table design means scan metrics > >> at > >> > > scale > >> > > > > > will > >> > > > > > > > have > >> > > > > > > > > > > high > >> > > > > > > > > > > > >> > write concurrency per table. Schema needs to > >> > expose > >> > > > > enough > >> > > > > > > > > > structure > >> > > > > > > > > > > > for > >> > > > > > > > > > > > >> > backends to shard by entity or time range. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > 4. Read/write path consistency > >> > > > > > > > > > > > >> > • Writes go through > PolarisMetricsManager > >> on > >> > > > > > > > > > MetaStoreManager. > >> > > > > > > > > > > > >> Reads > >> > > > > > > > > > > > >> > bypass MetaStoreManager and go straight to > >> > > > > > BasePersistence, > >> > > > > > > > > > > excluding > >> > > > > > > > > > > > >> > non-JDBC backends from the read API. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > *Third: cleanup and alignment* > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > 5. PolarisMetricsReporter naming > >> > > > > > > > > > > > >> > • Only handles IRC > >> (ScanReport/CommitReport), > >> > > > > doesn't > >> > > > > > > > cover > >> > > > > > > > > > > > generic > >> > > > > > > > > > > > >> > tables or operational metrics. Name is > broader > >> > than > >> > > > > scope. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > 6. PolarisMetricsManager facade passthrough > >> > > > > > > > > > > > >> > • Entire default method is > >> > > > > > > > > > > > >> callCtx.getMetaStore().writeScanReport(). > >> > > > > > > > > > > > >> > Zero logic, passes Level 1 straight through > to > >> > Level > >> > > > 3. > >> > > > > > Same > >> > > > > > > > > > > > >> anti-pattern > >> > > > > > > > > > > > >> > as PolarisEventManager. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > 7. Iceberg community alignment > >> > > > > > > > > > > > >> > • Payload-type extension needs > discussion > >> on > >> > > > > > > dev@iceberg. > >> > > > > > > > > > > > >> obelix74's > >> > > > > > > > > > > > >> > Feb thread got zero replies. Needs a > committer > >> > > voice. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > Lets confirm prioritization in the first > >> session. > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > -ej > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > On Tue, Apr 21, 2026 at 3:18 PM Yufei Gu < > >> > > > > > > > [email protected]> > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> >> Thanks everyone for continuing to drive this > >> > > > forward. I > >> > > > > > > agree > >> > > > > > > > > > that > >> > > > > > > > > > > > the > >> > > > > > > > > > > > >> >> problem is getting complex enough that a > more > >> > > > > structured > >> > > > > > > > > > discussion > >> > > > > > > > > > > > >> would > >> > > > > > > > > > > > >> >> help. > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> +1 on setting up a biweekly sync for the > >> metrics > >> > > > > > > > architecture. > >> > > > > > > > > > I’m > >> > > > > > > > > > > > >> happy > >> > > > > > > > > > > > >> >> to > >> > > > > > > > > > > > >> >> join. > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> Yufei > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> On Tue, Apr 21, 2026 at 2:34 PM EJ Wang < > >> > > > > > > > > > > > >> [email protected]> > >> > > > > > > > > > > > >> >> wrote: > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> >> > Also, I've been looking more closely at > the > >> > > > > > *persistence > >> > > > > > > > > schema > >> > > > > > > > > > > in > >> > > > > > > > > > > > >> the > >> > > > > > > > > > > > >> >> > current metrics work*, and I think > there's a > >> > > > > structural > >> > > > > > > > > > rigidity > >> > > > > > > > > > > > >> problem > >> > > > > > > > > > > > >> >> > worth raising before the shape gets locked > >> in. > >> > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > >> >> > Right now we have two separate tables > >> > > > > > > (scan_metrics_report > >> > > > > > > > > and > >> > > > > > > > > > > > >> >> > commit_metrics_report), each with ~25 > >> flattened > >> > > > > columns > >> > > > > > > > that > >> > > > > > > > > > > > directly > >> > > > > > > > > > > > >> >> > mirror the Iceberg report fields. The SPI > >> > follows > >> > > > the > >> > > > > > > same > >> > > > > > > > > > split: > >> > > > > > > > > > > > >> >> > writeScanReport and writeCommitReport as > >> > separate > >> > > > > > > methods, > >> > > > > > > > > with > >> > > > > > > > > > > > >> per-type > >> > > > > > > > > > > > >> >> > record classes, converters, and model > >> objects. > >> > > *The > >> > > > > > > > practical > >> > > > > > > > > > > cost: > >> > > > > > > > > > > > >> >> > adding a new metric type (operational > >> metrics, > >> > > for > >> > > > > > > example) > >> > > > > > > > > > > > requires > >> > > > > > > > > > > > >> a > >> > > > > > > > > > > > >> >> new > >> > > > > > > > > > > > >> >> > table, a new SPI method, a new record > >> class, a > >> > > new > >> > > > > > model > >> > > > > > > > > > class, a > >> > > > > > > > > > > > new > >> > > > > > > > > > > > >> >> > converter branch, and a schema migration*. > >> > > That's a > >> > > > > lot > >> > > > > > > of > >> > > > > > > > > > > surface > >> > > > > > > > > > > > >> area > >> > > > > > > > > > > > >> >> > for what should be "one more kind of > >> metric." > >> > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > >> >> > *My bias* would be toward a single metrics > >> > table > >> > > > with > >> > > > > > *a > >> > > > > > > > > typed > >> > > > > > > > > > > JSON > >> > > > > > > > > > > > >> >> > payload*. Something like: metric_type > >> (enum), > >> > > > > > entity_id, > >> > > > > > > > > > > > >> >> > table_identifier, snapshot_id (nullable), > >> > > > > received_ts, > >> > > > > > > > > > > > >> schema_version, > >> > > > > > > > > > > > >> >> and > >> > > > > > > > > > > > >> >> > a payload column for the metric-specific > >> data. > >> > > The > >> > > > > > > > > metric_type > >> > > > > > > > > > + > >> > > > > > > > > > > > >> >> > schema_version pair gives us a > >> > forward-compatible > >> > > > > > > contract > >> > > > > > > > > for > >> > > > > > > > > > > the > >> > > > > > > > > > > > >> >> payload > >> > > > > > > > > > > > >> >> > shape. Adding a new metric type becomes an > >> enum > >> > > > value > >> > > > > > > and a > >> > > > > > > > > > > payload > >> > > > > > > > > > > > >> >> schema, > >> > > > > > > > > > > > >> >> > not a schema migration. One thing I think > we > >> > need > >> > > > to > >> > > > > be > >> > > > > > > > > > > deliberate > >> > > > > > > > > > > > >> >> about is > >> > > > > > > > > > > > >> >> > the partition key design. If all metric > >> types > >> > > land > >> > > > in > >> > > > > > one > >> > > > > > > > > > table, > >> > > > > > > > > > > > scan > >> > > > > > > > > > > > >> >> > metrics at scale (high concurrency, high > >> > > frequency > >> > > > > > across > >> > > > > > > > > many > >> > > > > > > > > > > > >> tables) > >> > > > > > > > > > > > >> >> > could easily create hot partitions. We'd > >> want > >> > the > >> > > > > > > > persistence > >> > > > > > > > > > > layer > >> > > > > > > > > > > > >> to > >> > > > > > > > > > > > >> >> be > >> > > > > > > > > > > > >> >> > able to shard by entity or time range, and > >> that > >> > > > means > >> > > > > > the > >> > > > > > > > > > logical > >> > > > > > > > > > > > >> schema > >> > > > > > > > > > > > >> >> > needs to expose enough structure for > >> backends > >> > to > >> > > > > > > partition > >> > > > > > > > > on. > >> > > > > > > > > > I > >> > > > > > > > > > > > >> don't > >> > > > > > > > > > > > >> >> > think the current flattened layout gives > us > >> > that. > >> > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > >> >> > This is getting complex enough that I > don't > >> > think > >> > > > > > ad-hoc > >> > > > > > > > > PR/ML > >> > > > > > > > > > > > >> threads > >> > > > > > > > > > > > >> >> > will converge well. *Would people be open > >> to a > >> > > > > biweekly > >> > > > > > > > sync > >> > > > > > > > > > for > >> > > > > > > > > > > > >> metrics > >> > > > > > > > > > > > >> >> > architecture?* I think 30 minutes every > two > >> > weeks > >> > > > > with > >> > > > > > > > > > interested > >> > > > > > > > > > > > >> >> parties > >> > > > > > > > > > > > >> >> > would be enough to work through the > schema, > >> SPI > >> > > > > shape, > >> > > > > > > and > >> > > > > > > > > read > >> > > > > > > > > > > API > >> > > > > > > > > > > > >> >> design > >> > > > > > > > > > > > >> >> > together. Happy to help set that up. > >> > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > >> >> > -ej > >> > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > >> >> > On Mon, Apr 20, 2026 at 2:19 PM EJ Wang < > >> > > > > > > > > > > > >> [email protected] > >> > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > >> >> > wrote: > >> > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > >> >> >> Reviewed #4115, left a comment on the > code > >> > > > > > organization > >> > > > > > > > > side. > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> One thing stood out: the metrics write > path > >> > > enters > >> > > > > > > through > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on > MetaStoreManager, > >> but > >> > > the > >> > > > > new > >> > > > > > > > read > >> > > > > > > > > > path > >> > > > > > > > > > > > >> >> bypasses > >> > > > > > > > > > > > >> >> >> MetaStoreManager entirely and goes > >> straight to > >> > > > > > > > > BasePersistence > >> > > > > > > > > > > via > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). That means > the > >> > read > >> > > > API > >> > > > > > only > >> > > > > > > > > works > >> > > > > > > > > > > for > >> > > > > > > > > > > > >> >> backends > >> > > > > > > > > > > > >> >> >> that implement BasePersistence. NoSQL and > >> > remote > >> > > > > > > backends > >> > > > > > > > > > can't > >> > > > > > > > > > > > >> >> participate. > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> Stepping back, I think the metrics > >> subsystem > >> > is > >> > > > > > growing > >> > > > > > > > into > >> > > > > > > > > > > > >> something > >> > > > > > > > > > > > >> >> >> real (write + read + REST API + AuthZ + > >> > > > pagination) > >> > > > > > *but > >> > > > > > > > the > >> > > > > > > > > > > > >> >> persistence > >> > > > > > > > > > > > >> >> >> side is split across two layers in a way > >> > that's > >> > > > hard > >> > > > > > to > >> > > > > > > > > > > extend*. I > >> > > > > > > > > > > > >> put > >> > > > > > > > > > > > >> >> >> together two diagrams to show what I mean > >> (my > >> > > best > >> > > > > > > > effort). > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> *Current state* (Diagram 1): three > >> interfaces > >> > at > >> > > > > three > >> > > > > > > > > > different > >> > > > > > > > > > > > >> >> levels. > >> > > > > > > > > > > > >> >> >> The engine-facing SPI > >> (PolarisMetricsReporter) > >> > > is > >> > > > > > clean. > >> > > > > > > > But > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager on MetaStoreManager > >> is a > >> > > > > > > passthrough > >> > > > > > > > > to > >> > > > > > > > > > > > >> >> >> MetricsPersistence on BasePersistence. > The > >> > @Beta > >> > > > > > > > annotation > >> > > > > > > > > > and > >> > > > > > > > > > > > SPI > >> > > > > > > > > > > > >> >> javadoc > >> > > > > > > > > > > > >> >> >> are on the BasePersistence layer, while > the > >> > > actual > >> > > > > > > > extension > >> > > > > > > > > > > > points > >> > > > > > > > > > > > >> >> >> (PolarisMetricsReporter, > >> > PolarisMetricsManager) > >> > > > > carry > >> > > > > > no > >> > > > > > > > > > > stability > >> > > > > > > > > > > > >> >> >> annotation. The write path goes through > the > >> > > > > > > > MetaStoreManager > >> > > > > > > > > > > > layer, > >> > > > > > > > > > > > >> the > >> > > > > > > > > > > > >> >> >> read path doesn't. > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> *What I envision* (Diagram 2): two SPIs > at > >> two > >> > > > > levels. > >> > > > > > > > > > > > >> >> >> PolarisMetricsReporter stays as the > >> > > engine-facing > >> > > > > SPI. > >> > > > > > > > > > > > >> >> >> PolarisMetricsManager becomes the > >> > backend-facing > >> > > > SPI > >> > > > > > > with > >> > > > > > > > > both > >> > > > > > > > > > > > write > >> > > > > > > > > > > > >> >> and > >> > > > > > > > > > > > >> >> >> read methods at the MetaStoreManager > level, > >> > > where > >> > > > > any > >> > > > > > > > > backend > >> > > > > > > > > > > > (JDBC, > >> > > > > > > > > > > > >> >> NoSQL, > >> > > > > > > > > > > > >> >> >> remote) can implement them. > >> MetricsPersistence > >> > > on > >> > > > > > > > > > > BasePersistence > >> > > > > > > > > > > > >> goes > >> > > > > > > > > > > > >> >> >> away. Where metrics actually land is an > >> > > > > implementation > >> > > > > > > > > detail, > >> > > > > > > > > > > > not a > >> > > > > > > > > > > > >> >> core > >> > > > > > > > > > > > >> >> >> interface. > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> *Minor naming thing*: > >> PolarisMetricsReporter > >> > is > >> > > > > > broader > >> > > > > > > > than > >> > > > > > > > > > > what > >> > > > > > > > > > > > it > >> > > > > > > > > > > > >> >> >> actually handles. It only accepts Iceberg > >> REST > >> > > > > Catalog > >> > > > > > > > > metrics > >> > > > > > > > > > > > >> >> (ScanReport, > >> > > > > > > > > > > > >> >> >> CommitReport via MetricsReport). Generic > >> table > >> > > > > metrics > >> > > > > > > or > >> > > > > > > > > > > > >> operational > >> > > > > > > > > > > > >> >> >> metrics aren't in scope. Not blocking, > but > >> > worth > >> > > > > > noting > >> > > > > > > if > >> > > > > > > > > the > >> > > > > > > > > > > > >> metrics > >> > > > > > > > > > > > >> >> >> surface expands. > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> *Rough sketch of how to get there*: > >> > > > > > > > > > > > >> >> >> 1. Add read methods to > >> PolarisMetricsManager > >> > > > > > > > > > (listScanReports, > >> > > > > > > > > > > > >> >> >> listCommitReports) with default no-op, > >> same as > >> > > the > >> > > > > > > > existing > >> > > > > > > > > > > write > >> > > > > > > > > > > > >> >> methods. > >> > > > > > > > > > > > >> >> >> (Probably make PolarisMetricsManager more > >> > > explicit > >> > > > > on > >> > > > > > > > being > >> > > > > > > > > > > > Iceberg > >> > > > > > > > > > > > >> >> >> specific like package name or class name > >> etc.) > >> > > > > > > > > > > > >> >> >> 2. Wire MetricsReportsService through > >> > > > > > MetaStoreManager > >> > > > > > > > > > instead > >> > > > > > > > > > > > of > >> > > > > > > > > > > > >> >> >> callContext.getMetaStore(). > >> > > > > > > > > > > > >> >> >> 3. Extract metrics persistence from > >> > > > > > > > > JdbcBasePersistenceImpl > >> > > > > > > > > > > into > >> > > > > > > > > > > > >> its > >> > > > > > > > > > > > >> >> >> own class. That file carries ~7 > >> > > responsibilities, > >> > > > > > > metrics > >> > > > > > > > > > being > >> > > > > > > > > > > > one > >> > > > > > > > > > > > >> of > >> > > > > > > > > > > > >> >> them. > >> > > > > > > > > > > > >> >> >> 4. Remove MetricsPersistence from > >> > > > BasePersistence. > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> *None of this needs to happen in #4115. > >> But if > >> > > the > >> > > > > > > > direction > >> > > > > > > > > > > makes > >> > > > > > > > > > > > >> >> sense, > >> > > > > > > > > > > > >> >> >> it would be good to align before the > >> metrics > >> > > > surface > >> > > > > > > grows > >> > > > > > > > > > > > further. > >> > > > > > > > > > > > >> >> Curious > >> > > > > > > > > > > > >> >> >> what others think.* > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> *My mental model note*: Level 1 > >> > > MetaStoreManager; > >> > > > > > level > >> > > > > > > 2 > >> > > > > > > > > > > > >> transactional > >> > > > > > > > > > > > >> >> >> persistence; level 3 base persistence > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> Diagram 1 > >> > > > > > > > > > > > >> >> >> < > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.plantuml.com/plantuml/uml/bLHDR-Cs4BthLmpIYupw0zbkKQ1r3M-S7Bp8xhhM7WCOb3IM65EaGD9EX2RzxHrHb4CxRelwa4YSDu_lpOVcnZ9jzvM8BBS2uGjQpJC3dtHMSekPtMk44IpsMgEqa5XcCOhCZikQQLP1pR8TAp2n3ILhmZDP20m0fcIvUkAoW2qJXd9z1bpToO9BX3WXu0ucy5rpgGPNm0nW5_epUWtm2Ue3pn3kMOFQmKntGZW0BYtgBSi8k5A2QMwybJNMIbFiGSR9QZc4nUqIvikStF0jHprua5C-amge42aNt3R0f5JaaoivdV2Pkqbx4hee4ymOkBh5BTiB-_uIeGeo8zL8rPsPl4DktdEiK1jkB1NdZCRbrSTecDe_mlHbF0wvBmCkaOH5_S8a_TTTKI6-nmCAkEw4LpxsZ-LbYLKQFKMNOgf_wuM7_bV9gOer5SYMMksBSWXFcbi49KNZXNLicwfe3TETC7gPdPqI7uBcHMb1RSzYq34c6PDUM9mn8HRsUTZEiDBve3NjVZumBj0U7SS37mGO7vcwtiK-_pU7U7L_f-digo9YbhSwIfMRwIITKGXbxdIUTCGF1SeCJxloKsU-3k9ddRbX1eDq1q_fx1JbBGT0glVyXimDuP4TQ5qpCAmnGEj2s_6n5mtn1z-97-63itFQZLPO1Ev2tu_WF7Ju-VPc0Skg5bYXxBhkY1xpD7EM_7fyflSpIsqMgVth5xhVr4eQxWQ8enaSAJQSG16yFSDuJ798rrcXr_3n-lfdk7icQjEBmFujL7AodiP_Y4Z7-YxvtZNs4zMgpNTl6tF8sglyPsmqchrjvQ-m-aP94r-TwCA2Ka8upPJZwtvSpoYCXkYMZU2NXvRMBfq9P3i3Le4VAZUAlUZ_oPKsxPgY0Q_BSKLkyr9bhQhQrJjo_x3TPlIB0DPjnMfcIoYP0QaYw1a0fTKDr8fB6ntNuvmoL1ZGkXa69Njh43zf9GiGxHQrA_jDYWRSzF5--WmTVrN97_Sm8LbLUy_lGBmLanJjFkDlGkRqjA_4tm00 > >> > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > >> >> >> : > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> [image: image.png] > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> Diagram 2 > >> > > > > > > > > > > > >> >> >> < > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://www.plantuml.com/plantuml/uml/VLLDR-8m4BtdLupO2sWBLVU8AaGB7AXAbssGzb896SS9RXqxjKqBwkv_tt7iV43fdaZYDpFlpRmnOsE9jhjSH9PRmM31hERKm8scMsuPjJlDe0yheZDc8RR4iYWoBrmMH9CS2a9VICPYUy1OZN0YCy5Q0BCbYNhdCeEK28En8G8wCvbnoQ0R8_05Bc6bkLIz3X03p1zzH7zR-9ZfDquPt9C3qoNCX2yV4G2NbkcKu5jdgGJHt0GbZwnG6i-UP3TUpk5gM6Ldqke350eZUqzoCft3U9xWHvxoa5-7K4nF1J46EbEMafsmdrCBbQ44gVggy18IZrn_ph5asd1ZiIKdQSgueZvjXrQFSFrdC3YN-nXmBacxbGiYyLVxLaBtdhqn0LSzdBDhqQtQoOJeGyad3z0lUqnYgpGB6Ns8oVyta00Dy_WnX0tIOZ8v6SYxHll1TrH6aejAik-mh-AphVFCwSUQqFypElag5QRGFDjQKEd96K1P8QP41c9TzA_IIQyvdAWyv_RSiS3skb0_EzDDkK2v5xWF6MiGFlvhpFLcD2Dq2pml14gaF67eQkmd8gulDoC4kSOu6KVpkvlUJg1RTbWISU40RdBUUS_9XfRZ2dwxm_SW8LYFISgm_MnlDQ6M9P1gbKEc4X-2pH_FvJCkCqm9pbVjD6LrwdLeOrDWfOaqc8Wh9BE85oNKxkNQ6o4yGRy_Eae0G_G8tZv81d3bHDB23WOdisohVr3nh_j6lbSjbNaLRTc8UgtPbAU1J_tygOfZX9DWEJeHDvYx-qmSi5FgNLPZwHrHcUsncGQ5-skhUclpE5fo4ounpFauYrUbkU6ccfnxMvitwag4IyerhTxj8In_Oj1bDO4pQru674loYrGlULHLEGCjwJJ8gDoVZR8MxO4BT3IzRvIcAQKezC6xpziGnTyImrfEGyJI_OcKfgtxIvnTqFEMS17L9Z-jsARN5FmTheP7HtSdtOMT0B4GY2FYHXxgQmMtj2bRqiLFGapiVe1_QVKDrkqXcm83aFEXnMYCZ-xlyHy > >> > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > >> >> >> : > >> > > > > > > > > > > > >> >> >> [image: image.png] > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> -ej > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >> On Wed, Apr 15, 2026 at 8:22 AM Dmitri > >> > > > Bourlatchkov > >> > > > > < > >> > > > > > > > > > > > >> [email protected]> > >> > > > > > > > > > > > >> >> >> wrote: > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> >>> Hi All, > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> Heads up: The current state of PR [4115] > >> > looks > >> > > > > pretty > >> > > > > > > > solid > >> > > > > > > > > > to > >> > > > > > > > > > > > me. > >> > > > > > > > > > > > >> I > >> > > > > > > > > > > > >> >> >>> believe this PR is approaching a > mergeable > >> > > > > condition. > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> Please post your reviews if you have any > >> > > > comments. > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> [4115] > >> > > > https://github.com/apache/polaris/pull/4115 > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> Thanks, > >> > > > > > > > > > > > >> >> >>> Dmitri. > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> On Tue, Mar 3, 2026 at 3:29 PM Anand > Kumar > >> > > > Sankaran > >> > > > > > via > >> > > > > > > > > dev < > >> > > > > > > > > > > > >> >> >>> [email protected]> wrote: > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >>> > Hi Yufei and Dmitri, > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > Here is a proposal for the REST > >> endpoints > >> > for > >> > > > > > metrics > >> > > > > > > > and > >> > > > > > > > > > > > events. > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > https://github.com/apache/polaris/pull/3924/changes > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > I did not see any precursors for > >> raising a > >> > PR > >> > > > for > >> > > > > > > > > > proposals, > >> > > > > > > > > > > so > >> > > > > > > > > > > > >> >> trying > >> > > > > > > > > > > > >> >> >>> > this. Please let me know what you > >> think. > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > - > >> > > > > > > > > > > > >> >> >>> > Anand > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > From: Anand Kumar Sankaran < > >> > > > > > > [email protected] > >> > > > > > > > > > >> > > > > > > > > > > > >> >> >>> > Date: Monday, March 2, 2026 at > 10:25 AM > >> > > > > > > > > > > > >> >> >>> > To: [email protected] < > >> > > > > [email protected] > >> > > > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and > Audit > >> > > Trail > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > About the REST API, based on my use > >> cases: > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > 1. > >> > > > > > > > > > > > >> >> >>> > I want to be able to query commit > >> metrics > >> > to > >> > > > > track > >> > > > > > > > files > >> > > > > > > > > > > added > >> > > > > > > > > > > > / > >> > > > > > > > > > > > >> >> >>> removed > >> > > > > > > > > > > > >> >> >>> > per commit, along with record counts. > >> The > >> > > > > ingestion > >> > > > > > > > > > pipeline > >> > > > > > > > > > > > that > >> > > > > > > > > > > > >> >> >>> writes > >> > > > > > > > > > > > >> >> >>> > this data is owned by us and we are > >> > > guaranteed > >> > > > to > >> > > > > > > write > >> > > > > > > > > > this > >> > > > > > > > > > > > >> >> >>> information > >> > > > > > > > > > > > >> >> >>> > for each write. > >> > > > > > > > > > > > >> >> >>> > 2. > >> > > > > > > > > > > > >> >> >>> > I want to be able to query scan > metrics > >> for > >> > > > > read. I > >> > > > > > > > > > > understand > >> > > > > > > > > > > > >> >> clients > >> > > > > > > > > > > > >> >> >>> do > >> > > > > > > > > > > > >> >> >>> > not fulfill this requirement. > >> > > > > > > > > > > > >> >> >>> > 3. > >> > > > > > > > > > > > >> >> >>> > I want to be able to query the events > >> table > >> > > > > (events > >> > > > > > > are > >> > > > > > > > > > > > >> persisted) - > >> > > > > > > > > > > > >> >> >>> this > >> > > > > > > > > > > > >> >> >>> > may supersede #2, I am not sure yet. > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > All this information is in the JDBC > >> based > >> > > > > > persistence > >> > > > > > > > > model > >> > > > > > > > > > > and > >> > > > > > > > > > > > >> is > >> > > > > > > > > > > > >> >> >>> > persisted in the metastore. I > currently > >> > don’t > >> > > > > have > >> > > > > > a > >> > > > > > > > need > >> > > > > > > > > > to > >> > > > > > > > > > > > >> query > >> > > > > > > > > > > > >> >> >>> > prometheus or open telemetry. I do > >> publish > >> > > some > >> > > > > > > events > >> > > > > > > > to > >> > > > > > > > > > > > >> Prometheus > >> > > > > > > > > > > > >> >> >>> and > >> > > > > > > > > > > > >> >> >>> > they are forwarded to our dashboards > >> > > elsewhere. > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > About the CLI utilities, I meant the > >> admin > >> > > user > >> > > > > > > > > utilities. > >> > > > > > > > > > In > >> > > > > > > > > > > > >> one of > >> > > > > > > > > > > > >> >> >>> the > >> > > > > > > > > > > > >> >> >>> > earliest drafts of my proposal, > Prashant > >> > > > > mentioned > >> > > > > > > that > >> > > > > > > > > the > >> > > > > > > > > > > > >> metrics > >> > > > > > > > > > > > >> >> >>> tables > >> > > > > > > > > > > > >> >> >>> > can grow indefinitely and that a > similar > >> > > > problem > >> > > > > > > exists > >> > > > > > > > > > with > >> > > > > > > > > > > > the > >> > > > > > > > > > > > >> >> events > >> > > > > > > > > > > > >> >> >>> > table as well. We discussed that > >> cleaning > >> > up > >> > > of > >> > > > > old > >> > > > > > > > > records > >> > > > > > > > > > > > from > >> > > > > > > > > > > > >> >> both > >> > > > > > > > > > > > >> >> >>> > metrics tables and events tables can > be > >> > done > >> > > > via > >> > > > > a > >> > > > > > > CLI > >> > > > > > > > > > > utility. > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > I see that Yufei has covered the > >> discussion > >> > > > about > >> > > > > > > > > > > datasources. > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > - > >> > > > > > > > > > > > >> >> >>> > Anand > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > From: Yufei Gu <[email protected]> > >> > > > > > > > > > > > >> >> >>> > Date: Friday, February 27, 2026 at > >> 9:54 PM > >> > > > > > > > > > > > >> >> >>> > To: [email protected] < > >> > > > > [email protected] > >> > > > > > > > >> > > > > > > > > > > > >> >> >>> > Subject: Re: Polaris Telemetry and > Audit > >> > > Trail > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > This Message Is From an External > Sender > >> > > > > > > > > > > > >> >> >>> > This message came from outside your > >> > > > organization. > >> > > > > > > > > > > > >> >> >>> > Report Suspicious< > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B699ote2uMwpOZw8i0QMCGO2Szc-HshuABGhGvwPJcymE6G2oUUxtS8xDkSrtGTPm_I3QnVDHoLMk50m9v8z_nZKTkd-bnVUbreF1u0WnfV_X5eYevZl_$ > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > As I mentioned in > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://urldefense.com/v3/__https://github.com/apache/polaris/issues/3890__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKOxrvDU0$ > >> > > > > > > > > > > > >> >> >>> , > >> > > > > > > > > > > > >> >> >>> > supporting > >> > > > > > > > > > > > >> >> >>> > multiple data sources is not a trivial > >> > > change. > >> > > > I > >> > > > > > > would > >> > > > > > > > > > > strongly > >> > > > > > > > > > > > >> >> >>> recommend > >> > > > > > > > > > > > >> >> >>> > starting with a design document to > >> > carefully > >> > > > > > evaluate > >> > > > > > > > the > >> > > > > > > > > > > > >> >> architectural > >> > > > > > > > > > > > >> >> >>> > implications and long term impact. > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > A REST endpoint to query metrics seems > >> > > > reasonable > >> > > > > > > given > >> > > > > > > > > the > >> > > > > > > > > > > > >> current > >> > > > > > > > > > > > >> >> >>> JDBC > >> > > > > > > > > > > > >> >> >>> > based persistence model. That said, we > >> may > >> > > also > >> > > > > > > > consider > >> > > > > > > > > > > > >> alternative > >> > > > > > > > > > > > >> >> >>> > storage models. For example, if we > later > >> > > adopt > >> > > > a > >> > > > > > time > >> > > > > > > > > > series > >> > > > > > > > > > > > >> system > >> > > > > > > > > > > > >> >> >>> such as > >> > > > > > > > > > > > >> >> >>> > Prometheus to store metrics, the query > >> > model > >> > > > and > >> > > > > > > access > >> > > > > > > > > > > > patterns > >> > > > > > > > > > > > >> >> would > >> > > > > > > > > > > > >> >> >>> be > >> > > > > > > > > > > > >> >> >>> > fundamentally different. Designing the > >> REST > >> > > API > >> > > > > > > without > >> > > > > > > > > > > > >> considering > >> > > > > > > > > > > > >> >> >>> these > >> > > > > > > > > > > > >> >> >>> > potential evolutions may limit > >> flexibility. > >> > > I'd > >> > > > > > > suggest > >> > > > > > > > > to > >> > > > > > > > > > > > start > >> > > > > > > > > > > > >> >> with > >> > > > > > > > > > > > >> >> >>> the > >> > > > > > > > > > > > >> >> >>> > use case. > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > Yufei > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > On Fri, Feb 27, 2026 at 3:42 PM Dmitri > >> > > > > > Bourlatchkov < > >> > > > > > > > > > > > >> >> [email protected]> > >> > > > > > > > > > > > >> >> >>> > wrote: > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > Hi Anand, > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > Sharing my view... subject to > >> discussion: > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > 1. Adding non-IRC REST API to > Polaris > >> is > >> > > > > > perfectly > >> > > > > > > > > fine. > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > Figuring out specific endpoint URIs > >> and > >> > > > > payloads > >> > > > > > > > might > >> > > > > > > > > > > > require > >> > > > > > > > > > > > >> a > >> > > > > > > > > > > > >> >> few > >> > > > > > > > > > > > >> >> >>> > > roundtrips, so opening a separate > >> thread > >> > > for > >> > > > > that > >> > > > > > > > might > >> > > > > > > > > > be > >> > > > > > > > > > > > >> best. > >> > > > > > > > > > > > >> >> >>> > > Contributors commonly create Google > >> Docs > >> > > for > >> > > > > new > >> > > > > > > API > >> > > > > > > > > > > > proposals > >> > > > > > > > > > > > >> too > >> > > > > > > > > > > > >> >> >>> (they > >> > > > > > > > > > > > >> >> >>> > > fairly easy to update as the email > >> > > discussion > >> > > > > > > > > > progresses). > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > There was a suggestion to try > Markdown > >> > > (with > >> > > > > PRs) > >> > > > > > > for > >> > > > > > > > > > > > proposals > >> > > > > > > > > > > > >> >> [1] > >> > > > > > > > > > > > >> >> >>> ... > >> > > > > > > > > > > > >> >> >>> > > feel free to give it a try if you > are > >> > > > > comfortable > >> > > > > > > > with > >> > > > > > > > > > > that. > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > 2. Could you clarify whether you > mean > >> end > >> > > > user > >> > > > > > > > > utilities > >> > > > > > > > > > or > >> > > > > > > > > > > > >> admin > >> > > > > > > > > > > > >> >> >>> user > >> > > > > > > > > > > > >> >> >>> > > utilities? In the latter case those > >> might > >> > > be > >> > > > > more > >> > > > > > > > > > suitable > >> > > > > > > > > > > > for > >> > > > > > > > > > > > >> the > >> > > > > > > > > > > > >> >> >>> Admin > >> > > > > > > > > > > > >> >> >>> > > CLI (java) not the Python CLI, IMHO. > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > Why would these utilities be common > >> with > >> > > > > events? > >> > > > > > > > IMHO, > >> > > > > > > > > > > event > >> > > > > > > > > > > > >> use > >> > > > > > > > > > > > >> >> >>> cases > >> > > > > > > > > > > > >> >> >>> > are > >> > > > > > > > > > > > >> >> >>> > > distinct from scan/commit metrics. > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > 3. I'd prefer separating metrics > >> > > persistence > >> > > > > from > >> > > > > > > > > > MetaStore > >> > > > > > > > > > > > >> >> >>> persistence > >> > > > > > > > > > > > >> >> >>> > at > >> > > > > > > > > > > > >> >> >>> > > the code level, so that they could > be > >> > mixed > >> > > > and > >> > > > > > > > matched > >> > > > > > > > > > > > >> >> >>> independently. > >> > > > > > > > > > > > >> >> >>> > The > >> > > > > > > > > > > > >> >> >>> > > separate datasource question will > >> become > >> > a > >> > > > > > > non-issue > >> > > > > > > > > with > >> > > > > > > > > > > > that > >> > > > > > > > > > > > >> >> >>> approach, > >> > > > > > > > > > > > >> >> >>> > I > >> > > > > > > > > > > > >> >> >>> > > guess. > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > The rationale for separating scan > >> metrics > >> > > and > >> > > > > > > > metastore > >> > > > > > > > > > > > >> >> persistence > >> > > > > > > > > > > > >> >> >>> is > >> > > > > > > > > > > > >> >> >>> > that > >> > > > > > > > > > > > >> >> >>> > > "cascading deletes" between them are > >> > hardly > >> > > > > ever > >> > > > > > > > > > required. > >> > > > > > > > > > > > >> >> >>> Furthermore, > >> > > > > > > > > > > > >> >> >>> > the > >> > > > > > > > > > > > >> >> >>> > > data and query patterns are very > >> > different > >> > > so > >> > > > > > > > different > >> > > > > > > > > > > > >> >> technologies > >> > > > > > > > > > > > >> >> >>> > might > >> > > > > > > > > > > > >> >> >>> > > be beneficial in each case. > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > [1] > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://urldefense.com/v3/__https://lists.apache.org/thread/yto2wp982t43h1mqjwnslswhws5z47cy__;!!Iz9xO38YGHZK!5EuyFFkk3vhRWVIRvQAWBSQfpJkTMA9HxugzDwXmN0LPPqhEFxYkFRGVhtb8AqUwXtDh2OplcMnbMDHKxYDakNU$ > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > Cheers, > >> > > > > > > > > > > > >> >> >>> > > Dmitri. > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > On Fri, Feb 27, 2026 at 6:19 PM > Anand > >> > Kumar > >> > > > > > > Sankaran > >> > > > > > > > > via > >> > > > > > > > > > > dev > >> > > > > > > > > > > > < > >> > > > > > > > > > > > >> >> >>> > > [email protected]> wrote: > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > > Thanks all. This PR is merged now. > >> > > > > > > > > > > > >> >> >>> > > > > >> > > > > > > > > > > > >> >> >>> > > > Here are the follow-up features / > >> work > >> > > > > needed. > >> > > > > > > > These > >> > > > > > > > > > > were > >> > > > > > > > > > > > >> all > >> > > > > > > > > > > > >> >> >>> part of > >> > > > > > > > > > > > >> >> >>> > > the > >> > > > > > > > > > > > >> >> >>> > > > merged PR at some point in time > and > >> > were > >> > > > > > removed > >> > > > > > > to > >> > > > > > > > > > > reduce > >> > > > > > > > > > > > >> >> scope. > >> > > > > > > > > > > > >> >> >>> > > > > >> > > > > > > > > > > > >> >> >>> > > > Please let me know what you think. > >> > > > > > > > > > > > >> >> >>> > > > > >> > > > > > > > > > > > >> >> >>> > > > > >> > > > > > > > > > > > >> >> >>> > > > 1. A REST API to paginate > through > >> > > table > >> > > > > > > metrics. > >> > > > > > > > > > This > >> > > > > > > > > > > > >> will be > >> > > > > > > > > > > > >> >> >>> > non-IRC > >> > > > > > > > > > > > >> >> >>> > > > standard addition. > >> > > > > > > > > > > > >> >> >>> > > > 2. Utilities for managing old > >> > records, > >> > > > > > should > >> > > > > > > be > >> > > > > > > > > > > common > >> > > > > > > > > > > > >> with > >> > > > > > > > > > > > >> >> >>> events. > >> > > > > > > > > > > > >> >> >>> > > > There was some discussion that it > >> > belongs > >> > > > to > >> > > > > > the > >> > > > > > > > CLI. > >> > > > > > > > > > > > >> >> >>> > > > 3. Separate datasource > (metrics, > >> > > events, > >> > > > > > even > >> > > > > > > > > other > >> > > > > > > > > > > > >> tables?). > >> > > > > > > > > > > > >> >> >>> > > > > >> > > > > > > > > > > > >> >> >>> > > > > >> > > > > > > > > > > > >> >> >>> > > > Anything else? > >> > > > > > > > > > > > >> >> >>> > > > > >> > > > > > > > > > > > >> >> >>> > > > - > >> > > > > > > > > > > > >> >> >>> > > > Anand > >> > > > > > > > > > > > >> >> >>> > > > > >> > > > > > > > > > > > >> >> >>> > > > > >> > > > > > > > > > > > >> >> >>> > > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > > >> > > > > > > > > > > > >> >> >>> > >> > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
