Hi All, Great idea about counting dropped reports.
Yufei: Does [5395] work from your POV, given that failures are reported to Micrometer now? Thanks, Dmitri. [5395] https://github.com/apache/polaris/pull/5395 On Mon, Oct 26, 2026 at 6:53 AM vignesh a <[email protected]> wrote: > Hi JB, Yufei, > > A dedicated drop counter (e.g. polaris.metrics.reports.dropped, > tagged by reporter) makes sense to me. Operators can alert on it > directly, and Polaris doesn't have to return a 500 on > POST .../metrics. > > I'll add it on #5395 [1]. > > Thanks, > Vignesh > > [1] https://github.com/apache/polaris/pull/5395 > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré <[email protected]> > wrote: > > > Hi all, > > > > Nice discussion! > > > > I lean towards the proposal in #5395. The reasons are the ones Dmitri > gave. > > The metrics POST is unidirectional and telemetry-like, so clients gain > > nothing from seeing a persistence failure. The only status the IRC spec > > allows for it is a 5xx, which is not really actionable for them :) > > > > I also agree with Yufei that a silent drop should not leave operators > > "blind". Today the 500 is the only server-side signal we got, and a WARN > > log is probably easy to miss (or ignore intentionally :)). > > Maybe we could add a dedicated counter for dropped reports, for instance > > polaris.metrics.reports.dropped, tagged by reporter type? Operators could > > then alert on it directly. That would cover the visibility use case > without > > surfacing as catalog 5xx. > > > > Yufei's point about excluding the metrics endpoint from the core > > availability SLO is also valid. Imho, that depends on how each deployment > > aggregates its metrics, so I would consider a complement more than a > > replacement for the server-side change. > > > > If the counter is acceptable, I'm +1 landing #5395 with it. > > > > Thanks! > > > > Regards > > JB > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a <[email protected]> > wrote: > > > > > Hi all, > > > > > > PR #5395 [1] is about to land, and I wanted to flag it here because it > > > changes REST behaviour compared to today. > > > > > > With this change, POST .../metrics no longer returns a 500 when the > > > persisting reporter's DB write fails. The error is logged and the > > > report is dropped. Errors from other reporters still propagate through > > > the handler as before, and authorization is unchanged. > > > > > > Iceberg Java already ignores errors from the metrics endpoint. This > > > change keeps a broken metrics store from showing up as catalog 5xx > > > errors. > > > > > > Thoughts welcome, especially if anyone relies on the current 500 > > > behaviour. > > > > > > Thanks, > > > Vignesh > > > > > > [1] https://github.com/apache/polaris/pull/5395 > > > > > >
