I think it's preferable, to return a 500 to clients. The [IRC metrics endpoint spec]( https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml) doesn't say that metrics reporting is best effort. I recall Anand mentioning a use case where metrics could be used for billing or auditing. In that case, client side retries don't seem unreasonable.
I'd prefer to keep things simple and treat this as a normal REST endpoint, where the server clearly communicates when an operation fails. If we really want to change this behavior, I think we should first discuss it with the Iceberg community and clarify or update the metrics endpoint semantics in the spec. Yufei On Mon, Oct 26, 2026 at 4:24 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > Great idea about counting dropped reports. > > Yufei: Does [5395] work from your POV, given that failures are reported to > Micrometer now? > > Thanks, > Dmitri. > > [5395] https://github.com/apache/polaris/pull/5395 > > On Mon, Oct 26, 2026 at 6:53 AM vignesh a <[email protected]> wrote: > > > Hi JB, Yufei, > > > > A dedicated drop counter (e.g. polaris.metrics.reports.dropped, > > tagged by reporter) makes sense to me. Operators can alert on it > > directly, and Polaris doesn't have to return a 500 on > > POST .../metrics. > > > > I'll add it on #5395 [1]. > > > > Thanks, > > Vignesh > > > > [1] https://github.com/apache/polaris/pull/5395 > > > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi all, > > > > > > Nice discussion! > > > > > > I lean towards the proposal in #5395. The reasons are the ones Dmitri > > gave. > > > The metrics POST is unidirectional and telemetry-like, so clients gain > > > nothing from seeing a persistence failure. The only status the IRC spec > > > allows for it is a 5xx, which is not really actionable for them :) > > > > > > I also agree with Yufei that a silent drop should not leave operators > > > "blind". Today the 500 is the only server-side signal we got, and a > WARN > > > log is probably easy to miss (or ignore intentionally :)). > > > Maybe we could add a dedicated counter for dropped reports, for > instance > > > polaris.metrics.reports.dropped, tagged by reporter type? Operators > could > > > then alert on it directly. That would cover the visibility use case > > without > > > surfacing as catalog 5xx. > > > > > > Yufei's point about excluding the metrics endpoint from the core > > > availability SLO is also valid. Imho, that depends on how each > deployment > > > aggregates its metrics, so I would consider a complement more than a > > > replacement for the server-side change. > > > > > > If the counter is acceptable, I'm +1 landing #5395 with it. > > > > > > Thanks! > > > > > > Regards > > > JB > > > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a <[email protected]> > > wrote: > > > > > > > Hi all, > > > > > > > > PR #5395 [1] is about to land, and I wanted to flag it here because > it > > > > changes REST behaviour compared to today. > > > > > > > > With this change, POST .../metrics no longer returns a 500 when the > > > > persisting reporter's DB write fails. The error is logged and the > > > > report is dropped. Errors from other reporters still propagate > through > > > > the handler as before, and authorization is unchanged. > > > > > > > > Iceberg Java already ignores errors from the metrics endpoint. This > > > > change keeps a broken metrics store from showing up as catalog 5xx > > > > errors. > > > > > > > > Thoughts welcome, especially if anyone relies on the current 500 > > > > behaviour. > > > > > > > > Thanks, > > > > Vignesh > > > > > > > > [1] https://github.com/apache/polaris/pull/5395 > > > > > > > > > >
