Hi Anand, thanks for chiming in! > If we need guaranteed capture, that should be handled server side: retry or > buffer the write after we've accepted the report.
That makes sense for failures after Polaris receives the report. But what about failures before the request reaches Polaris, such as network or proxy failures? Wouldn't we still need client side retries if reliable delivery is required? > If you still want the endpoint semantics clarified, I think raising it on the > Iceberg dev list is reasonable. Can we do that now? Since we're changing the observable REST behavior, I'd prefer to clarify the intended semantics with the Iceberg community first. On Mon, Nov 2, 2026 at 2:26 PM Anand Kumar Sankaran via dev <[email protected]> wrote: > > Hi Dmitri, Yufei, > > Thanks for pulling me in. Yes, I did raise billing/auditing as a use case > for the metrics data. > > Having thought about it, I agree with Dmitri that client-side retries are > the wrong place to get durability. The Java client ignores errors from this > endpoint, and reporting is optional, so a 500 doesn't give us a reliable > audit trail either. If we need guaranteed capture, that should be handled > server-side: retry or buffer the write after we've accepted the report. I'm > happy to take that on as a follow-up. > > For #5395, I'm fine with drop-plus-counter. Failed writes are logged and > counted in polaris.metrics.reports.dropped, so operators can alert on them > without the failure showing up as a catalog 5xx. I'd suggest we document in > the metrics docs that the persisting reporter is best-effort today. > > Yufei, if you still want the endpoint semantics clarified, I think raising > it on the Iceberg dev list is reasonable, but I wouldn't block this PR on > it. Does the counter plus the best-effort documentation work for you? > > Thanks, > Anand > > On Mon, Nov 2, 2026 at 7:41 AM Dmitri Bourlatchkov via dev < > [email protected]> wrote: > > > Hi Yufei, Anand, For metrics, doing client-side retries is an overkill, > > IMHO. Metric reports cannot logically conflict with any other writes. If > > the server is not able to persist a piece of metrics data it already > > received, retries should happen > > > > > > Hi Yufei, Anand, > > > > For metrics, doing client-side retries is an overkill, IMHO. > > > > Metric reports cannot logically conflict with any other writes. If the > > server is not able to persist a piece of metrics data it already received, > > retries should happen server-side. > > > > Including Anand in this email directly, hoping for some background info > > sharing and preferred execution approach. > > > > Thanks, > > Dmitri. > > > > On Tue, Oct 27, 2026 at 2:32 PM Yufei Gu <[email protected]> wrote: > > > > > I think it's preferable, to return a 500 to clients. The [IRC metrics > > > endpoint spec]( > > > > > > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$ > > > ) > > > doesn't say that metrics reporting is best effort. I recall Anand > > > mentioning a use case where metrics could be used for billing or auditing. > > > In that case, client side retries don't seem unreasonable. > > > > > > I'd prefer to keep things simple and treat this as a normal REST endpoint, > > > where the server clearly communicates when an operation fails. > > > > > > If we really want to change this behavior, I think we should first discuss > > > it with the Iceberg community and clarify or update the metrics endpoint > > > semantics in the spec. > > > > > > Yufei > > > > > > > > > On Mon, Oct 26, 2026 at 4:24 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > Great idea about counting dropped reports. > > > > > > > > Yufei: Does [5395] work from your POV, given that failures are reported > > > to > > > > Micrometer now? > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > [5395] > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > > > > > > > > On Mon, Oct 26, 2026 at 6:53 AM vignesh a <[email protected]> > > > wrote: > > > > > > > > > Hi JB, Yufei, > > > > > > > > > > A dedicated drop counter (e.g. polaris.metrics.reports.dropped, > > > > > tagged by reporter) makes sense to me. Operators can alert on it > > > > > directly, and Polaris doesn't have to return a 500 on > > > > > POST .../metrics. > > > > > > > > > > I'll add it on #5395 [1]. > > > > > > > > > > Thanks, > > > > > Vignesh > > > > > > > > > > [1] > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > > > > > > > > > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré <[email protected]> > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > Nice discussion! > > > > > > > > > > > > I lean towards the proposal in #5395. The reasons are the ones > > > > > > Dmitri > > > > > gave. > > > > > > The metrics POST is unidirectional and telemetry-like, so clients > > > gain > > > > > > nothing from seeing a persistence failure. The only status the IRC > > > spec > > > > > > allows for it is a 5xx, which is not really actionable for them :) > > > > > > > > > > > > I also agree with Yufei that a silent drop should not leave > > > > > > operators > > > > > > "blind". Today the 500 is the only server-side signal we got, and a > > > > WARN > > > > > > log is probably easy to miss (or ignore intentionally :)). > > > > > > Maybe we could add a dedicated counter for dropped reports, for > > > > instance > > > > > > polaris.metrics.reports.dropped, tagged by reporter type? Operators > > > > could > > > > > > then alert on it directly. That would cover the visibility use case > > > > > without > > > > > > surfacing as catalog 5xx. > > > > > > > > > > > > Yufei's point about excluding the metrics endpoint from the core > > > > > > availability SLO is also valid. Imho, that depends on how each > > > > deployment > > > > > > aggregates its metrics, so I would consider a complement more than a > > > > > > replacement for the server-side change. > > > > > > > > > > > > If the counter is acceptable, I'm +1 landing #5395 with it. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Regards > > > > > > JB > > > > > > > > > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a <[email protected]> > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > PR #5395 [1] is about to land, and I wanted to flag it here > > > > > > > because > > > > it > > > > > > > changes REST behaviour compared to today. > > > > > > > > > > > > > > With this change, POST .../metrics no longer returns a 500 when > > > > > > > the > > > > > > > persisting reporter's DB write fails. The error is logged and the > > > > > > > report is dropped. Errors from other reporters still propagate > > > > through > > > > > > > the handler as before, and authorization is unchanged. > > > > > > > > > > > > > > Iceberg Java already ignores errors from the metrics endpoint. > > > > > > > This > > > > > > > change keeps a broken metrics store from showing up as catalog 5xx > > > > > > > errors. > > > > > > > > > > > > > > Thoughts welcome, especially if anyone relies on the current 500 > > > > > > > behaviour. > > > > > > > > > > > > > > Thanks, > > > > > > > Vignesh > > > > > > > > > > > > > > [1] > > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > [image: Workday Logo] <https://www.workday.com/> > Anand Kumar Sankaran > Sr Principal, Software Engineer (Distributed Systems) > Office of the CTO > [email protected]
