Hi all, Anand raised this on the Iceberg list [1].
Eduard (RESTMetricsReporter) said reporting is best-effort, clients are not expected to retry, and a catalog may return 204 or 5xx when it can't persist a report. He doesn't want the spec to be prescriptive. Dan said returning 204 when the request failed is wrong, and that retry-vs-ignore is a client choice. He also doesn't want "best-effort" written into the spec. Yufei asked for explicit spec wording. Eduard and Dan both said no spec change, for different reasons. #5395 [2] stays drop + polaris.metrics.reports.dropped. That's the catalog choosing 204 after a failed persist, with the drop visible to operators. Iceberg Java still ignores the status either way. Does this unblock landing #5395? Thanks, Vignesh [1] https://lists.apache.org/thread.html/1nfwnk1j8sppmsry675ygr1jbxyhvrdx [2] https://github.com/apache/polaris/pull/5395 On Tue, 3 Nov 2026 at 06:22, Anand Kumar Sankaran via dev < [email protected]> wrote: > Started > https://lists.apache.org/thread.html/1nfwnk1j8sppmsry675ygr1jbxyhvrdx. > > On Mon, Nov 2, 2026 at 3:36 PM Anand Kumar Sankaran < > [email protected]> wrote: > > > Hi Yufei, > > > > Fair point on pre-Polaris failures. Server-side retry only covers what > > happens after we accept the report. Network and proxy failures would need > > client-side retry, which only works if clients treat the response as > > meaningful. Today they don't (the Java client ignores it), so I don't see > > #5395 making that worse. It changes what we do after receipt, not what > > clients can rely on. > > > > I agree the spec question is worth asking. I'll draft a short post for > the > > Iceberg dev list this week: Is POST .../metrics best-effort or > > reliable-delivery, and what should a catalog return when it can't > persist a > > report? I'll link it here. > > > > I'd rather not hold #5395 for that. Failed writes are logged and counted, > > so operators can see them. If Iceberg settles on stricter semantics we > can > > adjust. Would you be OK with that, or do you want the PR to wait? > > > > Thanks, > > Anand > > > > On Mon, Nov 2, 2026 at 2:37 PM Yufei Gu <[email protected]> wrote: > > > >> Hi Anand, thanks for chiming in! > If we need guaranteed capture, that > >> should be handled server side: retry or buffer the write after we've > >> accepted the report. That makes sense for failures after Polaris > receives > >> the report. But what about > >> > >> > >> Hi Anand, thanks for chiming in! > >> > >> > If we need guaranteed capture, that should be handled server side: > retry or buffer the write after we've accepted the report. > >> > >> That makes sense for failures after Polaris receives the report. But > >> what about failures before the request reaches Polaris, such as > >> network or proxy failures? Wouldn't we still need client side retries > >> if reliable delivery is required? > >> > >> > If you still want the endpoint semantics clarified, I think raising > it on the Iceberg dev list is reasonable. > >> > >> Can we do that now? Since we're changing the observable REST behavior, > >> I'd prefer to clarify the intended semantics with the Iceberg > >> community first. > >> > >> On Mon, Nov 2, 2026 at 2:26 PM Anand Kumar Sankaran via dev > >> <[email protected]> wrote: > >> > > >> > Hi Dmitri, Yufei, > >> > > >> > Thanks for pulling me in. Yes, I did raise billing/auditing as a use > case > >> > for the metrics data. > >> > > >> > Having thought about it, I agree with Dmitri that client-side retries > are > >> > the wrong place to get durability. The Java client ignores errors > from this > >> > endpoint, and reporting is optional, so a 500 doesn't give us a > reliable > >> > audit trail either. If we need guaranteed capture, that should be > handled > >> > server-side: retry or buffer the write after we've accepted the > report. I'm > >> > happy to take that on as a follow-up. > >> > > >> > For #5395, I'm fine with drop-plus-counter. Failed writes are logged > and > >> > counted in polaris.metrics.reports.dropped, so operators can alert on > them > >> > without the failure showing up as a catalog 5xx. I'd suggest we > document in > >> > the metrics docs that the persisting reporter is best-effort today. > >> > > >> > Yufei, if you still want the endpoint semantics clarified, I think > raising > >> > it on the Iceberg dev list is reasonable, but I wouldn't block this > PR on > >> > it. Does the counter plus the best-effort documentation work for you? > >> > > >> > Thanks, > >> > Anand > >> > > >> > On Mon, Nov 2, 2026 at 7:41 AM Dmitri Bourlatchkov via dev < > >> > [email protected]> wrote: > >> > > >> > > Hi Yufei, Anand, For metrics, doing client-side retries is an > overkill, > >> > > IMHO. Metric reports cannot logically conflict with any other > writes. If > >> > > the server is not able to persist a piece of metrics data it already > >> > > received, retries should happen > >> > > > >> > > > >> > > Hi Yufei, Anand, > >> > > > >> > > For metrics, doing client-side retries is an overkill, IMHO. > >> > > > >> > > Metric reports cannot logically conflict with any other writes. If > the > >> > > server is not able to persist a piece of metrics data it already > received, > >> > > retries should happen server-side. > >> > > > >> > > Including Anand in this email directly, hoping for some background > info > >> > > sharing and preferred execution approach. > >> > > > >> > > Thanks, > >> > > Dmitri. > >> > > > >> > > On Tue, Oct 27, 2026 at 2:32 PM Yufei Gu <[email protected]> > wrote: > >> > > > >> > > > I think it's preferable, to return a 500 to clients. The [IRC > metrics > >> > > > endpoint spec]( > >> > > > > >> > > > > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$ > >> > < > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$%3E>> > > ) > >> > > > doesn't say that metrics reporting is best effort. I recall Anand > >> > > > mentioning a use case where metrics could be used for billing or > auditing. > >> > > > In that case, client side retries don't seem unreasonable. > >> > > > > >> > > > I'd prefer to keep things simple and treat this as a normal REST > endpoint, > >> > > > where the server clearly communicates when an operation fails. > >> > > > > >> > > > If we really want to change this behavior, I think we should > first discuss > >> > > > it with the Iceberg community and clarify or update the metrics > endpoint > >> > > > semantics in the spec. > >> > > > > >> > > > Yufei > >> > > > > >> > > > > >> > > > On Mon, Oct 26, 2026 at 4:24 PM Dmitri Bourlatchkov < > [email protected]> > >> > > > wrote: > >> > > > > >> > > > > Hi All, > >> > > > > > >> > > > > Great idea about counting dropped reports. > >> > > > > > >> > > > > Yufei: Does [5395] work from your POV, given that failures are > reported > >> > > > to > >> > > > > Micrometer now? > >> > > > > > >> > > > > Thanks, > >> > > > > Dmitri. > >> > > > > > >> > > > > [5395] > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > >> > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> > > > > >> > > > > On Mon, Oct 26, 2026 at 6:53 AM vignesh a < > [email protected]> > >> > > > wrote: > >> > > > > > >> > > > > > Hi JB, Yufei, > >> > > > > > > >> > > > > > A dedicated drop counter (e.g. > polaris.metrics.reports.dropped, > >> > > > > > tagged by reporter) makes sense to me. Operators can alert on > it > >> > > > > > directly, and Polaris doesn't have to return a 500 on > >> > > > > > POST .../metrics. > >> > > > > > > >> > > > > > I'll add it on #5395 [1]. > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Vignesh > >> > > > > > > >> > > > > > [1] > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > >> > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> > > > > > >> > > > > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré < > [email protected]> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hi all, > >> > > > > > > > >> > > > > > > Nice discussion! > >> > > > > > > > >> > > > > > > I lean towards the proposal in #5395. The reasons are the > ones Dmitri > >> > > > > > gave. > >> > > > > > > The metrics POST is unidirectional and telemetry-like, so > clients > >> > > > gain > >> > > > > > > nothing from seeing a persistence failure. The only status > the IRC > >> > > > spec > >> > > > > > > allows for it is a 5xx, which is not really actionable for > them :) > >> > > > > > > > >> > > > > > > I also agree with Yufei that a silent drop should not leave > operators > >> > > > > > > "blind". Today the 500 is the only server-side signal we > got, and a > >> > > > > WARN > >> > > > > > > log is probably easy to miss (or ignore intentionally :)). > >> > > > > > > Maybe we could add a dedicated counter for dropped reports, > for > >> > > > > instance > >> > > > > > > polaris.metrics.reports.dropped, tagged by reporter type? > Operators > >> > > > > could > >> > > > > > > then alert on it directly. That would cover the visibility > use case > >> > > > > > without > >> > > > > > > surfacing as catalog 5xx. > >> > > > > > > > >> > > > > > > Yufei's point about excluding the metrics endpoint from the > core > >> > > > > > > availability SLO is also valid. Imho, that depends on how > each > >> > > > > deployment > >> > > > > > > aggregates its metrics, so I would consider a complement > more than a > >> > > > > > > replacement for the server-side change. > >> > > > > > > > >> > > > > > > If the counter is acceptable, I'm +1 landing #5395 with it. > >> > > > > > > > >> > > > > > > Thanks! > >> > > > > > > > >> > > > > > > Regards > >> > > > > > > JB > >> > > > > > > > >> > > > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a < > [email protected]> > >> > > > > > wrote: > >> > > > > > > > >> > > > > > > > Hi all, > >> > > > > > > > > >> > > > > > > > PR #5395 [1] is about to land, and I wanted to flag it > here because > >> > > > > it > >> > > > > > > > changes REST behaviour compared to today. > >> > > > > > > > > >> > > > > > > > With this change, POST .../metrics no longer returns a > 500 when the > >> > > > > > > > persisting reporter's DB write fails. The error is logged > and the > >> > > > > > > > report is dropped. Errors from other reporters still > propagate > >> > > > > through > >> > > > > > > > the handler as before, and authorization is unchanged. > >> > > > > > > > > >> > > > > > > > Iceberg Java already ignores errors from the metrics > endpoint. This > >> > > > > > > > change keeps a broken metrics store from showing up as > catalog 5xx > >> > > > > > > > errors. > >> > > > > > > > > >> > > > > > > > Thoughts welcome, especially if anyone relies on the > current 500 > >> > > > > > > > behaviour. > >> > > > > > > > > >> > > > > > > > Thanks, > >> > > > > > > > Vignesh > >> > > > > > > > > >> > > > > > > > [1] > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > >> > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > >> > -- > >> > [image: Workday Logo] <https://www.workday.com/ > > >> > Anand Kumar Sankaran > >> > Sr Principal, Software Engineer (Distributed Systems) > >> > Office of the CTO > >> > [email protected] > >> > >> > > > > -- > > [image: Workday Logo] <https://www.workday.com/> > > Anand Kumar Sankaran > > Sr Principal, Software Engineer (Distributed Systems) > > Office of the CTO > > [email protected] > > > > > -- > [image: Workday Logo] <https://www.workday.com/> > Anand Kumar Sankaran > Sr Principal, Software Engineer (Distributed Systems) > Office of the CTO > [email protected] >
