Hi Yufei, I misread Eduard.
His point was that a catalog may return 204 or 5xx, and he doesn't want the spec to prescribe which. I treated that, plus "no spec change," as unblocking a Polaris 204 on persist failure. That was wrong. Dan said returning a success code when there's a failure is wrong. Your last Iceberg mail said 204 when the operation failed is misleading. Spec unchanged does not unblock Polaris 204. I shouldn't have framed the recap as unblocking the landing. Thanks, *Vignesh* On Thu, 5 Nov 2026 at 23:39, Yufei Gu <[email protected]> wrote: > > Does this unblock landing #5395? > > I don't think so. I'd recommend to read through the orignal thread, > https://lists.apache.org/thread/1nfwnk1j8sppmsry675ygr1jbxyhvrdx. It's > a small thread and wont' take too much time to read it through. > > Yufei > > On Thu, Nov 5, 2026 at 4:02 AM vignesh a <[email protected]> wrote: > > > > Hi all, > > > > Anand raised this on the Iceberg list [1]. > > > > Eduard (RESTMetricsReporter) said reporting is best-effort, clients > > are not expected to retry, and a catalog may return 204 or 5xx when it > > can't persist a report. He doesn't want the spec to be prescriptive. > > > > Dan said returning 204 when the request failed is wrong, and that > > retry-vs-ignore is a client choice. He also doesn't want "best-effort" > > written into the spec. > > > > Yufei asked for explicit spec wording. Eduard and Dan both said no > > spec change, for different reasons. > > > > #5395 [2] stays drop + polaris.metrics.reports.dropped. That's the > > catalog choosing 204 after a failed persist, with the drop visible to > > operators. Iceberg Java still ignores the status either way. > > > > Does this unblock landing #5395? > > > > Thanks, > > Vignesh > > > > [1] > https://lists.apache.org/thread.html/1nfwnk1j8sppmsry675ygr1jbxyhvrdx > > [2] https://github.com/apache/polaris/pull/5395 > > > > On Tue, 3 Nov 2026 at 06:22, Anand Kumar Sankaran via dev < > [email protected]> wrote: > >> > >> Started > >> https://lists.apache.org/thread.html/1nfwnk1j8sppmsry675ygr1jbxyhvrdx. > >> > >> On Mon, Nov 2, 2026 at 3:36 PM Anand Kumar Sankaran < > >> [email protected]> wrote: > >> > >> > Hi Yufei, > >> > > >> > Fair point on pre-Polaris failures. Server-side retry only covers what > >> > happens after we accept the report. Network and proxy failures would > need > >> > client-side retry, which only works if clients treat the response as > >> > meaningful. Today they don't (the Java client ignores it), so I don't > see > >> > #5395 making that worse. It changes what we do after receipt, not what > >> > clients can rely on. > >> > > >> > I agree the spec question is worth asking. I'll draft a short post > for the > >> > Iceberg dev list this week: Is POST .../metrics best-effort or > >> > reliable-delivery, and what should a catalog return when it can't > persist a > >> > report? I'll link it here. > >> > > >> > I'd rather not hold #5395 for that. Failed writes are logged and > counted, > >> > so operators can see them. If Iceberg settles on stricter semantics > we can > >> > adjust. Would you be OK with that, or do you want the PR to wait? > >> > > >> > Thanks, > >> > Anand > >> > > >> > On Mon, Nov 2, 2026 at 2:37 PM Yufei Gu <[email protected]> wrote: > >> > > >> >> Hi Anand, thanks for chiming in! > If we need guaranteed capture, > that > >> >> should be handled server side: retry or buffer the write after we've > >> >> accepted the report. That makes sense for failures after Polaris > receives > >> >> the report. But what about > >> >> > >> >> > >> >> Hi Anand, thanks for chiming in! > >> >> > >> >> > If we need guaranteed capture, that should be handled server side: > retry or buffer the write after we've accepted the report. > >> >> > >> >> That makes sense for failures after Polaris receives the report. But > >> >> what about failures before the request reaches Polaris, such as > >> >> network or proxy failures? Wouldn't we still need client side retries > >> >> if reliable delivery is required? > >> >> > >> >> > If you still want the endpoint semantics clarified, I think > raising it on the Iceberg dev list is reasonable. > >> >> > >> >> Can we do that now? Since we're changing the observable REST > behavior, > >> >> I'd prefer to clarify the intended semantics with the Iceberg > >> >> community first. > >> >> > >> >> On Mon, Nov 2, 2026 at 2:26 PM Anand Kumar Sankaran via dev > >> >> <[email protected]> wrote: > >> >> > > >> >> > Hi Dmitri, Yufei, > >> >> > > >> >> > Thanks for pulling me in. Yes, I did raise billing/auditing as a > use case > >> >> > for the metrics data. > >> >> > > >> >> > Having thought about it, I agree with Dmitri that client-side > retries are > >> >> > the wrong place to get durability. The Java client ignores errors > from this > >> >> > endpoint, and reporting is optional, so a 500 doesn't give us a > reliable > >> >> > audit trail either. If we need guaranteed capture, that should be > handled > >> >> > server-side: retry or buffer the write after we've accepted the > report. I'm > >> >> > happy to take that on as a follow-up. > >> >> > > >> >> > For #5395, I'm fine with drop-plus-counter. Failed writes are > logged and > >> >> > counted in polaris.metrics.reports.dropped, so operators can alert > on them > >> >> > without the failure showing up as a catalog 5xx. I'd suggest we > document in > >> >> > the metrics docs that the persisting reporter is best-effort today. > >> >> > > >> >> > Yufei, if you still want the endpoint semantics clarified, I think > raising > >> >> > it on the Iceberg dev list is reasonable, but I wouldn't block > this PR on > >> >> > it. Does the counter plus the best-effort documentation work for > you? > >> >> > > >> >> > Thanks, > >> >> > Anand > >> >> > > >> >> > On Mon, Nov 2, 2026 at 7:41 AM Dmitri Bourlatchkov via dev < > >> >> > [email protected]> wrote: > >> >> > > >> >> > > Hi Yufei, Anand, For metrics, doing client-side retries is an > overkill, > >> >> > > IMHO. Metric reports cannot logically conflict with any other > writes. If > >> >> > > the server is not able to persist a piece of metrics data it > already > >> >> > > received, retries should happen > >> >> > > > >> >> > > > >> >> > > Hi Yufei, Anand, > >> >> > > > >> >> > > For metrics, doing client-side retries is an overkill, IMHO. > >> >> > > > >> >> > > Metric reports cannot logically conflict with any other writes. > If the > >> >> > > server is not able to persist a piece of metrics data it already > received, > >> >> > > retries should happen server-side. > >> >> > > > >> >> > > Including Anand in this email directly, hoping for some > background info > >> >> > > sharing and preferred execution approach. > >> >> > > > >> >> > > Thanks, > >> >> > > Dmitri. > >> >> > > > >> >> > > On Tue, Oct 27, 2026 at 2:32 PM Yufei Gu <[email protected]> > wrote: > >> >> > > > >> >> > > > I think it's preferable, to return a 500 to clients. The [IRC > metrics > >> >> > > > endpoint spec]( > >> >> > > > > >> >> > > > > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$ > >> >> > < > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$%3E>> > > ) > >> >> > > > doesn't say that metrics reporting is best effort. I recall > Anand > >> >> > > > mentioning a use case where metrics could be used for billing > or auditing. > >> >> > > > In that case, client side retries don't seem unreasonable. > >> >> > > > > >> >> > > > I'd prefer to keep things simple and treat this as a normal > REST endpoint, > >> >> > > > where the server clearly communicates when an operation fails. > >> >> > > > > >> >> > > > If we really want to change this behavior, I think we should > first discuss > >> >> > > > it with the Iceberg community and clarify or update the > metrics endpoint > >> >> > > > semantics in the spec. > >> >> > > > > >> >> > > > Yufei > >> >> > > > > >> >> > > > > >> >> > > > On Mon, Oct 26, 2026 at 4:24 PM Dmitri Bourlatchkov < > [email protected]> > >> >> > > > wrote: > >> >> > > > > >> >> > > > > Hi All, > >> >> > > > > > >> >> > > > > Great idea about counting dropped reports. > >> >> > > > > > >> >> > > > > Yufei: Does [5395] work from your POV, given that failures > are reported > >> >> > > > to > >> >> > > > > Micrometer now? > >> >> > > > > > >> >> > > > > Thanks, > >> >> > > > > Dmitri. > >> >> > > > > > >> >> > > > > [5395] > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > >> >> > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> > > > > >> >> > > > > On Mon, Oct 26, 2026 at 6:53 AM vignesh a < > [email protected]> > >> >> > > > wrote: > >> >> > > > > > >> >> > > > > > Hi JB, Yufei, > >> >> > > > > > > >> >> > > > > > A dedicated drop counter (e.g. > polaris.metrics.reports.dropped, > >> >> > > > > > tagged by reporter) makes sense to me. Operators can alert > on it > >> >> > > > > > directly, and Polaris doesn't have to return a 500 on > >> >> > > > > > POST .../metrics. > >> >> > > > > > > >> >> > > > > > I'll add it on #5395 [1]. > >> >> > > > > > > >> >> > > > > > Thanks, > >> >> > > > > > Vignesh > >> >> > > > > > > >> >> > > > > > [1] > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > >> >> > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> > > > > > >> >> > > > > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré < > [email protected]> > >> >> > > > > > wrote: > >> >> > > > > > > >> >> > > > > > > Hi all, > >> >> > > > > > > > >> >> > > > > > > Nice discussion! > >> >> > > > > > > > >> >> > > > > > > I lean towards the proposal in #5395. The reasons are > the ones Dmitri > >> >> > > > > > gave. > >> >> > > > > > > The metrics POST is unidirectional and telemetry-like, > so clients > >> >> > > > gain > >> >> > > > > > > nothing from seeing a persistence failure. The only > status the IRC > >> >> > > > spec > >> >> > > > > > > allows for it is a 5xx, which is not really actionable > for them :) > >> >> > > > > > > > >> >> > > > > > > I also agree with Yufei that a silent drop should not > leave operators > >> >> > > > > > > "blind". Today the 500 is the only server-side signal we > got, and a > >> >> > > > > WARN > >> >> > > > > > > log is probably easy to miss (or ignore intentionally > :)). > >> >> > > > > > > Maybe we could add a dedicated counter for dropped > reports, for > >> >> > > > > instance > >> >> > > > > > > polaris.metrics.reports.dropped, tagged by reporter > type? Operators > >> >> > > > > could > >> >> > > > > > > then alert on it directly. That would cover the > visibility use case > >> >> > > > > > without > >> >> > > > > > > surfacing as catalog 5xx. > >> >> > > > > > > > >> >> > > > > > > Yufei's point about excluding the metrics endpoint from > the core > >> >> > > > > > > availability SLO is also valid. Imho, that depends on > how each > >> >> > > > > deployment > >> >> > > > > > > aggregates its metrics, so I would consider a complement > more than a > >> >> > > > > > > replacement for the server-side change. > >> >> > > > > > > > >> >> > > > > > > If the counter is acceptable, I'm +1 landing #5395 with > it. > >> >> > > > > > > > >> >> > > > > > > Thanks! > >> >> > > > > > > > >> >> > > > > > > Regards > >> >> > > > > > > JB > >> >> > > > > > > > >> >> > > > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a < > [email protected]> > >> >> > > > > > wrote: > >> >> > > > > > > > >> >> > > > > > > > Hi all, > >> >> > > > > > > > > >> >> > > > > > > > PR #5395 [1] is about to land, and I wanted to flag it > here because > >> >> > > > > it > >> >> > > > > > > > changes REST behaviour compared to today. > >> >> > > > > > > > > >> >> > > > > > > > With this change, POST .../metrics no longer returns a > 500 when the > >> >> > > > > > > > persisting reporter's DB write fails. The error is > logged and the > >> >> > > > > > > > report is dropped. Errors from other reporters still > propagate > >> >> > > > > through > >> >> > > > > > > > the handler as before, and authorization is unchanged. > >> >> > > > > > > > > >> >> > > > > > > > Iceberg Java already ignores errors from the metrics > endpoint. This > >> >> > > > > > > > change keeps a broken metrics store from showing up as > catalog 5xx > >> >> > > > > > > > errors. > >> >> > > > > > > > > >> >> > > > > > > > Thoughts welcome, especially if anyone relies on the > current 500 > >> >> > > > > > > > behaviour. > >> >> > > > > > > > > >> >> > > > > > > > Thanks, > >> >> > > > > > > > Vignesh > >> >> > > > > > > > > >> >> > > > > > > > [1] > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ > >> >> > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > > >> >> > > >> >> > -- > >> >> > [image: Workday Logo] <https://www.workday.com/ > > >> >> > Anand Kumar Sankaran > >> >> > Sr Principal, Software Engineer (Distributed Systems) > >> >> > Office of the CTO > >> >> > [email protected] > >> >> > >> >> > >> > > >> > -- > >> > [image: Workday Logo] <https://www.workday.com/> > >> > Anand Kumar Sankaran > >> > Sr Principal, Software Engineer (Distributed Systems) > >> > Office of the CTO > >> > [email protected] > >> > > >> > >> > >> -- > >> [image: Workday Logo] <https://www.workday.com/> > >> Anand Kumar Sankaran > >> Sr Principal, Software Engineer (Distributed Systems) > >> Office of the CTO > >> [email protected] >
