> Does this unblock landing #5395? I don't think so. I'd recommend to read through the orignal thread, https://lists.apache.org/thread/1nfwnk1j8sppmsry675ygr1jbxyhvrdx. It's a small thread and wont' take too much time to read it through.
Yufei On Thu, Nov 5, 2026 at 4:02 AM vignesh a <[email protected]> wrote: > > Hi all, > > Anand raised this on the Iceberg list [1]. > > Eduard (RESTMetricsReporter) said reporting is best-effort, clients > are not expected to retry, and a catalog may return 204 or 5xx when it > can't persist a report. He doesn't want the spec to be prescriptive. > > Dan said returning 204 when the request failed is wrong, and that > retry-vs-ignore is a client choice. He also doesn't want "best-effort" > written into the spec. > > Yufei asked for explicit spec wording. Eduard and Dan both said no > spec change, for different reasons. > > #5395 [2] stays drop + polaris.metrics.reports.dropped. That's the > catalog choosing 204 after a failed persist, with the drop visible to > operators. Iceberg Java still ignores the status either way. > > Does this unblock landing #5395? > > Thanks, > Vignesh > > [1] https://lists.apache.org/thread.html/1nfwnk1j8sppmsry675ygr1jbxyhvrdx > [2] https://github.com/apache/polaris/pull/5395 > > On Tue, 3 Nov 2026 at 06:22, Anand Kumar Sankaran via dev > <[email protected]> wrote: >> >> Started >> https://lists.apache.org/thread.html/1nfwnk1j8sppmsry675ygr1jbxyhvrdx. >> >> On Mon, Nov 2, 2026 at 3:36 PM Anand Kumar Sankaran < >> [email protected]> wrote: >> >> > Hi Yufei, >> > >> > Fair point on pre-Polaris failures. Server-side retry only covers what >> > happens after we accept the report. Network and proxy failures would need >> > client-side retry, which only works if clients treat the response as >> > meaningful. Today they don't (the Java client ignores it), so I don't see >> > #5395 making that worse. It changes what we do after receipt, not what >> > clients can rely on. >> > >> > I agree the spec question is worth asking. I'll draft a short post for the >> > Iceberg dev list this week: Is POST .../metrics best-effort or >> > reliable-delivery, and what should a catalog return when it can't persist a >> > report? I'll link it here. >> > >> > I'd rather not hold #5395 for that. Failed writes are logged and counted, >> > so operators can see them. If Iceberg settles on stricter semantics we can >> > adjust. Would you be OK with that, or do you want the PR to wait? >> > >> > Thanks, >> > Anand >> > >> > On Mon, Nov 2, 2026 at 2:37 PM Yufei Gu <[email protected]> wrote: >> > >> >> Hi Anand, thanks for chiming in! > If we need guaranteed capture, that >> >> should be handled server side: retry or buffer the write after we've >> >> accepted the report. That makes sense for failures after Polaris receives >> >> the report. But what about >> >> >> >> >> >> Hi Anand, thanks for chiming in! >> >> >> >> > If we need guaranteed capture, that should be handled server side: >> >> > retry or buffer the write after we've accepted the report. >> >> >> >> That makes sense for failures after Polaris receives the report. But >> >> what about failures before the request reaches Polaris, such as >> >> network or proxy failures? Wouldn't we still need client side retries >> >> if reliable delivery is required? >> >> >> >> > If you still want the endpoint semantics clarified, I think raising it >> >> > on the Iceberg dev list is reasonable. >> >> >> >> Can we do that now? Since we're changing the observable REST behavior, >> >> I'd prefer to clarify the intended semantics with the Iceberg >> >> community first. >> >> >> >> On Mon, Nov 2, 2026 at 2:26 PM Anand Kumar Sankaran via dev >> >> <[email protected]> wrote: >> >> > >> >> > Hi Dmitri, Yufei, >> >> > >> >> > Thanks for pulling me in. Yes, I did raise billing/auditing as a use >> >> > case >> >> > for the metrics data. >> >> > >> >> > Having thought about it, I agree with Dmitri that client-side retries >> >> > are >> >> > the wrong place to get durability. The Java client ignores errors from >> >> > this >> >> > endpoint, and reporting is optional, so a 500 doesn't give us a reliable >> >> > audit trail either. If we need guaranteed capture, that should be >> >> > handled >> >> > server-side: retry or buffer the write after we've accepted the report. >> >> > I'm >> >> > happy to take that on as a follow-up. >> >> > >> >> > For #5395, I'm fine with drop-plus-counter. Failed writes are logged and >> >> > counted in polaris.metrics.reports.dropped, so operators can alert on >> >> > them >> >> > without the failure showing up as a catalog 5xx. I'd suggest we >> >> > document in >> >> > the metrics docs that the persisting reporter is best-effort today. >> >> > >> >> > Yufei, if you still want the endpoint semantics clarified, I think >> >> > raising >> >> > it on the Iceberg dev list is reasonable, but I wouldn't block this PR >> >> > on >> >> > it. Does the counter plus the best-effort documentation work for you? >> >> > >> >> > Thanks, >> >> > Anand >> >> > >> >> > On Mon, Nov 2, 2026 at 7:41 AM Dmitri Bourlatchkov via dev < >> >> > [email protected]> wrote: >> >> > >> >> > > Hi Yufei, Anand, For metrics, doing client-side retries is an >> >> > > overkill, >> >> > > IMHO. Metric reports cannot logically conflict with any other writes. >> >> > > If >> >> > > the server is not able to persist a piece of metrics data it already >> >> > > received, retries should happen >> >> > > >> >> > > >> >> > > Hi Yufei, Anand, >> >> > > >> >> > > For metrics, doing client-side retries is an overkill, IMHO. >> >> > > >> >> > > Metric reports cannot logically conflict with any other writes. If the >> >> > > server is not able to persist a piece of metrics data it already >> >> > > received, >> >> > > retries should happen server-side. >> >> > > >> >> > > Including Anand in this email directly, hoping for some background >> >> > > info >> >> > > sharing and preferred execution approach. >> >> > > >> >> > > Thanks, >> >> > > Dmitri. >> >> > > >> >> > > On Tue, Oct 27, 2026 at 2:32 PM Yufei Gu <[email protected]> wrote: >> >> > > >> >> > > > I think it's preferable, to return a 500 to clients. The [IRC >> >> > > > metrics >> >> > > > endpoint spec]( >> >> > > > >> >> > > > https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$ >> >> > <https://urldefense.com/v3/__https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc0lc0gIHg$%3E>> >> >> > > ) >> >> > > > doesn't say that metrics reporting is best effort. I recall Anand >> >> > > > mentioning a use case where metrics could be used for billing or >> >> > > > auditing. >> >> > > > In that case, client side retries don't seem unreasonable. >> >> > > > >> >> > > > I'd prefer to keep things simple and treat this as a normal REST >> >> > > > endpoint, >> >> > > > where the server clearly communicates when an operation fails. >> >> > > > >> >> > > > If we really want to change this behavior, I think we should first >> >> > > > discuss >> >> > > > it with the Iceberg community and clarify or update the metrics >> >> > > > endpoint >> >> > > > semantics in the spec. >> >> > > > >> >> > > > Yufei >> >> > > > >> >> > > > >> >> > > > On Mon, Oct 26, 2026 at 4:24 PM Dmitri Bourlatchkov >> >> > > > <[email protected]> >> >> > > > wrote: >> >> > > > >> >> > > > > Hi All, >> >> > > > > >> >> > > > > Great idea about counting dropped reports. >> >> > > > > >> >> > > > > Yufei: Does [5395] work from your POV, given that failures are >> >> > > > > reported >> >> > > > to >> >> > > > > Micrometer now? >> >> > > > > >> >> > > > > Thanks, >> >> > > > > Dmitri. >> >> > > > > >> >> > > > > [5395] >> >> > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ >> >> > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> >> >> > > > >> >> > > > > On Mon, Oct 26, 2026 at 6:53 AM vignesh a <[email protected]> >> >> > > > wrote: >> >> > > > > >> >> > > > > > Hi JB, Yufei, >> >> > > > > > >> >> > > > > > A dedicated drop counter (e.g. polaris.metrics.reports.dropped, >> >> > > > > > tagged by reporter) makes sense to me. Operators can alert on it >> >> > > > > > directly, and Polaris doesn't have to return a 500 on >> >> > > > > > POST .../metrics. >> >> > > > > > >> >> > > > > > I'll add it on #5395 [1]. >> >> > > > > > >> >> > > > > > Thanks, >> >> > > > > > Vignesh >> >> > > > > > >> >> > > > > > [1] >> >> > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ >> >> > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> >> >> > > > > >> >> > > > > > On Sun, 25 Oct 2026 at 15:02, Jean-Baptiste Onofré >> >> > > > > > <[email protected]> >> >> > > > > > wrote: >> >> > > > > > >> >> > > > > > > Hi all, >> >> > > > > > > >> >> > > > > > > Nice discussion! >> >> > > > > > > >> >> > > > > > > I lean towards the proposal in #5395. The reasons are the >> >> > > > > > > ones Dmitri >> >> > > > > > gave. >> >> > > > > > > The metrics POST is unidirectional and telemetry-like, so >> >> > > > > > > clients >> >> > > > gain >> >> > > > > > > nothing from seeing a persistence failure. The only status >> >> > > > > > > the IRC >> >> > > > spec >> >> > > > > > > allows for it is a 5xx, which is not really actionable for >> >> > > > > > > them :) >> >> > > > > > > >> >> > > > > > > I also agree with Yufei that a silent drop should not leave >> >> > > > > > > operators >> >> > > > > > > "blind". Today the 500 is the only server-side signal we got, >> >> > > > > > > and a >> >> > > > > WARN >> >> > > > > > > log is probably easy to miss (or ignore intentionally :)). >> >> > > > > > > Maybe we could add a dedicated counter for dropped reports, >> >> > > > > > > for >> >> > > > > instance >> >> > > > > > > polaris.metrics.reports.dropped, tagged by reporter type? >> >> > > > > > > Operators >> >> > > > > could >> >> > > > > > > then alert on it directly. That would cover the visibility >> >> > > > > > > use case >> >> > > > > > without >> >> > > > > > > surfacing as catalog 5xx. >> >> > > > > > > >> >> > > > > > > Yufei's point about excluding the metrics endpoint from the >> >> > > > > > > core >> >> > > > > > > availability SLO is also valid. Imho, that depends on how each >> >> > > > > deployment >> >> > > > > > > aggregates its metrics, so I would consider a complement more >> >> > > > > > > than a >> >> > > > > > > replacement for the server-side change. >> >> > > > > > > >> >> > > > > > > If the counter is acceptable, I'm +1 landing #5395 with it. >> >> > > > > > > >> >> > > > > > > Thanks! >> >> > > > > > > >> >> > > > > > > Regards >> >> > > > > > > JB >> >> > > > > > > >> >> > > > > > > On Thu, Oct 22, 2026 at 8:12 PM vignesh a >> >> > > > > > > <[email protected]> >> >> > > > > > wrote: >> >> > > > > > > >> >> > > > > > > > Hi all, >> >> > > > > > > > >> >> > > > > > > > PR #5395 [1] is about to land, and I wanted to flag it here >> >> > > > > > > > because >> >> > > > > it >> >> > > > > > > > changes REST behaviour compared to today. >> >> > > > > > > > >> >> > > > > > > > With this change, POST .../metrics no longer returns a 500 >> >> > > > > > > > when the >> >> > > > > > > > persisting reporter's DB write fails. The error is logged >> >> > > > > > > > and the >> >> > > > > > > > report is dropped. Errors from other reporters still >> >> > > > > > > > propagate >> >> > > > > through >> >> > > > > > > > the handler as before, and authorization is unchanged. >> >> > > > > > > > >> >> > > > > > > > Iceberg Java already ignores errors from the metrics >> >> > > > > > > > endpoint. This >> >> > > > > > > > change keeps a broken metrics store from showing up as >> >> > > > > > > > catalog 5xx >> >> > > > > > > > errors. >> >> > > > > > > > >> >> > > > > > > > Thoughts welcome, especially if anyone relies on the >> >> > > > > > > > current 500 >> >> > > > > > > > behaviour. >> >> > > > > > > > >> >> > > > > > > > Thanks, >> >> > > > > > > > Vignesh >> >> > > > > > > > >> >> > > > > > > > [1] >> >> > > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$ >> >> > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/5395__;!!Iz9xO38YGHZK!88C-Y3G5me0iEtnGWSN94r_L96KgNkKmMZTAo02hxOJJFKhHeGP9X6Jx1dGQ8vzQSWSJz_1mfiBMmc2SYH9SuA$%3E>> >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > > >> >> > >> >> > -- >> >> > [image: Workday Logo] <https://www.workday.com/ > >> >> > Anand Kumar Sankaran >> >> > Sr Principal, Software Engineer (Distributed Systems) >> >> > Office of the CTO >> >> > [email protected] >> >> >> >> >> > >> > -- >> > [image: Workday Logo] <https://www.workday.com/> >> > Anand Kumar Sankaran >> > Sr Principal, Software Engineer (Distributed Systems) >> > Office of the CTO >> > [email protected] >> > >> >> >> -- >> [image: Workday Logo] <https://www.workday.com/> >> Anand Kumar Sankaran >> Sr Principal, Software Engineer (Distributed Systems) >> Office of the CTO >> [email protected]
