Hi all,
I'm a contributor to Apache Polaris. We have a question about the REST
catalog's metrics endpoint (POST
/v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics, reportMetrics)
that we'd like to settle with the Iceberg community before changing server
behavior.
*Background*
Polaris can persist metrics reports sent to this endpoint. Today, if the
database write fails, the endpoint returns a 500. A pending Polaris PR [1]
would log the failure, increment a drop counter, and return 204 instead.
The motivation is that a degraded metrics store shouldn't show up as a
catalog failure. The discussion on the Polaris dev list [2] has split on
this.
*What the spec says*
The spec defines the operation as "Send a metrics report to this endpoint
to be processed by the backend". It documents 204, 400, 401, 403, 404, 419,
503 and 5XX. It doesn't say whether reporting is best-effort or whether a
catalog must signal a failure to process a report. It also doesn't say what
a client should do on a 5xx.
*What clients do*
As far as I know, the Java client's REST metrics reporter catches and logs
errors from this endpoint and does not retry. I haven't checked the other
implementations (PyIceberg, iceberg-rust, iceberg-go), so please correct me.
*The two readings*
*1. Best-effort telemetry.* The client sends the report and moves on. A
catalog may drop a report it can't persist, as long as it surfaces that to
operators. A 5xx tells the client nothing it can act on.
*2. A normal REST write.* If the catalog can't process the report, it says
so with a 5xx, and clients may retry. This matters for anyone using the
reports for auditing or billing, though it only helps if clients actually
retry.
*Questions*
1. Is reportMetrics intended to be best-effort? Is a catalog allowed to
accept a report and discard it, or fail to process it, without signaling
failure?
2. If a catalog can't process a report, what should it return? Is 5xx the
expected answer, or is 204 acceptable?
3. Should clients be expected to retry a failed report? If so, the spec
should say that. If not, it should say reporting is not guaranteed delivery.
4. Should the spec state any of this in the endpoint description, so
implementations behave consistently?
We'd like to follow whatever the community decides rather than invent
Polaris-specific behavior. I'm happy to send a spec PR with clarifying
wording once there's a direction.
Thanks,
Anand
[1] https://github.com/apache/polaris/pull/5395
[2] https://lists.apache.org/thread/0r7fpq153d2ny0tbt162051dh0jv3wjb
--
[image: Workday Logo] <https://www.workday.com/>
Anand Kumar Sankaran
Sr Principal, Software Engineer (Distributed Systems)
Office of the CTO
[email protected]
