nastra commented on code in PR #16250: URL: https://github.com/apache/iceberg/pull/16250#discussion_r3460140252
########## docs/docs/metrics-reporting.md: ########## @@ -120,6 +120,227 @@ This is the default when using the [`RESTCatalog`](https://github.com/apache/ice Sending metrics via REST can be controlled with the `rest-metrics-reporting-enabled` (defaults to `true`) property. +### [`OtelMetricsReporter`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/metrics/OtelMetricsReporter.java) + +Exports [`ScanReport`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/metrics/ScanReport.java) and [`CommitReport`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/metrics/CommitReport.java) through the [OpenTelemetry](https://opentelemetry.io/) API as `iceberg.scan.*` and `iceberg.commit.*` metrics. Any OTLP-compatible backend (Prometheus, CloudWatch, Datadog, Grafana Cloud, Honeycomb, etc.) can receive them through a host-owned OpenTelemetry SDK. + +#### Host responsibilities + +`OtelMetricsReporter` does not own the OpenTelemetry SDK. It calls `GlobalOpenTelemetry.get().getMeter("org.apache.iceberg")` in `initialize(...)` and reports through whatever SDK the host application has registered. If no SDK has been registered, OpenTelemetry returns the no-op implementation and metric calls are silently dropped — the standard OpenTelemetry API contract. + +The host application is therefore responsible for: + +1. Adding the OpenTelemetry **API**, **SDK**, and a **metric exporter** matching the target backend to the runtime classpath. Iceberg core compiles against `opentelemetry-api` only; it does not bundle the SDK or any exporter. +2. Building and registering an `OpenTelemetrySdk`, either via the [OpenTelemetry Java agent](https://opentelemetry.io/docs/zero-code/java/agent/) (which auto-instruments the SDK at JVM startup) or programmatically with `OpenTelemetrySdk.builder()...buildAndRegisterGlobal()`. +3. Configuring the exporter's endpoint, credentials, batching, retry, and resource attributes — typically via the [standard OpenTelemetry environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/) (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME`, `OTEL_EXPORTER_OTLP_HEADERS`, ...). + +Because the host owns the SDK, Iceberg has no reporter-specific catalog properties for endpoint, protocol, headers, intervals, or resource attributes. The catalog only needs to know the reporter class: + +``` +metrics-reporter-impl=org.apache.iceberg.metrics.OtelMetricsReporter +``` + +#### Attribute set + +By default the reporter attaches the following attributes to every emitted metric point: + +| Metrics | Default attributes | +|---|---| +| Scan metrics (`iceberg.scan.*`) | `iceberg.table.name` | +| Commit metrics (`iceberg.commit.*`) | `iceberg.table.name`, `iceberg.operation` | + +The attribute set is configurable via the `iceberg.otel.metrics.attributes` catalog property, which takes a comma-separated allowlist of attribute short names. Recognized names: + +- `table-name` — emits `iceberg.table.name` +- `schema-id` — emits `iceberg.schema.id` (opt-in; useful for correlating scan performance with schema evolution) +- `operation` — emits `iceberg.operation` + +Attributes whose short names are not listed are omitted from emitted metric points. To additionally include the schema id: + +``` +iceberg.otel.metrics.attributes=table-name,schema-id,operation +``` + +To omit `iceberg.table.name` entirely in deployments with a very large number of tables: + +``` +iceberg.otel.metrics.attributes=operation +``` + +To emit metrics with no attributes at all (single aggregate time series per metric): + +``` +iceberg.otel.metrics.attributes= Review Comment: we might need to double-check whether it's easy/possible to pass an empty string via Spark. Alternatively, maybe have a separate `none` option -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
