I also support this PR. Iceberg has good metrics and I agree @Yufei Gu <[email protected]> catalog is not right place for that.
On Fri, May 22, 2026 at 2:48 PM Ryan Blue <[email protected]> wrote: > I think it is fine to add `opentelemetry-api` as `compileOnly`. We now > have checks in place that should prevent future dependency leaks into our > runtime Jars. > > My only concern is making sure that this is usable. I'm not familiar with > how OpenTelemetry works, but if this is expected to be bundled into > applications that use it then we generally want to meet those assumptions. > For libraries with multiple back-ends (like logging, for example) we would > normally include the API so that we can load classes that use it, but not > the specific back-end (log4j, logback, etc.). And when that API is very > commonly distributed (like slf4j) we usually don't include it. We just want > to avoid surprise failures for end users by meeting their expectations. (As > long as it doesn't require bundling the whole world) > > Ryan > > On Thu, May 21, 2026 at 10:22 AM Yufei Gu <[email protected]> wrote: > >> Hi Noritaka, >> >> Thanks for writing this up. I think this is a good direction overall. >> Using OpenTelemetry as a vendor neutral integration layer makes sense here. >> It gives Iceberg clients a standard way to integrate with modern >> observability platforms without adding per backend integrations into the >> project. Moreover, even for REST catalogs, the catalog itself is usually >> not the right place to serve as a full metrics monitoring platform. In >> practice, metrics still need to flow into systems like Prometheus, >> CloudWatch, Datadog, or Grafana Cloud for storage, dashboards, alerting, >> and correlation. The main extra value a REST catalog could provide is >> enrichment with catalog specific metadata, but that feels like a relatively >> minor benefit compared to having a standard observability integration path >> overall. >> >> Yufei >> >> >> On Wed, May 20, 2026 at 6:39 PM Noritaka Sekiyama via dev < >> [email protected]> wrote: >> >>> Hi all, >>> >>> I'd like to propose adding an OpenTelemetry-based MetricsReporter to >>> iceberg-core that exports ScanReport and CommitReport to any OTLP-compatible >>> backend. >>> >>> # Background >>> Iceberg ships three built-in MetricsReporter implementations today: >>> LoggingMetricsReporter, InMemoryMetricsReporter (Spark-internal), and >>> RESTMetricsReporter (REST catalog only). >>> None of them give users an out-of-the-box way to ship scan/commit >>> metrics to an external observability platform. >>> The gap applies to Spark users on non-REST catalogs and to all non-Spark >>> engines (Trino, Flink, etc.). >>> >>> # Motivation >>> OpenTelemetry is the vendor-neutral CNCF standard for telemetry, >>> supported by every major observability backend (Prometheus, CloudWatch, >>> Datadog, Grafana Cloud, etc.). >>> A single OTLP-based MetricsReporter in Iceberg lets users reach all of >>> these without per-vendor integrations in the project. >>> This is complementary to #14360, which adds OTel support to HTTPClient >>> at the REST-catalog HTTP layer; this proposal covers the Iceberg-level >>> ScanReport / CommitReport layer. >>> >>> # Proposal >>> Issue: https://github.com/apache/iceberg/issues/16169 >>> PR: https://github.com/apache/iceberg/pull/16250 >>> >>> The reporter follows the same SDK-ownership philosophy as #14360 - the >>> host application (Spark/Flink/Trino/...) registers an OpenTelemetrySdk via >>> GlobalOpenTelemetry, and the reporter just looks up a Meter from it. >>> The reporter has zero Iceberg-specific catalog properties; everything >>> else is owned by the host. >>> >>> The PR has been validated end-to-end against two unrelated OTLP backends >>> (Databricks Zerobus and Amazon CloudWatch) - full procedures and queries >>> are linked from the PR. >>> >>> # On dependencies >>> Given the current sensitivity around new runtime dependencies in 1.11, >>> the PR adds only opentelemetry-api to iceberg-core as compileOnly. >>> The OpenTelemetry SDK and OTLP exporters are not added to the runtime >>> classpath >>> - they come from the host application. >>> opentelemetry-sdk / -sdk-testing are testImplementation only. >>> >>> # Questions for the community >>> >>> Q1. Any objection to taking the opentelemetry-api compileOnly >>> dependency in iceberg-core? >>> Q2. Module placement: iceberg-core (current PR), or a >>> separate iceberg-opentelemetry module? >>> >>> Thanks, >>> Noritaka Sekiyama, Databricks >>> >>
