Hi all, I'd like to get broader input on the design of table-level filtering for MetricsReporter. Gabor raised two design concerns on the PR and suggested bringing this to the list before moving forward, which I think is right.
# Background Iceberg has no built-in way to restrict which tables' ScanReports and CommitReports reach the configured MetricsReporter. In deployments with many tables, users frequently want metrics for only a subset: production databases but not staging, or everything except noisy scratch and benchmark tables. The proposal sits above any reporter implementation, so it applies uniformly to LoggingMetricsReporter, RESTMetricsReporter, and any custom user-supplied reporter. Some context on where this came from, since it matters for how much weight to give the OTel angle below: this was proposed in the DISCUSS thread for an OpenTelemetry MetricsReporter [1], where per-table metric cardinality came up. That reporter is still an open proposal under review, not a merged feature, so I've scoped this PR deliberately as reporter-agnostic rather than OTel-specific. It stands on its own for the reporters that exist today, and would apply to an OTel reporter as well if that one lands. Issue: https://github.com/apache/iceberg/issues/16573 PR: https://github.com/apache/iceberg/pull/16574 # Proposal The current PR adds two catalog properties holding Java regexes matched against the table name, with exclude winning over include when both are set: metrics-reporter.table-name.include=prod_db\..* metrics-reporter.table-name.exclude=.*\.tmp_.* When neither is set the resolved reporter is returned unwrapped, so the default path is unchanged. # Q1. Is the table name the right thing to predicate on? The concern is that a regex written today can silently capture tables created tomorrow, and the failure mode is invisible: metrics simply stop appearing for a table nobody thought about. Four shapes seem possible. 1. Table-name regex (current PR). Expresses the motivating cases directly, and has precedent in iceberg-kafka-connect, which compiles a user-supplied route-regex from config. Carries exactly the risk above. 2. Exact-match list of table names. No surprise capture, fully auditable, but cannot express "only production tables" without enumerating and maintaining every name, which given Q2 also means a catalog restart per addition. 3. Namespace-level filtering. Most motivating cases are really "this whole database" rather than "these specific tables", and a namespace is structural rather than a naming convention. It cannot express intra-namespace exclusions, and ScanReport/CommitReport expose only a flat tableName() string today, so it would mean parsing the name back into an identifier. 4. Exact list as the default, regex as an opt-in. Two mechanisms for one job. I lean toward the regex or namespace shape, without strong conviction. This is ultimately a question about how operators organize tables in practice, which is why I'd like more input here specifically. # Q2. Catalog properties are static for the catalog's lifetime The patterns compile once at catalog initialization, so changing them requires restarting the client-side catalog. Engines that create a catalog per query are unaffected; engines that hold a single long-lived catalog connector are not. This is not specific to this feature: metrics-reporter-impl and every other catalog property behave the same way. 1. Accept the static lifecycle (current PR). Consistent with existing properties, but tuning becomes a restart-level operation, which in practice means nobody tunes it. 2. Add reload support. This would mean designing a mechanism for re-reading catalog properties after initialization, which Iceberg does not have today. That seems too large to settle as a side effect of this feature. 3. Leave dynamic control to reporter implementations. MetricsReporter.initialize(Map) already exists, so users needing runtime control can implement it against whatever config source they already operate. This narrows the proposal to the common static case. I think the second is out of scope, which makes this a choice between the first and third. # Should this exist in the framework at all? Worth naming the option of shipping no built-in filter and leaving this to reporter implementations and backend tooling. Since the idea originated in the OTel discussion I looked into how OpenTelemetry expects this to be solved, and the answer is narrower than I had assumed. Note this section is about the OTel model as prior art; it does not assume the OTel reporter proposal [1] is accepted. 1. SDK Views cannot express this. Instrument selection predicates are name, type, unit, meter_name, meter_version and meter_schema_url; attribute values are not selection criteria, and attribute_keys is an allow/exclude list of attribute keys rather than values. So dropping the streams where a table-name attribute matches tmp.* is not expressible as a View [2]. 2. In the OTel model, value-based filtering lives either in the Collector via filterprocessor, or at instrumentation time before recording. Filtering in the reporter layer is the latter, so it is not a deviation from how OTel expects this to work. 3. The Collector path only covers OTel users, and only if such a reporter lands at all. LoggingMetricsReporter and RESTMetricsReporter have no equivalent mechanism, and a per-reporter solution means each one re-solves the problem slightly differently, which was the original argument for putting it above the reporter layer. So leaving it to backend tooling is legitimate, but it specifically serves OTel-with-a-Collector users and not the others. I'm happy to rework the PR toward whichever shape the community prefers, including dropping it if the consensus is that this doesn't belong in the framework. Input on Q1 in particular would be helpful. Thanks, Noritaka Sekiyama, Databricks [1] https://github.com/apache/iceberg/pull/16250 [2] https://opentelemetry.io/docs/specs/otel/metrics/sdk/
