Prabhat Mishra created IMPALA-15447:
---------------------------------------
Summary: Iceberg scan telemetry (Stream A + B) to report Iceberg
scan metrics.
Key: IMPALA-15447
URL: https://issues.apache.org/jira/browse/IMPALA-15447
Project: IMPALA
Issue Type: Improvement
Components: Catalog, Frontend
Reporter: Prabhat Mishra
Description:
Adds two-stream Iceberg query telemetry to the Impala frontend (Java/fe):
Stream A — Scan Reports (IcebergScanPlanner.java, IcebergUtil.java):
- Wires a catalog-configured side MetricsReporter alongside the existing
InMemoryMetricsReporter in IcebergScanPlanner. The existing
InMemoryMetricsReporter is kept because it feeds
addIcebergScanMetricsToProfile() (query profile).
- The side reporter is loaded once per JVM (double-checked locking, volatile)
via CatalogUtil.loadMetricsReporter, driven by the metrics-reporter-impl JVM
system property set through Cloudera Manager.
- For the fast-path where planFiles() is skipped, synthesizes a faithful
ScanReport from already-computed file descriptors and posts it only to the side
reporter (InMemory reporter and query profile are left unchanged).
- Stamps a query-id and engine-name onto every scan report via a new
IcebergUtil.planFiles overload that passes scan options as report metadata,
enabling cross-stream correlation downstream.
Stream B — Query Shape (IcebergQueryShapeCollector.java):
- New class IcebergQueryShapeCollector runs once per query, right after
Planner.createPlans.
- Walks the plan tree, extracts equi-join keys and ORDER BY columns that
resolve to Iceberg tables.
- POSTs a per-table query-shape event via the same telemetry library (called
reflectively to avoid a compile-time dependency), tagged with the identical
query-id used in Stream A.
- Hooked into Frontend.createExecRequest (Frontend.java).
Both streams are entirely best-effort: neither throws nor blocks query
planning, and both no-op when no side reporter is configured (preserving
today's default behavior byte-for-byte).
---
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
