[
https://issues.apache.org/jira/browse/IMPALA-15447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhat Mishra updated IMPALA-15447:
------------------------------------
Description:
Adds two-stream Iceberg query telemetry to the Impala frontend (Java/fe):
Stream A — Scan Reports (IcebergScanPlanner.java, IcebergUtil.java):
- Wires a catalog-configured side MetricsReporter alongside the existing
InMemoryMetricsReporter in IcebergScanPlanner. The existing
InMemoryMetricsReporter is kept because it feeds
addIcebergScanMetricsToProfile() (query profile).
- The side reporter is loaded once per JVM (double-checked locking, volatile)
via CatalogUtil.loadMetricsReporter, driven by the metrics-reporter-impl JVM
system property
- For the fast-path where planFiles() is skipped, synthesizes a faithful
ScanReport from already-computed file descriptors and posts it only to the side
reporter (InMemory reporter and query profile are left unchanged).
- Stamps a query-id and engine-name onto every scan report via a new
IcebergUtil.planFiles overload that passes scan options as report metadata,
enabling cross-stream correlation downstream.
Stream B — Query Shape (IcebergQueryShapeCollector.java):
- New class IcebergQueryShapeCollector runs once per query, right after
Planner.createPlans.
- Walks the plan tree, extracts equi-join keys and ORDER BY columns that
resolve to Iceberg tables.
- POSTs a per-table query-shape event via the same telemetry library (called
reflectively to avoid a compile-time dependency), tagged with the identical
query-id used in Stream A.
- Hooked into Frontend.createExecRequest (Frontend.java).
Both streams are entirely best-effort: neither throws nor blocks query
planning, and both no-op when no side reporter is configured (preserving
today's default behavior byte-for-byte).
—
was:
Adds two-stream Iceberg query telemetry to the Impala frontend (Java/fe):
Stream A — Scan Reports (IcebergScanPlanner.java, IcebergUtil.java):
- Wires a catalog-configured side MetricsReporter alongside the existing
InMemoryMetricsReporter in IcebergScanPlanner. The existing
InMemoryMetricsReporter is kept because it feeds
addIcebergScanMetricsToProfile() (query profile).
- The side reporter is loaded once per JVM (double-checked locking, volatile)
via CatalogUtil.loadMetricsReporter, driven by the metrics-reporter-impl JVM
system property set through Cloudera Manager.
- For the fast-path where planFiles() is skipped, synthesizes a faithful
ScanReport from already-computed file descriptors and posts it only to the side
reporter (InMemory reporter and query profile are left unchanged).
- Stamps a query-id and engine-name onto every scan report via a new
IcebergUtil.planFiles overload that passes scan options as report metadata,
enabling cross-stream correlation downstream.
Stream B — Query Shape (IcebergQueryShapeCollector.java):
- New class IcebergQueryShapeCollector runs once per query, right after
Planner.createPlans.
- Walks the plan tree, extracts equi-join keys and ORDER BY columns that
resolve to Iceberg tables.
- POSTs a per-table query-shape event via the same telemetry library (called
reflectively to avoid a compile-time dependency), tagged with the identical
query-id used in Stream A.
- Hooked into Frontend.createExecRequest (Frontend.java).
Both streams are entirely best-effort: neither throws nor blocks query
planning, and both no-op when no side reporter is configured (preserving
today's default behavior byte-for-byte).
—
> Iceberg scan telemetry (Stream A + B) to report Iceberg scan metrics.
> ---------------------------------------------------------------------
>
> Key: IMPALA-15447
> URL: https://issues.apache.org/jira/browse/IMPALA-15447
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Prabhat Mishra
> Priority: Major
>
> Adds two-stream Iceberg query telemetry to the Impala frontend (Java/fe):
> Stream A — Scan Reports (IcebergScanPlanner.java, IcebergUtil.java):
> - Wires a catalog-configured side MetricsReporter alongside the existing
> InMemoryMetricsReporter in IcebergScanPlanner. The existing
> InMemoryMetricsReporter is kept because it feeds
> addIcebergScanMetricsToProfile() (query profile).
> - The side reporter is loaded once per JVM (double-checked locking,
> volatile) via CatalogUtil.loadMetricsReporter, driven by the
> metrics-reporter-impl JVM system property
> - For the fast-path where planFiles() is skipped, synthesizes a faithful
> ScanReport from already-computed file descriptors and posts it only to the
> side reporter (InMemory reporter and query profile are left unchanged).
> - Stamps a query-id and engine-name onto every scan report via a new
> IcebergUtil.planFiles overload that passes scan options as report metadata,
> enabling cross-stream correlation downstream.
> Stream B — Query Shape (IcebergQueryShapeCollector.java):
> - New class IcebergQueryShapeCollector runs once per query, right after
> Planner.createPlans.
> - Walks the plan tree, extracts equi-join keys and ORDER BY columns that
> resolve to Iceberg tables.
> - POSTs a per-table query-shape event via the same telemetry library (called
> reflectively to avoid a compile-time dependency), tagged with the identical
> query-id used in Stream A.
> - Hooked into Frontend.createExecRequest (Frontend.java).
> Both streams are entirely best-effort: neither throws nor blocks query
> planning, and both no-op when no side reporter is configured (preserving
> today's default behavior byte-for-byte).
> —
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
