parthchandra opened a new pull request, #6027:
URL: https://github.com/apache/datafusion-comet/pull/6027
## Which issue does this PR close?
Closes #6026
## Rationale for this change
When Comet runs an Iceberg scan natively, the Spark UI showed 0 for every
Iceberg planning metric
(file counts, manifest counts, sizes, planning duration) and had no scan
time. Plain Spark + Iceberg
shows all of these (they come from Iceberg's Java planning), so people lost
that visibility whenever
Comet took over the scan. These numbers help you understand how much data a
query read and how much
work planning did.
## What changes are included in this PR?
All changes are on the Comet side (no iceberg-rust changes needed):
1. **Send the Iceberg planning metrics to the UI.** The values were already
computed on the driver
from Iceberg's Java planning, but they were never posted, so the UI kept
showing 0. Posting is
done in a new `sendDriverMetrics()` step. A native leaf scan does not
always run its own
`doExecuteColumnar` (when it is fused under a parent native operator such
as a filter, the parent
runs the whole subtree as one RDD), so the post is driven from
`PlanDataInjector.findAllPlanData`,
the plan walk that reaches every leaf scan at execution time.
2. **Make this a general leaf-scan hook.** `sendDriverMetrics()` is a
`CometLeafExec` method
(a no-op by default), called uniformly from `findAllPlanData` right next
to the existing
`ensureSubqueriesResolved()` hook. `CometIcebergNativeScanExec` overrides
it. Any future native
leaf scan with driver-side metrics can opt in the same way instead of
adding another special case.
3. **Add a "scan time" metric.** Wired the native scan's compute time
(`elapsed_compute`) to a Spark
timing metric named "scan time".
4. **Actually measure the scan time.** The native scan was not timing itself
— `record_poll` only
records output rows — so "scan time" stayed at 0 no matter how much data
it read. Added an
`elapsed_compute` timer around the work in the scan's `poll_next`.
## How are these changes tested?
- New test in `CometIcebergNativeSuite` that runs a native Iceberg scan and
checks the planning
metrics and scan time actually reach the Spark UI, by reading them back
from the SQL status store
(the same source the UI uses), not just from the plan node.
- Existing native `iceberg_scan` Rust tests still pass.
- Verified manually in `spark-shell` on a 20M-row table: the planning
numbers match what Iceberg-Java
reports (1 manifest, 16 data files, ~159 MiB scanned, ~204 ms planning)
and scan time reads a real
value (~6 s across tasks).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
