peterxcli opened a new pull request, #5554:
URL: https://github.com/apache/datafusion-comet/pull/5554
## Which issue does this PR close?
Part of #3996.
## Rationale for this change
The direct native shuffle-read path bypasses Spark's normal decode path, so
its records-read metric was not updated and temporary shuffle-read metrics were
not merged into task metrics. This left the Spark Stage and SQL UIs with
missing or zero native shuffle-read values.
## What changes are included in this PR?
- Pass the existing SQL shuffle-read metrics callback through
`CometShuffleBlockIterator`.
- Update Spark's records-read metric after each successfully decoded native
shuffle batch.
- Merge temporary shuffle-read metrics when a direct native shuffle task
completes.
- Extend the task-metrics regression test to require the direct native
`ShuffleScan` path and verify the exact record count.
## How are these changes tested?
- `cargo build`
- `cargo fmt --all -- --check`
- `./mvnw test -Dtest=none
-Dsuites='org.apache.spark.sql.comet.CometTaskMetricsSuite per-task native
shuffle metrics' -Dscalastyle.skip=true`
- Before rebasing onto the latest `main`, the same fix was also verified
with a built Comet JAR on a fresh Spark 4.1.2 standalone cluster with two
workers:
- executor logs confirmed `ShuffleScanExec`
- the Stage UI showed `Shuffle Read Size / Records: 2004.6 KiB / 500000`
- the SQL UI showed 500,000 records, 2004.6 KiB read, and 32 local blocks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
