yew1eb opened a new pull request, #3872: URL: https://github.com/apache/celeborn/pull/3872
## What changes were proposed in this PR? Track the total bytes handed to `ShuffleClient` by each mapper **before** compression/encryption, and surface the aggregated **compression ratio (uncompressed/compressed)** on the Celeborn tab of the Spark UI. Close #CELEBORN-2479. (issue: apache/celeborn CELEBORN-2479) ### Change breakdown - **Writers** (`HashBasedShuffleWriter`, `SortBasedPusher` via `SortBasedShuffleWriter`) count uncompressed bytes and report them at `mapperEnd` through a new extensible `MapperMetrics` message. - **Protocol**: new `PbMapperMetrics` message carried by `PbMapperEnd` as field 10 (`metrics`); `ControlMessages.MapperEnd` gains a defaulted `MapperMetrics` parameter, so existing callers remain source/binary compatible. Future mapper-side metrics should extend `MapperMetrics` instead of `PbMapperEnd`. - **LifecycleManager** gains `registerMapperEndCallback`; on each successfully finished mapper attempt it invokes the callback with the metrics (only effective when LifecycleManager runs in the driver JVM, which is the Spark case). - **SparkShuffleManager** registers a callback that posts `CelebornMapperMetricsEvent` (a `SparkListenerEvent` with `logEvent() = true`) onto the driver listener bus, so the metric is available in the live UI **and** replayable by the History Server from the event log. - **UI**: `CelebornListener` aggregates the events into `AggregatedTaskInfoUIData.uncompressedShuffleBytes`; `CelebornShufflePage` renders `Compression Ratio (uncompressed/compressed): <uncompressed>/<written> = x.xx`. Old clients / unreported values degrade to `N/A`. - **docker**: register `spark.plugins=...CelebornPlugin` in `docker/spark/conf/spark-defaults.conf` so the local docker-compose cluster exercises the Celeborn UI tab end-to-end. ## How was this patch tested? - New unit tests: `CelebornListenerSuite.aggregateMapperMetricsEvents`, `CelebornListenerSuite.mapperMetricsEventJsonRoundtrip` (event-log roundtrip), extended `UtilsSuite` MapperEnd ser/de roundtrip with `MapperMetrics`. - End-to-end on the docker-compose stack (celeborn:dev + celeborn-spark:dev, Spark 3.5.9, hash writer, lz4): ran a Spark job whose shuffle went through Celeborn and confirmed the Celeborn tab shows the compression ratio while the driver is alive. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
