yew1eb opened a new pull request, #3869:
URL: https://github.com/apache/celeborn/pull/3869
## What
Extends the CELEBORN-2415 Spark UI skeleton into a full observability tab:
shuffle write/read metrics collected end-to-end from the executors, aggregated
per shuffle / per worker / globally, and rendered in a Uniffle-style page —
with full HistoryServer replay support.
Fixes CELEBORN-2477 (UI extension configs were read by bare keys from
SparkConf and thus never picked up user-set `spark.celeborn.*` values).
## Implementation
Data flow (mirrors how Spark's own status system works):
```
executor instrumentation
→ MapperEnd RPC (WriteMetrics + PushWorkerStats)
→ ReportShuffleReadMetrics RPC (ReadMetrics + WorkerReadCost) [new]
→ driver callbacks → CelebornEvent on listener bus (status queue → event
log)
→ CelebornListener aggregation (global / per-shuffle / per-worker)
→ KVStore → CelebornStatusStore → CelebornShufflePage
```
- **Write path**: `PushState` / `InFlightRequestTracker` / `DataPushQueue`
accumulate copy / serialize / compress / queue-wait / inflight-wait / drain
timings, slow pushes and per-worker push stats (RTT, soft/hard split,
congestions, last failure reason).
- **Read path**: `ReadStreamStats` (new) accumulates decompress / chunk-wait
/ deserialize / copy timings, retries, peer switches, excludes and per-worker
read cost.
- **Driver**: `SparkShuffleManager` / `LifecycleManager` forward the reports
as `CelebornEvent`s onto the listener bus status queue, so custom events
persist into the event log and are rebuilt on HistoryServer replay (opt-in
marker persisted when `spark.plugins` contains `CelebornPlugin`).
- **Listener**: a fused `CelebornListener` consumes both native
`TaskMetrics` (per-task shuffle write/read) and the custom events, with a 5s
throttled flush, a final flush on store close, and a `retainedShuffles` cap on
assignment rows.
- **Page** sections (all collapsible): headline stat tiles (shuffles /
workers / write+read bytes / compression ratio / fallback total), Per-Shuffle
Details (assignments outer-joined with per-shuffle write and read metrics, with
derived `% of total write` and speed columns), write/read time breakdowns
(Metric | Duration | Count), per-worker write/read tables (avg RTT, sorted by
bytes), Celeborn Properties, Shuffle Assignments, and Shuffle Fallback.
- **Gating**: `celeborn.client.spark.ui.enabled` (default `true`, together
with `spark.ui.enabled`); when off, no callbacks are registered and no events
are posted — zero overhead.
## Config read fix (CELEBORN-2477)
`CelebornUIUtils.isUIEnabled` and `CelebornListener` previously read their
configs by bare keys (`celeborn.client.spark.ui.enabled` / `.retainedShuffles`)
directly from `SparkConf`, which never match the user-set `spark.celeborn.*`
keys. Both now go through a `CelebornConf` conversion and the config entries,
so the documented keys apply, the entry defaults are honored, and the
`retainedShuffle` alias works. The conversion is local to the `spark-3-ui`
module because `SparkUtils`' class init drags in spark-sql's
`UnsafeRowSerializer`, which is not on this module's classpath and is absent in
RDD-only applications.
## Compatibility
- No existing KVStore entity or event is modified; new entities carry no-arg
constructors and reads fall back to empty defaults, so old event logs replay
unchanged and old readers ignore the new entities.
- A single page implementation serves Spark 3.x (javax.servlet) and Spark
4.x (jakarta.servlet) through `SparkServletBridge`.
## Tests
- `CelebornListenerSuite` (13): throttle/flush, gating (incl. the new
`spark.celeborn.*` prefix case), redaction, retainedShuffles eviction,
per-shuffle write/read aggregation, replay sequence.
- `CelebornShufflePageSuite` (2): renders all sections against an in-memory
store; derived columns and default orderings.
- `ReadStreamStatsSuiteJ`: read-side stats accumulation.
Verified with `mvn -Pspark-3.5 -pl client-spark/spark-3-ui -am test`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
