wangyum opened a new pull request, #58968: URL: https://github.com/apache/spark/pull/58968
### What changes were proposed in this pull request? `LiveExecutorStageSummary` no longer keeps a `v1.TaskMetrics` object graph (`createMetrics` / `addMetrics`). It stores only the 10 longs that `ExecutorStageSummary` already exposes, and accumulates them with `addTaskMetrics`. `TaskEnd` and `ExecutorMetricsUpdate` call `addTaskMetrics` instead of allocating a new metrics tree. `doUpdate()` still writes the same `ExecutorStageSummary` fields, so the REST/UI payload is unchanged. ### Why are the changes needed? The executor-stage UI only needs: `inputBytes`, `inputRecords`, `outputBytes`, `outputRecords`, `shuffleRead`, `shuffleReadRecords`, `shuffleWrite`, `shuffleWriteRecords`, `memoryBytesSpilled`, `diskBytesSpilled`. The live object previously held a full `TaskMetrics` + `Input`/`Output`/`ShuffleRead`/`ShufflePushRead`/`ShuffleWrite` tree per `(stage, executor)`. Most of those fields are never shown. `addMetrics` also allocated a new tree on every task update, so even short jobs paid extra GC. On a long-lived driver (many live stages × many executors) this shows up as tens of millions of `TaskMetrics` instances. This change is a memory-shape fix: same numbers, fewer objects. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - New unit test: `LiveExecutorStageSummary` has no `v1.TaskMetrics` field. - `AppStatusListenerWithInMemoryStoreSuite` (24 tests), including SPARK-24415 late task metrics and executor-summary cases. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
