luoyuqinlaura opened a new pull request, #59966: URL: https://github.com/apache/spark/pull/59966
### What changes were proposed in this pull request? - In `SQLAppStatusListener.aggregateMetrics`, group driver accumulator updates by accumulator id and append each group to the metric array once, instead of copying the array for every update. - Change `LiveExecutionData.driverAccumUpdates` from `Seq` to `Vector` so appends are cheap while snapshots stay immutable for concurrent readers. ### Why are the changes needed? `aggregateMetrics` appended driver updates one at a time with `Arrays.copyOf(prev, prev.length + 1)`, which is O(N^2) in the number of updates for the same accumulator id, and `driverAccumUpdates ++ accumUpdates` rebuilt the list on every event. For applications with a very large number of driver accumulator updates (e.g. ~459K and ~918K TaskEnd events), this dominates History Server replay time. Used Spark's built-in `org.apache.spark.benchmark.Benchmark` framework. (1)Benchmark for commit 1: `aggregateMetrics` grouping Measures the time to process N accumulated updates during metric aggregation. numbers of updates(N) | Old – Best Time (ms) | New – Best Time (ms) | Speedup -- | -- | -- | -- 10,000 | 10 | ~0.01 | ~94x 50,000 | 196 | 1 | ~278x 100,000 | 745 | 1 | ~515x 200,000 | 2,903 | 3 | ~958x (2)Benchmark for commit 2: driverAccumUpdates append (List vs Vector) Number of Updates | List (ms) | Vector (ms) | Speedup -- | -- | -- | -- 10,000 | 85 | 1 | ~155x 50,000 | 2,118 | 3 | ~744x 100,000 | 8,766 | 6 | ~1,517x 200,000 | 36,176 | 12 | ~3,016x ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added a unit test in `SQLAppStatusListenerSuite` for repeated driver updates of the same accumulator, and ran the existing `SQLAppStatusListener*Suite`s. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
