Yuqin Luo created SPARK-60759:
---------------------------------
Summary: SQLAppStatusListener.aggregateMetrics has O(N^2) cost
when merging driver accumulator updates
Key: SPARK-60759
URL: https://issues.apache.org/jira/browse/SPARK-60759
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.6
Reporter: Yuqin Luo
During event-log replay, Spark's {{SQLAppStatusListener.aggregateMetrics()}}
repeatedly copies the accumulated metric arrays, which becomes extremely
expensive for these apps.
A similar O(N²) issue exists in LiveExecutionData.driverAccumUpdates, where
repeated List ++ rebuilds the collection on every event.
h2.
*FIX:* Commit 1 — Group updates by ID and merge them once, avoiding repeated
array copies, reducing aggregation from O(N²) to O(N).
Commit 2 — Change {{LiveExecutionData.driverAccumUpdates}} from *List* to
*Vector* so new updates can be appended more efficiently.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
