Shujing Yang created SPARK-60571:
------------------------------------
Summary: Set the task result size accumulator in place in
TaskResultGetter
Key: SPARK-60571
URL: https://issues.apache.org/jira/browse/SPARK-60571
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Shujing Yang
When TaskResultGetter receives a task result, it sets the task result size on
the RESULT_SIZE accumulator with map, which rebuilds the whole accumulator
update sequence once per task result on the driver. The sequence can hold
hundreds of entries when SQL metrics are involved, and each name check
allocates a Some. The matching accumulator was already mutated in place, so the
rebuilt sequence held the same objects. Walk the updates with foreach, set the
matching LongAccumulator in place, and use Option.contains for the name check.
No behavior change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
