Shujing Yang created SPARK-60570:
------------------------------------
Summary: Add AccumulatorContext.getOrNull to avoid an Option per
driver-side accumulator lookup
Key: SPARK-60570
URL: https://issues.apache.org/jira/browse/SPARK-60570
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Shujing Yang
DAGScheduler.updateAccumulators runs on the driver for every finished task and
looks up every accumulator update the task carries, which can be hundreds when
SQL metrics are involved. AccumulatorContext.get returns an Option, so each
lookup allocates a Some that is dropped right away. Add
AccumulatorContext.getOrNull, which does the same lookup but returns null when
the accumulator is not registered or has been garbage collected. get delegates
to it, and updateAccumulators uses it directly. No behavior change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
