[
https://issues.apache.org/jira/browse/SPARK-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341753#comment-14341753
]
Josh Rosen commented on SPARK-3885:
-----------------------------------
I found a correctness issue in this patch, which I'll fix shortly: see
SPARK-6075
> Provide mechanism to remove accumulators once they are no longer used
> ---------------------------------------------------------------------
>
> Key: SPARK-3885
> URL: https://issues.apache.org/jira/browse/SPARK-3885
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.2, 1.1.0, 1.2.0
> Reporter: Josh Rosen
> Assignee: Ilya Ganelin
> Fix For: 1.4.0
>
>
> Spark does not currently provide any mechanism to delete accumulators after
> they are no longer used. This can lead to OOMs for long-lived SparkContexts
> that create many large accumulators.
> Part of the problem is that accumulators are registered in a global
> {{Accumulators}} registry. Maybe the fix would be as simple as using weak
> references in the Accumulators registry so that accumulators can be GC'd once
> they can no longer be used.
> In the meantime, here's a workaround that users can try:
> Accumulators have a public setValue() method that can be called (only by the
> driver) to change an accumulator’s value. You might be able to use this to
> reset accumulators’ values to smaller objects (e.g. the “zero” object of
> whatever your accumulator type is, or ‘null’ if you’re sure that the
> accumulator will never be accessed again).
> This issue was originally reported by [~nkronenfeld] on the dev mailing list:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Fwd-Accumulator-question-td8709.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
