[
https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088919#comment-16088919
]
Ryan Williams commented on SPARK-21425:
---------------------------------------
Digging a bit more, it seems like:
- there's no special handling of accumulators w/r/t sending them to tasks; a
separate copy gets sent to each task just like would happen to any local,
mutable variable on the driver.
- If a task-closure references a global variable, like a field of a singleton,
all tasks on a given executor/JVM will reference the same/global instance of
that variable, which is why I am seeing this "bug".
It seems like the lesson is to be a little careful having accumulators that are
global or otherwise get directly referenced by multiple tasks after
closure-serialization, as opposed to the normal pattern where they are declared
as local variables scoped to some instance that the closure-serializer makes a
copy of for each task.
For context, I had my accumulator-definitions out in a singleton object in the
first because I needed to avoid local accumulators causing an enclosing
Scalatest suite to be serialized, cf.
[scalatest#1013|https://github.com/scalatest/scalatest/issues/1013].
We can probably say "not a bug" here, though I was surprised that
accumulator-references aren't explicitly handled under serialization the way
{{Broadcast}}'s are, but I guess that should have been evident by a lack of
explicit {{Broadcast.value}}-style API for referencing them.
> LongAccumulator, DoubleAccumulator not threadsafe
> -------------------------------------------------
>
> Key: SPARK-21425
> URL: https://issues.apache.org/jira/browse/SPARK-21425
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0
> Reporter: Ryan Williams
>
> [AccumulatorV2
> docs|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L42-L43]
> acknowledge that accumulators must be concurrent-read-safe, but afaict they
> must also be concurrent-write-safe.
> The same docs imply that {{Int}} and {{Long}} meet either/both of these
> criteria, when afaict they do not.
> Relatedly, the provided
> [LongAccumulator|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L291]
> and
> [DoubleAccumulator|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L370]
> are not thread-safe, and should be expected to behave undefinedly when
> multiple concurrent tasks on the same executor write to them.
> [Here is a repro repo|https://github.com/ryan-williams/spark-bugs/tree/accum]
> with some simple applications that demonstrate incorrect results from
> {{LongAccumulator}}'s.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
