[
https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095744#comment-16095744
]
Shixiong Zhu commented on SPARK-21425:
--------------------------------------
[~rdub] I just realized we never document local-cluster mode. You can use
"--master local-cluster[N, cores, memory]" to start a cluster which runs
driver, worker and master in one process, and executors in their own processes
(See
https://github.com/apache/spark/blob/3ac60930865209bf804ec6506d9d8b0ddd613157/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala)
Spark assumes accumulators are always captured by closures. When starting a
task in an executor, Spark will deserialize the closure. In this step, it
create a copy for each accumulator in the executor. When a task finishes, the
value in the copied accumulator will be sent back to driver. Driver will merge
all values collected from tasks and set the final value to the accumulator in
the driver side. Because static accumulators cannot be captured by closures,
the mechanism doesn't work for them. See AccumulatorV2.writeReplace and
AccumulatorV2.readObject if you want to dig into details.
> LongAccumulator, DoubleAccumulator not threadsafe
> -------------------------------------------------
>
> Key: SPARK-21425
> URL: https://issues.apache.org/jira/browse/SPARK-21425
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0
> Reporter: Ryan Williams
> Priority: Minor
>
> [AccumulatorV2
> docs|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L42-L43]
> acknowledge that accumulators must be concurrent-read-safe, but afaict they
> must also be concurrent-write-safe.
> The same docs imply that {{Int}} and {{Long}} meet either/both of these
> criteria, when afaict they do not.
> Relatedly, the provided
> [LongAccumulator|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L291]
> and
> [DoubleAccumulator|https://github.com/apache/spark/blob/v2.2.0/core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala#L370]
> are not thread-safe, and should be expected to behave undefinedly when
> multiple concurrent tasks on the same executor write to them.
> [Here is a repro repo|https://github.com/ryan-williams/spark-bugs/tree/accum]
> with some simple applications that demonstrate incorrect results from
> {{LongAccumulator}}'s.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
