GitHub user seyfe opened a pull request:
https://github.com/apache/spark/pull/15425
[SPARK-17816] [Core] [Branch-2.0] Fix ConcurrentModificationException issue
in BlockStatusesAccumulator
## What changes were proposed in this pull request?
Replaced `BlockStatusesAccumulator` with `CollectionAccumulator` which is
thread safe and few more cleanups.
## How was this patch tested?
Tested in master branch and cherry-picked.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/seyfe/spark race_cond_jsonprotocal_branch-2.0
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15425.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15425
----
commit 678ee6b1d6308a81a5c2d83a196144f29c80434b
Author: Ergin Seyfe <[email protected]>
Date: 2016-10-11T03:41:31Z
[SPARK-17816][CORE] Fix ConcurrentModificationException issue in
BlockStatusesAccumulator
Change the BlockStatusesAccumulator to return immutable object when value
method is called.
Existing tests plus I verified this change by running a pipeline which
consistently repro this issue.
This is the stack trace for this exception:
`
java.util.ConcurrentModificationException
at
java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
at java.util.ArrayList$Itr.next(ArrayList.java:851)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at
scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:183)
at
scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45)
at
scala.collection.TraversableLike$class.to(TraversableLike.scala:590)
at scala.collection.AbstractTraversable.to(Traversable.scala:104)
at
scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:294)
at
scala.collection.AbstractTraversable.toList(Traversable.scala:104)
at
org.apache.spark.util.JsonProtocol$.accumValueToJson(JsonProtocol.scala:314)
at
org.apache.spark.util.JsonProtocol$$anonfun$accumulableInfoToJson$5.apply(JsonProtocol.scala:291)
at
org.apache.spark.util.JsonProtocol$$anonfun$accumulableInfoToJson$5.apply(JsonProtocol.scala:291)
at scala.Option.map(Option.scala:146)
at
org.apache.spark.util.JsonProtocol$.accumulableInfoToJson(JsonProtocol.scala:291)
at
org.apache.spark.util.JsonProtocol$$anonfun$taskInfoToJson$12.apply(JsonProtocol.scala:283)
at
org.apache.spark.util.JsonProtocol$$anonfun$taskInfoToJson$12.apply(JsonProtocol.scala:283)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.spark.util.JsonProtocol$.taskInfoToJson(JsonProtocol.scala:283)
at
org.apache.spark.util.JsonProtocol$.taskEndToJson(JsonProtocol.scala:145)
at
org.apache.spark.util.JsonProtocol$.sparkEventToJson(JsonProtocol.scala:76)
`
Author: Ergin Seyfe <[email protected]>
Closes #15371 from seyfe/race_cond_jsonprotocal.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
