GitHub user seyfe opened a pull request:
https://github.com/apache/spark/pull/15371
[SPARK-17463] [Core] TaskInfoToJson should map using List(immutable) rather
than ListBuffer(mutable)
## What changes were proposed in this pull request?
taskInfo.accumulables is ListBuffer and causing
java.util.ConcurrentModificationException.
Converted to List before doing map will fix the concurrency issue here.
## How was this patch tested?
This is the stack trace for this exception:
`
java.util.ConcurrentModificationException
at
java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
at java.util.ArrayList$Itr.next(ArrayList.java:851)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at
scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:183)
at
scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45)
at
scala.collection.TraversableLike$class.to(TraversableLike.scala:590)
at scala.collection.AbstractTraversable.to(Traversable.scala:104)
at
scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:294)
at
scala.collection.AbstractTraversable.toList(Traversable.scala:104)
at
org.apache.spark.util.JsonProtocol$.accumValueToJson(JsonProtocol.scala:314)
at
org.apache.spark.util.JsonProtocol$$anonfun$accumulableInfoToJson$5.apply(JsonProtocol.scala:291)
at
org.apache.spark.util.JsonProtocol$$anonfun$accumulableInfoToJson$5.apply(JsonProtocol.scala:291)
at scala.Option.map(Option.scala:146)
at
org.apache.spark.util.JsonProtocol$.accumulableInfoToJson(JsonProtocol.scala:291)
at
org.apache.spark.util.JsonProtocol$$anonfun$taskInfoToJson$12.apply(JsonProtocol.scala:283)
at
org.apache.spark.util.JsonProtocol$$anonfun$taskInfoToJson$12.apply(JsonProtocol.scala:283)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.spark.util.JsonProtocol$.taskInfoToJson(JsonProtocol.scala:283)
at
org.apache.spark.util.JsonProtocol$.taskEndToJson(JsonProtocol.scala:145)
at
org.apache.spark.util.JsonProtocol$.sparkEventToJson(JsonProtocol.scala:76)
`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/seyfe/spark race_cond_jsonprotocal
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15371.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15371
----
commit 07d9a8b26469dd6becc36f2dc9e17a58c53ff4e6
Author: Ergin Seyfe <[email protected]>
Date: 2016-10-06T01:34:57Z
Map for List rather than ListBuffer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
