[
https://issues.apache.org/jira/browse/SPARK-15002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421306#comment-15421306
]
Jamie Hutton commented on SPARK-15002:
--------------------------------------
Hi Sean,
Looking at the stack trace on the executors, quite a few of them have the
following stack trace - which mentions "fillInStackTrace" - it seems like kryo
is going wrong to me.
java.lang.Throwable.fillInStackTrace(Native Method)
java.lang.Throwable.fillInStackTrace(Throwable.java:783)
java.lang.Throwable.<init>(Throwable.java:265)
java.lang.Exception.<init>(Exception.java:66)
java.lang.ReflectiveOperationException.<init>(ReflectiveOperationException.java:56)
java.lang.NoSuchMethodException.<init>(NoSuchMethodException.java:51)
java.lang.Class.getConstructor0(Class.java:3082)
java.lang.Class.getConstructor(Class.java:1825)
com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:60)
com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:359)
com.esotericsoftware.kryo.Kryo.register(Kryo.java:394)
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$1.apply(KryoSerializer.scala:97)
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$1.apply(KryoSerializer.scala:96)
scala.collection.immutable.List.foreach(List.scala:381)
org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:96)
org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:274)
org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:259)
org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:175)
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:59)
org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)
(the last 3 lines above repeat a number of times in the trace so I truncated
them)
The main thread is clearly locked:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:202)
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:153)
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:623)
org.apache.spark.SparkContext.runJob(SparkContext.scala:1871)
> Calling unpersist can cause spark to hang indefinitely when writing out a
> result
> --------------------------------------------------------------------------------
>
> Key: SPARK-15002
> URL: https://issues.apache.org/jira/browse/SPARK-15002
> Project: Spark
> Issue Type: Bug
> Components: GraphX, Spark Core
> Affects Versions: 1.5.2, 1.6.0, 2.0.0
> Environment: AWS and Linux VM, both in spark-shell and spark-submit.
> tested in 1.5.2 and 1.6. Tested in 2.0
> Reporter: Jamie Hutton
>
> The following code will cause spark to hang indefinitely. It happens when you
> have an unpersist which is followed by some futher processing of that data
> (in my case writing it out).
> I first experienced this issue with graphX so my example below involved
> graphX code, however i suspect it might be more of a core issue than a graphx
> one. I have raised another bug with similar results (indefinite hanging) but
> in different circumstances, so these may well be linked
> Code to reproduce (can be run in spark-shell):
> import org.apache.spark.graphx._
> import org.apache.spark.rdd.RDD
> import org.apache.spark.sql.types._
> import org.apache.spark.sql._
> val r = scala.util.Random
> val list = (0L to 500L).map(i=>(i,r.nextInt(500).asInstanceOf[Long],"LABEL"))
> val distData = sc.parallelize(list)
> val edgesRDD = distData.map(x => Edge(x._1, x._2, x._3))
> val distinctNodes = distData.flatMap{row => Iterable((row._1, ("A")),(row._2,
> ("B")))}.distinct()
> val nodesRDD: RDD[(VertexId, (String))] = distinctNodes
> val graph = Graph(nodesRDD, edgesRDD)
> graph.persist()
> val ccGraph = graph.connectedComponents()
> ccGraph.cache
>
> val schema = StructType(Seq(StructField("id", LongType, false),
> StructField("netid", LongType, false)))
> val
> rdd=ccGraph.vertices.map(row=>(Row(row._1.asInstanceOf[Long],row._2.asInstanceOf[Long])))
> val builtEntityDF = sqlContext.createDataFrame(rdd, schema)
>
> /*this unpersist step causes the issue*/
> ccGraph.unpersist()
>
> /*write step hangs for ever*/
> builtEntityDF.write.format("parquet").mode("overwrite").save("/user/root/writetest.parquet")
>
> If you take out the ccGraph.unpersist() step the write step completes
> successfully
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
