[ https://issues.apache.org/jira/browse/SPARK-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054907#comment-14054907 ]
Daniel Darabos edited comment on SPARK-2403 at 7/8/14 12:59 PM: ---------------------------------------------------------------- I think DAGSchedulerActorSupervisor is supposed to kill the system when things like this happen. I am not sure why that does not happen in this case. I forgot to include the stack trace: {noformat} java.lang.IllegalArgumentException: Class is not registered: scala.collection.immutable.Range Note: To register this class use: kryo.register(scala.collection.immutable.Range.class); at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442) ~[kryo-2.21.jar:na] at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79) ~[kryo-2.21.jar:na] at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472) ~[kryo-2.21.jar:na] at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565) ~[kryo-2.21.jar:na] at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:101) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.rdd.ParallelCollectionPartition$$anonfun$writeObject$1.apply(ParallelCollectionRDD.scala:65) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.rdd.ParallelCollectionPartition$$anonfun$writeObject$1.apply(ParallelCollectionRDD.scala:65) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.util.Utils$.serializeViaNestedStream(Utils.scala:105) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.rdd.ParallelCollectionPartition.writeObject(ParallelCollectionRDD.scala:65) ~[spark-core_2.10-1.0.0.jar:1.0.0] at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_55] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55] at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) ~[na:1.7.0_55] at org.apache.spark.scheduler.ShuffleMapTask.writeExternal(ShuffleMapTask.scala:126) ~[spark-core_2.10-1.0.0.jar:1.0.0] at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) ~[na:1.7.0_55] at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:71) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:767) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:713) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:717) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:716) ~[spark-core_2.10-1.0.0.jar:1.0.0] at scala.collection.immutable.List.foreach(List.scala:318) ~[scala-library-2.10.4.jar:na] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:716) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:697) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1176) ~[spark-core_2.10-1.0.0.jar:1.0.0] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) [akka-actor_2.10-2.2.3.jar:2.2.3] at akka.actor.ActorCell.invoke(ActorCell.scala:456) [akka-actor_2.10-2.2.3.jar:2.2.3] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) [akka-actor_2.10-2.2.3.jar:2.2.3] at akka.dispatch.Mailbox.run(Mailbox.scala:219) [akka-actor_2.10-2.2.3.jar:2.2.3] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [akka-actor_2.10-2.2.3.jar:2.2.3] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.10.4.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.10.4.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.10.4.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.10.4.jar:na] {noformat} was (Author: darabos): I think DAGSchedulerActorSupervisor is supposed to kill the system when things like this happen. I am not sure why that does not happen in this case. I forgot to include the stack trace: java.lang.IllegalArgumentException: Class is not registered: scala.collection.immutable.Range Note: To register this class use: kryo.register(scala.collection.immutable.Range.class); at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442) ~[kryo-2.21.jar:na] at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79) ~[kryo-2.21.jar:na] at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472) ~[kryo-2.21.jar:na] at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565) ~[kryo-2.21.jar:na] at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:101) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.rdd.ParallelCollectionPartition$$anonfun$writeObject$1.apply(ParallelCollectionRDD.scala:65) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.rdd.ParallelCollectionPartition$$anonfun$writeObject$1.apply(ParallelCollectionRDD.scala:65) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.util.Utils$.serializeViaNestedStream(Utils.scala:105) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.rdd.ParallelCollectionPartition.writeObject(ParallelCollectionRDD.scala:65) ~[spark-core_2.10-1.0.0.jar:1.0.0] at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_55] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55] at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) ~[na:1.7.0_55] at org.apache.spark.scheduler.ShuffleMapTask.writeExternal(ShuffleMapTask.scala:126) ~[spark-core_2.10-1.0.0.jar:1.0.0] at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1458) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1429) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) ~[na:1.7.0_55] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) ~[na:1.7.0_55] at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:71) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:767) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:713) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:717) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:716) ~[spark-core_2.10-1.0.0.jar:1.0.0] at scala.collection.immutable.List.foreach(List.scala:318) ~[scala-library-2.10.4.jar:na] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:716) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:697) ~[spark-core_2.10-1.0.0.jar:1.0.0] at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1176) ~[spark-core_2.10-1.0.0.jar:1.0.0] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) [akka-actor_2.10-2.2.3.jar:2.2.3] at akka.actor.ActorCell.invoke(ActorCell.scala:456) [akka-actor_2.10-2.2.3.jar:2.2.3] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) [akka-actor_2.10-2.2.3.jar:2.2.3] at akka.dispatch.Mailbox.run(Mailbox.scala:219) [akka-actor_2.10-2.2.3.jar:2.2.3] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [akka-actor_2.10-2.2.3.jar:2.2.3] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.10.4.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.10.4.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.10.4.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.10.4.jar:na] > Spark stuck when class is not registered with Kryo > -------------------------------------------------- > > Key: SPARK-2403 > URL: https://issues.apache.org/jira/browse/SPARK-2403 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.0.0 > Reporter: Daniel Darabos > > We are using Kryo and require registering classes. When trying to serialize > something containing an unregistered class, Kryo will raise an exception. > DAGScheduler.submitMissingTasks runs in the scheduler thread and checks if > the contents of the task can be serialized by trying to serialize it: > https://github.com/apache/spark/blob/v1.0.0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L767 > It catches NotSerializableException and aborts the task with an error when > this happens. > The problem is, Kryo does not raise NotSerializableException for unregistered > classes. It raises IllegalArgumentException instead. This exception is not > caught and kills the scheduler thread. The application then hangs, waiting > indefinitely for the job to finish. > Catching IllegalArgumentException also is a quick fix. I'll send a pull > request for it if you agree. Thanks! -- This message was sent by Atlassian JIRA (v6.2#6252)