Daniel Darabos created SPARK-2403:
-------------------------------------
Summary: Spark stuck when class is not registered with Kryo
Key: SPARK-2403
URL: https://issues.apache.org/jira/browse/SPARK-2403
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0
Reporter: Daniel Darabos
We are using Kryo and require registering classes. When trying to serialize
something containing an unregistered class, Kryo will raise an exception.
DAGScheduler.submitMissingTasks runs in the scheduler thread and checks if the
contents of the task can be serialized by trying to serialize it:
https://github.com/apache/spark/blob/v1.0.0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L767
It catches NotSerializableException and aborts the task with an error when this
happens.
The problem is, Kryo does not raise NotSerializableException for unregistered
classes. It raises IllegalArgumentException instead. This exception is not
caught and kills the scheduler thread. The application then hangs, waiting
indefinitely for the job to finish.
Catching IllegalArgumentException also is a quick fix. I'll send a pull request
for it if you agree. Thanks!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
