[
https://issues.apache.org/jira/browse/SPARK-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Davidson resolved SPARK-2403.
-----------------------------------
Resolution: Fixed
Fix Version/s: 1.0.2
1.1.0
> Spark stuck when class is not registered with Kryo
> --------------------------------------------------
>
> Key: SPARK-2403
> URL: https://issues.apache.org/jira/browse/SPARK-2403
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Daniel Darabos
> Fix For: 1.1.0, 1.0.2
>
>
> We are using Kryo and require registering classes. When trying to serialize
> something containing an unregistered class, Kryo will raise an exception.
> DAGScheduler.submitMissingTasks runs in the scheduler thread and checks if
> the contents of the task can be serialized by trying to serialize it:
> https://github.com/apache/spark/blob/v1.0.0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L767
> It catches NotSerializableException and aborts the task with an error when
> this happens.
> The problem is, Kryo does not raise NotSerializableException for unregistered
> classes. It raises IllegalArgumentException instead. This exception is not
> caught and kills the scheduler thread. The application then hangs, waiting
> indefinitely for the job to finish.
> Catching IllegalArgumentException also is a quick fix. I'll send a pull
> request for it if you agree. Thanks!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
