Bartlomiej Alberski created SPARK-12407:
-------------------------------------------
Summary: ClassCast Exception when restarting spark streaming from
checkpoint
Key: SPARK-12407
URL: https://issues.apache.org/jira/browse/SPARK-12407
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.5.1
Reporter: Bartlomiej Alberski
I am receiving ClassCast Exception when restarting streaming application from
checkpoint:
{code}
java.lang.ClassCastException: [B cannot be cast to
pl.example.spark.StreamingTestReporter
at
pl.example.spark.StreamingTest$$anonfun$createContext$1$$anonfun$apply$2$$anonfun$apply$4$$anonfun$apply$5.apply(StreamingTest.scala:38)
at
pl.example.spark.StreamingTest$$anonfun$createContext$1$$anonfun$apply$2$$anonfun$apply$4$$anonfun$apply$5.apply(StreamingTest.scala:36)
at scala.collection.immutable.List.foreach(List.scala:318)
at
pl.example.spark.StreamingTest$$anonfun$createContext$1$$anonfun$apply$2$$anonfun$apply$4.apply(StreamingTest.scala:36)
at
pl.example.spark.StreamingTest$$anonfun$createContext$1$$anonfun$apply$2$$anonfun$apply$4.apply(StreamingTest.scala:33)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:898)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:898)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1848)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1848)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
It looks like the problem is connected with instance of class broadcasted to
the executors. I think that when restoring from checkpoint Id of broadcasted
value changes - this is the reason why read from broadcasted memory block
something else (instead of instance of our class). In my production code I
received sligthly different exception:
{code}
java.lang.ClassCastException: org.apache.spark.util.SerializableConfiguration
cannot be cast to com.example.sender.MyClassReporter
{code}
Below there are links for spark user list with discussion about issue as well
as minimal example that helps in reproducing issue.
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-streaming-java-lang-ClassCastException-org-apache-spark-util-SerializableConfiguration-on-restt-td25698.html
https://github.com/alberskib/spark-streaming-broadcast-issue
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
