[
https://issues.apache.org/jira/browse/SPARK-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979274#comment-14979274
]
Christopher Childs commented on SPARK-11193:
--------------------------------------------
Fabien's comment led me to writing up a minimal test case. The
ClassCastException points at Kryo writing out the mutable.HashMap and
re-instantiating it without the SynchronizedMap trait. Twitter's Chill library
includes default HashMap/Map serializers and using that produces the erroneous
behavior seen above.
https://gist.github.com/cjchilds/4015a0d8e02868646a33
I didn't confirm if it was the Chill mutable.HashMap serializer specifically or
the mutable.Map subclass one. If you aren't bothering with serializing
mutable.Map or mutable.HashMaps you can probably alter your custom Kryo
serialization registration (see Spark's KryoRegistrator trait) to not include
those.
> Spark 1.5+ Kinesis Streaming - ClassCastException when starting
> KinesisReceiver
> -------------------------------------------------------------------------------
>
> Key: SPARK-11193
> URL: https://issues.apache.org/jira/browse/SPARK-11193
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.5.0, 1.5.1
> Reporter: Phil Kallos
> Attachments: screen.png
>
>
> After upgrading from Spark 1.4.x -> 1.5.x, I am now unable to start a Kinesis
> Spark Streaming application, and am being consistently greeted with this
> exception:
> java.lang.ClassCastException: scala.collection.mutable.HashMap cannot be cast
> to scala.collection.mutable.SynchronizedMap
> at
> org.apache.spark.streaming.kinesis.KinesisReceiver.onStart(KinesisReceiver.scala:175)
> at
> org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148)
> at
> org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:130)
> at
> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:542)
> at
> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:532)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:1982)
> at
> org.apache.spark.SparkContext$$anonfun$38.apply(SparkContext.scala:1982)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Worth noting that I am able to reproduce this issue locally, and also on
> Amazon EMR (using the latest emr-release 4.1.0 which packages Spark 1.5.0).
> Also, I am not able to run the included kinesis-asl example.
> Built locally using:
> git checkout v1.5.1
> mvn -Pyarn -Pkinesis-asl -Phadoop-2.6 -DskipTests clean package
> Example run command:
> bin/run-example streaming.KinesisWordCountASL phibit-test kinesis-connector
> https://kinesis.us-east-1.amazonaws.com
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
