I'd like to resurrect this thread: http://mail-archives.apache.org/mod_mbox/spark-user/201403.mbox/%3c6d657d19-1ecf-4e92-bf15-cc4762ef9...@thekratos.com%3E
Basically when you call this particular Java-flavored overloading of KafkaUtils.createStream: https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala#L133 ... you get java.lang.NoSuchMethodException: java.lang.Object.<init>(kafka.utils.VerifiableProperties) at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getConstructor(Class.java:1693) at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108) This doesn't appear to be a version issue. It doesn't appear when calling other versions of this method. Other overloadings work (well, have other issues). Something is making it try to instantiate java.lang.Object as if it's a Decoder class. I am wondering about this code at https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala#L148 implicit val keyCmd: Manifest[U] = implicitly[Manifest[AnyRef]].asInstanceOf[Manifest[U]] implicit val valueCmd: Manifest[T] = implicitly[Manifest[AnyRef]].asInstanceOf[Manifest[T]] ... where U and T are key/value Decoder types. I don't know enough Scala to fully understand this, but is it possible this causes the reflective call later to lose the type and try to instantiate Object? The AnyRef made me wonder. @tdas I'm hoping you might have some insight as it came in this commit in January: https://github.com/apache/spark/commit/aa99f226a691ddcb4442d60f4cd4908f434cc4ce I'll file a JIRA if it's legitimate; just asking first.