I added https://issues.apache.org/jira/browse/SPARK-2103 to track this. I also ran into it. I don't have a fix, but, somehow I think someone with more understanding of Scala and Manifest objects might see the easy fix.
On Tue, Jun 10, 2014 at 5:15 PM, mpieck <mpi...@gazeta.pl> wrote: > Hi, > > I have the same problem when running Kafka to Spark Streaming pipeline from > Java with explicitely specified message decoders. I had thought, that it was > related to Eclipse environment, as suggested here, but it's not the case. I > have coded an example based on class: > > https://github.com/apache/spark/blob/branch-0.9/examples/src/main/java/org/apache/spark/streaming/examples/JavaKafkaWordCount.java > > and have builded shaded uber jar with all the deps and tried to run it from > command line. When I use the createStream method from the example class like > this: > > KafkaUtils.createStream(jssc, "zookeeper:port", "test", topicMap); > > everything is working fine, but when I explicitely specify message decoder > classes used in this method with another overloaded createStream method: > > KafkaUtils.createStream(jssc, String.class, String.class, > StringDecoder.class, StringDecoder.class, props, topicMap, > StorageLevels.MEMORY_AND_DISK_2); > > the applications stops with an error: > > 14/06/10 22:28:06 ERROR kafka.KafkaReceiver: Error receiving data > java.lang.NoSuchMethodException: > java.lang.Object.<init>(kafka.utils.VerifiableProperties) > at java.lang.Class.getConstructor0(Unknown Source) > at java.lang.Class.getConstructor(Unknown Source) > at > org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:108) > at > org.apache.spark.streaming.dstream.NetworkReceiver.start(NetworkInputDStream.scala:126) > > I have tried Spark versions 0.9.0-incubating, 0.9.0 and 0.9.1, but the error > occurs everywhere. Kafka StringDecoder class has the constructor with > VerifiableProperties parameter and all required classes are in the same uber > jar, so it is strange that scala/java cannot find it with reflection api. > Maybe there is some problem with Manifest/ClassTag usage in KafkaUtils or > KafkaInputDStream classes, but I'm not a Scala expert and cannot be sure > about it. The problematic code is the same from version 0.9 to the current > one, so it's still there. Unit test from the Spark project is working fine > with every KafkaUtils method, because the test does not try to register the > kafka stream, only checks the interface. > > Currently it is possible to use Kafka to Spark Streaming pipeline from Java > only with the default String message decoders, which makes this tool almost > useless (unless you are a great JSON fan). > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-in-KafkaReciever-tp2209p7347.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
