Easiest fix would be adding the kafka jars to the SparkContext while creating it.
Thanks Best Regards On Fri, Jul 11, 2014 at 4:39 AM, Dilip <dilip_ram...@hotmail.com> wrote: > Hi, > > I am trying to run a program with spark streaming using Kafka on a stand > alone system. These are my details: > > Spark 1.0.0 hadoop2 > Scala 2.10.3 > > I am trying a simple program using my custom sbt project but this is the > error I am getting: > > Exception in thread "main" java.lang.NoClassDefFoundError: > kafka/serializer/StringDecoder > at org.apache.spark.streaming.kafka.KafkaUtils$. > createStream(KafkaUtils.scala:55) > at org.apache.spark.streaming.kafka.KafkaUtils$. > createStream(KafkaUtils.scala:94) > at org.apache.spark.streaming.kafka.KafkaUtils.createStream( > KafkaUtils.scala) > at SimpleJavaApp.main(SimpleJavaApp.java:40) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.ClassNotFoundException: > kafka.serializer.StringDecoder > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > ... 11 more > > > here is my .sbt file: > > name := "Simple Project" > > version := "1.0" > > scalaVersion := "2.10.3" > > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.0.0" > > libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.0.0" > > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" > > libraryDependencies += "org.apache.spark" %% "spark-examples" % "1.0.0" > > libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % > "1.0.0" > > libraryDependencies += "org.apache.kafka" %% "kafka" % "0.8.0" > > resolvers += "Akka Repository" at "http://repo.akka.io/releases/"; > > resolvers += "Maven Repository" at "http://central.maven.org/maven2/"; > > > sbt package was successful. I also tried sbt "++2.10.3 package" to build > it for my scala version. Problem remains the same. > Can anyone help me out here? Ive been stuck on this for quite some time > now. > > Thank You, > Dilip >
