Thanks Akhil. I tried spark-submit and saw the same issue. I double checked
the versions and they look ok. Are you seeing any obvious issues?
sbt:
name := "Simple Project"
version := "1.1"
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.1.0",
"org.apache.spark" %% "spark-streaming" % "1.1.0",
"org.apache.spark" %% "spark-streaming-kafka" % "1.1.0",
"org.apache.kafka" %% "kafka" % "0.8.0"
)
spark-1.1.0-bin-hadoop1/bin/spark-submit --class "main.scala.SimpleApp"
--master "local[2]" simple-project_2.10-1.1.jar --jars
spark-streaming-kafka_2.10-1.1.0.jar,kafka_2.10-0.8.0.jar
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/streaming/kafka/KafkaUtils$
at main.scala.SimpleApp$delayedInit$body.apply(SimpleApp.scala:15)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at main.scala.SimpleApp$.main(SimpleApp.scala:11)
at main.scala.SimpleApp.main(SimpleApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.streaming.kafka.KafkaUtils$
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 17 more
On Tue, Oct 14, 2014 at 12:05 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Just make sure you have the same version of spark-streaming-kafka
> <http://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka_2.10>
> jar and spark in your classpath.
>
> Thanks
> Best Regards
>
> On Tue, Oct 14, 2014 at 9:02 AM, Gary Zhao <garyz...@gmail.com> wrote:
>
>> Hello
>>
>> I'm trying to connect kafka in spark shell, but failed as below. Could
>> you take a look what I missed.
>>
>> scala> val kafkaStream = KafkaUtils.createStream(ssc,
>> "test-vip.snc1:2181", "test_spark", Map("user-test"->1))
>> error: bad symbolic reference. A signature in KafkaUtils.class refers to
>> term serializer
>> in value kafka which is not available.
>> It may be completely missing from the current classpath, or the version on
>> the classpath might be incompatible with the version used when compiling
>> KafkaUtils.class.
>>
>> Thanks
>> Gary
>>
>
>
