Actually, i've 2 versions of Kafka (0.9 & 0.10) ..
btw, i was able to resolve the issue ..
sbt by default considers src/main/scala as default source location,
I'd changed the location to a different one.
I changed the build.sbt to point to the required location, that fixed the
issue
regds,
Karan Alang
On Mon, Jul 10, 2017 at 11:48 AM, David Newberger wrote:
> Karen,
>
> It looks like the Kafka version is incorrect. You mention Kafka 0.10
> however the classpath references Kafka 0.9
>
> Thanks,
>
> David
>
> On July 10, 2017 at 1:44:06 PM, karan alang (karan.al...@gmail.com) wrote:
>
> Hi All,
>
> I'm running Spark Streaming - Kafka integration using Spark 2.x & Kafka 10.
> & seems to be running into issues.
>
> I compiled the program using sbt, and the compilation went through fine.
> I was able able to import this into Eclipse & run the program from Eclipse.
>
> However, when i run the program using spark-submit, i'm getting the
> following error :
>
> --
>
>> $SPARK_HOME/bin/spark-submit --class
>> "structuredStreaming.kafka.StructuredKafkaWordCount1"
>> --master local[2] /Users/karanalang/Documents/Te
>> chnology/Coursera_spark_scala/structuredStreamingKafka/targe
>> t/scala-2.11/StructuredStreamingKafka-assembly-1.0.jar
>
>
>
>> *java.lang.ClassNotFoundException:
>> structuredStreaming.kafka.StructuredKafkaWordCount1*
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:348)
>> at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
>> $SparkSubmit$$runMain(SparkSubmit.scala:695)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit
>> .scala:187)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>
> --
>
> I've put the jar in the classpath, but i still get the error ->
>
> echo $CLASSPATH
>
> .:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.
>> 9.0.1/lib/jopt-simple-3.2.jar:/Users/karanalang/Documents/Te
>> chnology/kafka/kafka_2.11-0.9.0.1/lib/kafka-clients-0.9.0.1.
>> jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-
>> 0.9.0.1/lib/kafka_2.11-0.9.0.1.jar:/Users/karanalang/Documen
>> ts/Technology/kafka/kafka_2.11-0.9.0.1/lib/log4j-1.2.17.
>> jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-
>> 0.9.0.1/lib/metrics-core-2.2.0.jar:/Users/karanalang/
>> Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/scala-
>> library-2.11.7.jar:/Users/karanalang/Documents/
>> Technology/kafka/kafka_2.11-0.9.0.1/lib/slf4j-api-1.7.6.jar:
>> /Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.
>> 9.0.1/lib/slf4j-log4j12-1.7.6.jar:/Users/karanalang/
>> Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/snappy-
>> java-1.1.1.7.jar:/Users/karanalang/Documents/
>> Technology/kafka/kafka_2.11-0.9.0.1/lib/zkclient-0.7.jar:/
>> Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.
>> 9.0.1/lib/zookeeper-3.4.6.jar:/Users/karanalang/Documents/
>> Technology/ApacheSpark-v2.1/spark-2.1.0-bin-hadoop2.7/
>> jars/*.jar:/Users/karanalang/Documents/Technology/kafka/
>> mirrormaker_topic_rename-master/target/mmchangetopic-1.
>> 0-SNAPSHOT.jar:/Users/karanalang/Documents/Technology/
>> *Coursera_spark_scala/structuredStreamingKafka/target/scala-2.11/*
>> *StructuredStreamingKafka-assembly-1.0.jar*
>
>
> When i look inside the jar - *StructuredStreamingKafka-assembly-1.0.jar,
> i don't see the file "*StructuredKafkaWordCount1.class"
>
> Attaching my build.sbt.
>
> Any ideas on what i need to do ?
>
>
>
>
>
>
>
>
>
>
>
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
