Are you using sbt assembly? That's what will include all of the non-provided dependencies in a single jar along with your code. Otherwise you'd have to specify each separate jar in your spark-submit line, which is a pain.
On Wed, Feb 24, 2016 at 12:49 PM, Vinti Maheshwari <vinti.u...@gmail.com> wrote: > Hi Cody, > > I tried with the build file you provided, but it's not working for me, > getting same error: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/spark/streaming/kafka/KafkaUtils$ > > I am not getting this error while building (sbt package). I am getting > this error when i am running my spark-streaming program. > Do i need to specify kafka jar path manually with spark-submit --jars flag? > > My build.sbt: > > name := "NetworkStreaming" > libraryDependencies += "org.apache.hbase" % "hbase" % "0.92.1" > > libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.0.2" > > libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.0.0" > > libraryDependencies ++= Seq( > "org.apache.spark" % "spark-streaming_2.10" % "1.5.2", > "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2" > ) > > > > Regards, > ~Vinti > > On Wed, Feb 24, 2016 at 9:33 AM, Cody Koeninger <c...@koeninger.org> > wrote: > >> spark streaming is provided, kafka is not. >> >> This build file >> >> https://github.com/koeninger/kafka-exactly-once/blob/master/build.sbt >> >> includes some hacks for ivy issues that may no longer be strictly >> necessary, but try that build and see if it works for you. >> >> >> On Wed, Feb 24, 2016 at 11:14 AM, Vinti Maheshwari <vinti.u...@gmail.com> >> wrote: >> >>> Hello, >>> >>> I have tried multiple different settings in build.sbt but seems like >>> nothing is working. >>> Can anyone suggest the right syntax/way to include kafka with spark? >>> >>> Error >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/apache/spark/streaming/kafka/KafkaUtils$ >>> >>> build.sbt >>> libraryDependencies += "org.apache.hbase" % "hbase" % "0.92.1" >>> libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.0.2" >>> libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.0.0" >>> libraryDependencies ++= Seq( >>> "org.apache.spark" % "spark-streaming_2.10" % "1.5.2", >>> "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2", >>> "org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided", >>> "org.apache.spark" %% "spark-streaming-kafka" % "1.5.2" % "provided" >>> ) >>> >>> >>> Thanks, >>> Vinti >>> >>> >> >
