Ok, that build file I linked earlier has a minimal example of use. just running 'sbt assembly' given a similar build file should build a jar with all the dependencies.
On Wed, Feb 24, 2016 at 1:50 PM, Vinti Maheshwari <vinti.u...@gmail.com> wrote: > I am not using sbt assembly currently. I need to check how to use sbt > assembly. > > Regards, > ~Vinti > > On Wed, Feb 24, 2016 at 11:10 AM, Cody Koeninger <c...@koeninger.org> > wrote: > >> Are you using sbt assembly? That's what will include all of the >> non-provided dependencies in a single jar along with your code. Otherwise >> you'd have to specify each separate jar in your spark-submit line, which is >> a pain. >> >> On Wed, Feb 24, 2016 at 12:49 PM, Vinti Maheshwari <vinti.u...@gmail.com> >> wrote: >> >>> Hi Cody, >>> >>> I tried with the build file you provided, but it's not working for me, >>> getting same error: >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/apache/spark/streaming/kafka/KafkaUtils$ >>> >>> I am not getting this error while building (sbt package). I am getting >>> this error when i am running my spark-streaming program. >>> Do i need to specify kafka jar path manually with spark-submit --jars >>> flag? >>> >>> My build.sbt: >>> >>> name := "NetworkStreaming" >>> libraryDependencies += "org.apache.hbase" % "hbase" % "0.92.1" >>> >>> libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.0.2" >>> >>> libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.0.0" >>> >>> libraryDependencies ++= Seq( >>> "org.apache.spark" % "spark-streaming_2.10" % "1.5.2", >>> "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2" >>> ) >>> >>> >>> >>> Regards, >>> ~Vinti >>> >>> On Wed, Feb 24, 2016 at 9:33 AM, Cody Koeninger <c...@koeninger.org> >>> wrote: >>> >>>> spark streaming is provided, kafka is not. >>>> >>>> This build file >>>> >>>> https://github.com/koeninger/kafka-exactly-once/blob/master/build.sbt >>>> >>>> includes some hacks for ivy issues that may no longer be strictly >>>> necessary, but try that build and see if it works for you. >>>> >>>> >>>> On Wed, Feb 24, 2016 at 11:14 AM, Vinti Maheshwari < >>>> vinti.u...@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> I have tried multiple different settings in build.sbt but seems like >>>>> nothing is working. >>>>> Can anyone suggest the right syntax/way to include kafka with spark? >>>>> >>>>> Error >>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>> org/apache/spark/streaming/kafka/KafkaUtils$ >>>>> >>>>> build.sbt >>>>> libraryDependencies += "org.apache.hbase" % "hbase" % "0.92.1" >>>>> libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "1.0.2" >>>>> libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % >>>>> "1.0.0" >>>>> libraryDependencies ++= Seq( >>>>> "org.apache.spark" % "spark-streaming_2.10" % "1.5.2", >>>>> "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2", >>>>> "org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided", >>>>> "org.apache.spark" %% "spark-streaming-kafka" % "1.5.2" % "provided" >>>>> ) >>>>> >>>>> >>>>> Thanks, >>>>> Vinti >>>>> >>>>> >>>> >>> >> >
