Just remove "provided" for spark-streaming-kinesis-asl libraryDependencies += "org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.3.0"
On Fri, Apr 3, 2015 at 12:45 PM, Vadim Bichutskiy < vadim.bichuts...@gmail.com> wrote: > Thanks. So how do I fix it? > ᐧ > > On Fri, Apr 3, 2015 at 3:43 PM, Kelly, Jonathan <jonat...@amazon.com> > wrote: > >> spark-streaming-kinesis-asl is not part of the Spark distribution on >> your cluster, so you cannot have it be just a "provided" dependency. This >> is also why the KCL and its dependencies were not included in the assembly >> (but yes, they should be). >> >> >> ~ Jonathan Kelly >> >> From: Vadim Bichutskiy <vadim.bichuts...@gmail.com> >> Date: Friday, April 3, 2015 at 12:26 PM >> To: Jonathan Kelly <jonat...@amazon.com> >> Cc: "user@spark.apache.org" <user@spark.apache.org> >> Subject: Re: Spark + Kinesis >> >> Hi all, >> >> Good news! I was able to create a Kinesis consumer and assemble it into >> an "uber jar" following >> http://spark.apache.org/docs/latest/streaming-kinesis-integration.html >> and example >> https://github.com/apache/spark/blob/master/extras/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala >> . >> >> However when I try to spark-submit it I get the following exception: >> >> *Exception in thread "main" java.lang.NoClassDefFoundError: >> com/amazonaws/auth/AWSCredentialsProvider* >> >> Do I need to include KCL dependency in *build.sbt*, here's what it >> looks like currently: >> >> import AssemblyKeys._ >> name := "Kinesis Consumer" >> version := "1.0" >> organization := "com.myconsumer" >> scalaVersion := "2.11.5" >> >> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % >> "provided" >> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.3.0" >> % "provided" >> libraryDependencies += "org.apache.spark" %% >> "spark-streaming-kinesis-asl" % "1.3.0" % "provided" >> >> assemblySettings >> jarName in assembly := "consumer-assembly.jar" >> assemblyOption in assembly := (assemblyOption in >> assembly).value.copy(includeScala=false) >> >> Any help appreciated. >> >> Thanks, >> Vadim >> >> On Thu, Apr 2, 2015 at 1:15 PM, Kelly, Jonathan <jonat...@amazon.com> >> wrote: >> >>> It looks like you're attempting to mix Scala versions, so that's going >>> to cause some problems. If you really want to use Scala 2.11.5, you must >>> also use Spark package versions built for Scala 2.11 rather than 2.10. >>> Anyway, that's not quite the correct way to specify Scala dependencies in >>> build.sbt. Instead of placing the Scala version after the artifactId (like >>> "spark-core_2.10"), what you actually want is to use just "spark-core" with >>> two percent signs before it. Using two percent signs will make it use the >>> version of Scala that matches your declared scalaVersion. For example: >>> >>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % >>> "provided" >>> >>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % >>> "1.3.0" % "provided" >>> >>> libraryDependencies += "org.apache.spark" %% >>> "spark-streaming-kinesis-asl" % "1.3.0" >>> >>> I think that may get you a little closer, though I think you're >>> probably going to run into the same problems I ran into in this thread: >>> https://www.mail-archive.com/user@spark.apache.org/msg23891.html I >>> never really got an answer for that, and I temporarily moved on to other >>> things for now. >>> >>> >>> ~ Jonathan Kelly >>> >>> From: 'Vadim Bichutskiy' <vadim.bichuts...@gmail.com> >>> Date: Thursday, April 2, 2015 at 9:53 AM >>> To: "user@spark.apache.org" <user@spark.apache.org> >>> Subject: Spark + Kinesis >>> >>> Hi all, >>> >>> I am trying to write an Amazon Kinesis consumer Scala app that >>> processes data in the >>> Kinesis stream. Is this the correct way to specify *build.sbt*: >>> >>> ------- >>> *import AssemblyKeys._* >>> *name := "Kinesis Consumer"* >>> >>> >>> >>> >>> >>> >>> *version := "1.0" organization := "com.myconsumer" scalaVersion := >>> "2.11.5" libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" >>> % "1.3.0" % "provided", "org.apache.spark" % "spark-streaming_2.10" % >>> "1.3.0" "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.3.0")* >>> >>> >>> >>> * assemblySettings jarName in assembly := "consumer-assembly.jar" >>> assemblyOption in assembly := (assemblyOption in >>> assembly).value.copy(includeScala=false)* >>> -------- >>> >>> In *project/assembly.sbt* I have only the following line: >>> >>> *addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")* >>> >>> I am using sbt 0.13.7. I adapted Example 7.7 in the Learning Spark >>> book. >>> >>> Thanks, >>> Vadim >>> >>> >> >
