Re: Spark-Kafka integration - build failing with sbt
org.apache.spark.streaming.kafka.KafkaUtils is in the spark-streaming-kafka-0-8 project On Mon, Jun 19, 2017 at 1:01 PM, karan alang wrote: > Hi Cody - i do have a additional basic question .. > > When i tried to compile the code in Eclipse, i was not able to do that > > eg. > import org.apache.spark.streaming.kafka.KafkaUtils > > gave errors saying KafaUtils was not part of the package. > However, when i used sbt to compile - the compilation went through fine > > So, I assume additional libraries are being downloaded when i provide the > appropriate packages in LibraryDependencies ? > which ones would have helped compile this ? > > > > On Sat, Jun 17, 2017 at 2:53 PM, karan alang wrote: >> >> Thanks, Cody .. yes, was able to fix that. >> >> On Sat, Jun 17, 2017 at 1:18 PM, Cody Koeninger >> wrote: >>> >>> There are different projects for different versions of kafka, >>> spark-streaming-kafka-0-8 and spark-streaming-kafka-0-10 >>> >>> See >>> >>> http://spark.apache.org/docs/latest/streaming-kafka-integration.html >>> >>> On Fri, Jun 16, 2017 at 6:51 PM, karan alang >>> wrote: >>> > I'm trying to compile kafka & Spark Streaming integration code i.e. >>> > reading >>> > from Kafka using Spark Streaming, >>> > and the sbt build is failing with error - >>> > >>> > [error] (*:update) sbt.ResolveException: unresolved dependency: >>> > org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>> > >>> > Scala version -> 2.10.7 >>> > Spark Version -> 2.1.0 >>> > Kafka version -> 0.9 >>> > sbt version -> 0.13 >>> > >>> > Contents of sbt files is as shown below -> >>> > >>> > 1) >>> > vi spark_kafka_code/project/plugins.sbt >>> > >>> > addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2") >>> > >>> > 2) >>> > vi spark_kafka_code/sparkkafka.sbt >>> > >>> > import AssemblyKeys._ >>> > assemblySettings >>> > >>> > name := "SparkKafka Project" >>> > >>> > version := "1.0" >>> > scalaVersion := "2.11.7" >>> > >>> > val sparkVers = "2.1.0" >>> > >>> > // Base Spark-provided dependencies >>> > libraryDependencies ++= Seq( >>> > "org.apache.spark" %% "spark-core" % sparkVers % "provided", >>> > "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", >>> > "org.apache.spark" %% "spark-streaming-kafka" % sparkVers) >>> > >>> > mergeStrategy in assembly := { >>> > case m if m.toLowerCase.endsWith("manifest.mf") => >>> > MergeStrategy.discard >>> > case m if m.toLowerCase.startsWith("META-INF") => >>> > MergeStrategy.discard >>> > case "reference.conf" => >>> > MergeStrategy.concat >>> > case m if m.endsWith("UnusedStubClass.class") => >>> > MergeStrategy.discard >>> > case _ => MergeStrategy.first >>> > } >>> > >>> > i launch sbt, and then try to create an eclipse project, complete >>> > error is >>> > as shown below - >>> > >>> > - >>> > >>> > sbt >>> > [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins >>> > [info] Loading project definition from >>> > >>> > /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/project >>> > [info] Set current project to SparkKafka Project (in build >>> > >>> > file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/) >>> >> eclipse >>> > [info] About to create Eclipse project files for your project(s). >>> > [info] Updating >>> > >>> > {file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/}spark_kafka_code... >>> > [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... >>> > [warn] module not found: >>> > org.apache.spark#spark-streaming-kafka_2.11;2.1.0 >>> > [warn] local: tried >>> > [warn] >>> > >>> > /Users/karanalang/.ivy2/local/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>> > [warn] activator-launcher-local: tried >>> > [warn] >>> > >>> > /Users/karanalang/.activator/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>> > [warn] activator-local: tried >>> > [warn] >>> > >>> > /Users/karanalang/Documents/Technology/SCALA/activator-dist-1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>> > [warn] public: tried >>> > [warn] >>> > >>> > https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>> > [warn] typesafe-releases: tried >>> > [warn] >>> > >>> > http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>> > [warn] typesafe-ivy-releasez: tried >>> > [warn] >>> > >>> > http://repo.typesafe.com/typesafe/ivy-releases/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>> > [info] Resolving jline#jline;2.12.1 ... >>> > [warn] :: >>> > [warn] :: UNRESOLVED DEPENDENCIES :: >>> > [warn] :: >>>
Re: Spark-Kafka integration - build failing with sbt
Hi Cody - i do have a additional basic question .. When i tried to compile the code in Eclipse, i was not able to do that eg. import org.apache.spark.streaming.kafka.KafkaUtils gave errors saying KafaUtils was not part of the package. However, when i used sbt to compile - the compilation went through fine So, I assume additional libraries are being downloaded when i provide the appropriate packages in LibraryDependencies ? which ones would have helped compile this ? On Sat, Jun 17, 2017 at 2:53 PM, karan alang wrote: > Thanks, Cody .. yes, was able to fix that. > > On Sat, Jun 17, 2017 at 1:18 PM, Cody Koeninger > wrote: > >> There are different projects for different versions of kafka, >> spark-streaming-kafka-0-8 and spark-streaming-kafka-0-10 >> >> See >> >> http://spark.apache.org/docs/latest/streaming-kafka-integration.html >> >> On Fri, Jun 16, 2017 at 6:51 PM, karan alang >> wrote: >> > I'm trying to compile kafka & Spark Streaming integration code i.e. >> reading >> > from Kafka using Spark Streaming, >> > and the sbt build is failing with error - >> > >> > [error] (*:update) sbt.ResolveException: unresolved dependency: >> > org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >> > >> > Scala version -> 2.10.7 >> > Spark Version -> 2.1.0 >> > Kafka version -> 0.9 >> > sbt version -> 0.13 >> > >> > Contents of sbt files is as shown below -> >> > >> > 1) >> > vi spark_kafka_code/project/plugins.sbt >> > >> > addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2") >> > >> > 2) >> > vi spark_kafka_code/sparkkafka.sbt >> > >> > import AssemblyKeys._ >> > assemblySettings >> > >> > name := "SparkKafka Project" >> > >> > version := "1.0" >> > scalaVersion := "2.11.7" >> > >> > val sparkVers = "2.1.0" >> > >> > // Base Spark-provided dependencies >> > libraryDependencies ++= Seq( >> > "org.apache.spark" %% "spark-core" % sparkVers % "provided", >> > "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", >> > "org.apache.spark" %% "spark-streaming-kafka" % sparkVers) >> > >> > mergeStrategy in assembly := { >> > case m if m.toLowerCase.endsWith("manifest.mf") => >> MergeStrategy.discard >> > case m if m.toLowerCase.startsWith("META-INF") => >> MergeStrategy.discard >> > case "reference.conf" => >> MergeStrategy.concat >> > case m if m.endsWith("UnusedStubClass.class") => >> MergeStrategy.discard >> > case _ => MergeStrategy.first >> > } >> > >> > i launch sbt, and then try to create an eclipse project, complete >> error is >> > as shown below - >> > >> > - >> > >> > sbt >> > [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins >> > [info] Loading project definition from >> > /Users/karanalang/Documents/Technology/Coursera_spark_scala/ >> spark_kafka_code/project >> > [info] Set current project to SparkKafka Project (in build >> > file:/Users/karanalang/Documents/Technology/Coursera_spark_ >> scala/spark_kafka_code/) >> >> eclipse >> > [info] About to create Eclipse project files for your project(s). >> > [info] Updating >> > {file:/Users/karanalang/Documents/Technology/Coursera_spark_ >> scala/spark_kafka_code/}spark_kafka_code... >> > [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... >> > [warn] module not found: >> > org.apache.spark#spark-streaming-kafka_2.11;2.1.0 >> > [warn] local: tried >> > [warn] >> > /Users/karanalang/.ivy2/local/org.apache.spark/spark-streami >> ng-kafka_2.11/2.1.0/ivys/ivy.xml >> > [warn] activator-launcher-local: tried >> > [warn] >> > /Users/karanalang/.activator/repository/org.apache.spark/spa >> rk-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >> > [warn] activator-local: tried >> > [warn] >> > /Users/karanalang/Documents/Technology/SCALA/activator-dist- >> 1.3.10/repository/org.apache.spark/spark-streaming-kafka_2. >> 11/2.1.0/ivys/ivy.xml >> > [warn] public: tried >> > [warn] >> > https://repo1.maven.org/maven2/org/apache/spark/spark-stream >> ing-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >> > [warn] typesafe-releases: tried >> > [warn] >> > http://repo.typesafe.com/typesafe/releases/org/apache/spark/ >> spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >> > [warn] typesafe-ivy-releasez: tried >> > [warn] >> > http://repo.typesafe.com/typesafe/ivy-releases/org.apache. >> spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >> > [info] Resolving jline#jline;2.12.1 ... >> > [warn] :: >> > [warn] :: UNRESOLVED DEPENDENCIES :: >> > [warn] :: >> > [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not >> found >> > [warn] :: >> > [warn] >> > [warn] Note: Unresolved dependencies path: >> > [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 >> > (/Users/karanalang/Docu
Re: Spark-Kafka integration - build failing with sbt
Thanks, Cody .. yes, was able to fix that. On Sat, Jun 17, 2017 at 1:18 PM, Cody Koeninger wrote: > There are different projects for different versions of kafka, > spark-streaming-kafka-0-8 and spark-streaming-kafka-0-10 > > See > > http://spark.apache.org/docs/latest/streaming-kafka-integration.html > > On Fri, Jun 16, 2017 at 6:51 PM, karan alang > wrote: > > I'm trying to compile kafka & Spark Streaming integration code i.e. > reading > > from Kafka using Spark Streaming, > > and the sbt build is failing with error - > > > > [error] (*:update) sbt.ResolveException: unresolved dependency: > > org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > > > > Scala version -> 2.10.7 > > Spark Version -> 2.1.0 > > Kafka version -> 0.9 > > sbt version -> 0.13 > > > > Contents of sbt files is as shown below -> > > > > 1) > > vi spark_kafka_code/project/plugins.sbt > > > > addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2") > > > > 2) > > vi spark_kafka_code/sparkkafka.sbt > > > > import AssemblyKeys._ > > assemblySettings > > > > name := "SparkKafka Project" > > > > version := "1.0" > > scalaVersion := "2.11.7" > > > > val sparkVers = "2.1.0" > > > > // Base Spark-provided dependencies > > libraryDependencies ++= Seq( > > "org.apache.spark" %% "spark-core" % sparkVers % "provided", > > "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", > > "org.apache.spark" %% "spark-streaming-kafka" % sparkVers) > > > > mergeStrategy in assembly := { > > case m if m.toLowerCase.endsWith("manifest.mf") => > MergeStrategy.discard > > case m if m.toLowerCase.startsWith("META-INF") => > MergeStrategy.discard > > case "reference.conf" => MergeStrategy.concat > > case m if m.endsWith("UnusedStubClass.class") => > MergeStrategy.discard > > case _ => MergeStrategy.first > > } > > > > i launch sbt, and then try to create an eclipse project, complete > error is > > as shown below - > > > > - > > > > sbt > > [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins > > [info] Loading project definition from > > /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/project > > [info] Set current project to SparkKafka Project (in build > > file:/Users/karanalang/Documents/Technology/Coursera_ > spark_scala/spark_kafka_code/) > >> eclipse > > [info] About to create Eclipse project files for your project(s). > > [info] Updating > > {file:/Users/karanalang/Documents/Technology/Coursera_ > spark_scala/spark_kafka_code/}spark_kafka_code... > > [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... > > [warn] module not found: > > org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > > [warn] local: tried > > [warn] > > /Users/karanalang/.ivy2/local/org.apache.spark/spark- > streaming-kafka_2.11/2.1.0/ivys/ivy.xml > > [warn] activator-launcher-local: tried > > [warn] > > /Users/karanalang/.activator/repository/org.apache.spark/ > spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > > [warn] activator-local: tried > > [warn] > > /Users/karanalang/Documents/Technology/SCALA/activator- > dist-1.3.10/repository/org.apache.spark/spark-streaming- > kafka_2.11/2.1.0/ivys/ivy.xml > > [warn] public: tried > > [warn] > > https://repo1.maven.org/maven2/org/apache/spark/spark- > streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom > > [warn] typesafe-releases: tried > > [warn] > > http://repo.typesafe.com/typesafe/releases/org/apache/ > spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming- > kafka_2.11-2.1.0.pom > > [warn] typesafe-ivy-releasez: tried > > [warn] > > http://repo.typesafe.com/typesafe/ivy-releases/org. > apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > > [info] Resolving jline#jline;2.12.1 ... > > [warn] :: > > [warn] :: UNRESOLVED DEPENDENCIES :: > > [warn] :: > > [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not > found > > [warn] :: > > [warn] > > [warn] Note: Unresolved dependencies path: > > [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 > > (/Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/sparkkafka.sbt#L12-16) > > [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 > > [trace] Stack trace suppressed: run last *:update for the full output. > > [error] (*:update) sbt.ResolveException: unresolved dependency: > > org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > > [info] Updating > > {file:/Users/karanalang/Documents/Technology/Coursera_ > spark_scala/spark_kafka_code/}spark_kafka_code... > > [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... > > [warn] module not found: > > org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > >
Re: Spark-Kafka integration - build failing with sbt
There are different projects for different versions of kafka, spark-streaming-kafka-0-8 and spark-streaming-kafka-0-10 See http://spark.apache.org/docs/latest/streaming-kafka-integration.html On Fri, Jun 16, 2017 at 6:51 PM, karan alang wrote: > I'm trying to compile kafka & Spark Streaming integration code i.e. reading > from Kafka using Spark Streaming, > and the sbt build is failing with error - > > [error] (*:update) sbt.ResolveException: unresolved dependency: > org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > > Scala version -> 2.10.7 > Spark Version -> 2.1.0 > Kafka version -> 0.9 > sbt version -> 0.13 > > Contents of sbt files is as shown below -> > > 1) > vi spark_kafka_code/project/plugins.sbt > > addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2") > > 2) > vi spark_kafka_code/sparkkafka.sbt > > import AssemblyKeys._ > assemblySettings > > name := "SparkKafka Project" > > version := "1.0" > scalaVersion := "2.11.7" > > val sparkVers = "2.1.0" > > // Base Spark-provided dependencies > libraryDependencies ++= Seq( > "org.apache.spark" %% "spark-core" % sparkVers % "provided", > "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", > "org.apache.spark" %% "spark-streaming-kafka" % sparkVers) > > mergeStrategy in assembly := { > case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard > case m if m.toLowerCase.startsWith("META-INF") => MergeStrategy.discard > case "reference.conf" => MergeStrategy.concat > case m if m.endsWith("UnusedStubClass.class") => MergeStrategy.discard > case _ => MergeStrategy.first > } > > i launch sbt, and then try to create an eclipse project, complete error is > as shown below - > > - > > sbt > [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins > [info] Loading project definition from > /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/project > [info] Set current project to SparkKafka Project (in build > file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/) >> eclipse > [info] About to create Eclipse project files for your project(s). > [info] Updating > {file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/}spark_kafka_code... > [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... > [warn] module not found: > org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > [warn] local: tried > [warn] > /Users/karanalang/.ivy2/local/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [warn] activator-launcher-local: tried > [warn] > /Users/karanalang/.activator/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [warn] activator-local: tried > [warn] > /Users/karanalang/Documents/Technology/SCALA/activator-dist-1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [warn] public: tried > [warn] > https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom > [warn] typesafe-releases: tried > [warn] > http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom > [warn] typesafe-ivy-releasez: tried > [warn] > http://repo.typesafe.com/typesafe/ivy-releases/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [info] Resolving jline#jline;2.12.1 ... > [warn] :: > [warn] :: UNRESOLVED DEPENDENCIES :: > [warn] :: > [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > [warn] :: > [warn] > [warn] Note: Unresolved dependencies path: > [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 > (/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/sparkkafka.sbt#L12-16) > [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 > [trace] Stack trace suppressed: run last *:update for the full output. > [error] (*:update) sbt.ResolveException: unresolved dependency: > org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > [info] Updating > {file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/}spark_kafka_code... > [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... > [warn] module not found: > org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > [warn] local: tried > [warn] > /Users/karanalang/.ivy2/local/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [warn] activator-launcher-local: tried > [warn] > /Users/karanalang/.activator/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > [warn] activator-local: tried > [warn] > /Users/karanalang/Documents/Techno