I would upgrade your Scala version to 2.11.8 as Spark 2.0 uses Scala 2.11 by default.
On Sun, Nov 13, 2016 at 3:01 PM, Marco Mistroni <mmistr...@gmail.com> wrote: > HI all > i have a small Spark-based project which at the moment depends on jar > from Spark 1.6.0 > The project has few Spark examples plus one which depends on Flume > libraries > > > I am attempting to move to Spark 2.0, but i am having issues with > my dependencies > The stetup below works fine when compiled against 1.6.0 dependencies > > name := "SparkExamples" > version := "1.0" > scalaVersion := "2.10.5" > val sparkVersion = "1.6.0" > > > // Add a single dependency > libraryDependencies += "junit" % "junit" % "4.8" % "test" > libraryDependencies ++= Seq("org.slf4j" % "slf4j-api" % "1.7.5", > "org.slf4j" % "slf4j-simple" % "1.7.5", > "org.clapper" %% "grizzled-slf4j" % "1.0.2") > libraryDependencies += "org.apache.spark"%%"spark-core" % sparkVersion > libraryDependencies += "org.apache.spark"%%"spark-streaming" % > sparkVersion > libraryDependencies += "org.apache.spark"%%"spark-mllib" % sparkVersion > libraryDependencies += "org.apache.spark"%%"spark-streaming-flume" % > "1.3.0" > libraryDependencies += "org.apache.spark"%%"spark-sql" % sparkVersion > > > resolvers += "softprops-maven" at "http://dl.bintray.com/ > content/softprops/maven" > > > > This is the build.sbt version for using Spark 2 dependencies > > name := "SparkExamples" > version := "1.0" > scalaVersion := "2.10.6" > val sparkVersion = "2.0.1" > > > // Add a single dependency > libraryDependencies += "junit" % "junit" % "4.8" % "test" > libraryDependencies ++= Seq("org.slf4j" % "slf4j-api" % "1.7.5", > "org.slf4j" % "slf4j-simple" % "1.7.5", > "org.clapper" %% "grizzled-slf4j" % "1.0.2") > libraryDependencies += "org.apache.spark"%%"spark-core" % sparkVersion > libraryDependencies += "org.apache.spark"%%"spark-streaming" % > sparkVersion > libraryDependencies += "org.apache.spark"%%"spark-mllib" % sparkVersion > libraryDependencies += "org.apache.spark"%%"spark-streaming-flume-sink" % > "2.0.1" > libraryDependencies += "org.apache.spark"%%"spark-sql" % sparkVersion > resolvers += "softprops-maven" at "http://dl.bintray.com/ > content/softprops/maven" > > but the sbt compile fails miserably...below few of the errors (it actually > compiles like i forgot all the depencencies as it is complaining on all > org.apache.spark.ml and mllib packages > > [warn] Multiple dependencies with the same organization/name but different > versions. To avoid conflict, pick one version: > [warn] * org.apache.spark:spark-core_2.10:(1.6.1, 2.0.1) > [warn] * org.apache.spark:spark-streaming_2.10:(1.6.1, 2.0.1) > [warn] * org.apache.spark:spark-sql_2.10:(1.6.1, 2.0.1) > [warn] * org.apache.spark:spark-mllib_2.10:(1.6.1, 2.0.1) > [info] Resolving org.scala-lang#scala-library;2.10.6 ... > ......... > [warn] * org.apache.spark:spark-mllib_2.10:1.6.1 -> 2.0.1 > [warn] * org.apache.spark:spark-sql_2.10:1.6.1 -> 2.0.1 > [warn] * org.apache.spark:spark-streaming_2.10:1.6.1 -> 2.0.1 > [warn] * org.apache.spark:spark-core_2.10:1.6.1 -> 2.0.1 > [warn] Run 'evicted' to see detailed eviction warnings > [info] Compiling 18 Scala sources to C:\Users\marco\SparkExamples\ > target\scala-2.10\classes... > [error] C:\Users\marco\SparkExamples\src\main\scala\ > AnotherDecisionTreeExample.scala:2: object mllib is not a member of > package org.apache.spark > [error] import org.apache.spark.mllib.linalg.{ Vector, Vectors } > [error] ^ > [error] C:\Users\marco\SparkExamples\src\main\scala\ > AnotherDecisionTreeExample.scala:3: object mllib is not a member of > package org.apache.spark > [error] import org.apache.spark.mllib.regression.LabeledPoint > [error] ^ > [error] C:\Users\marco\SparkExamples\src\main\scala\ > AnotherDecisionTreeExample.scala:4: object classification is not a member > of package org.apache.spark.ml > [error] import org.apache.spark.ml.classification._ > [error] ^ > [error] C:\Users\marco\SparkExamples\src\main\scala\ > AnotherDecisionTreeExample.scala:5: object mllib is not a member of > package org.apache.spark > [error] import org.apache.spark.mllib.tree.DecisionTree > [error] ^ > [error] C:\Users\marco\SparkExamples\src\main\scala\ > AnotherDecisionTreeExample.scala:6: object mllib is not a member of > package org.apache.spark > [error] import org.apache.spark.mllib.tree.model.DecisionTreeModel > [error] ^ > [error] C:\Users\marco\SparkExamples\src\main\scala\ > AnotherDecisionTreeExample.scala:7: object mllib is not a member of > package org.apache.spark > [error] import org.apache.spark.mllib.util.MLUtils > [error] ^ > [error] C:\Users\marco\SparkExamples\src\main\scala\ > AnotherDecisionTreeExample.scala:9: object mllib is not a member of > package org.apache.spark > [error] import org.apache.spark.mllib.stat.{ MultivariateStatisticalSummary, > Statistics } > [error] ^ > [error] C:\Users\marco\SparkExamples\src\main\scala\ > AnotherDecisionTreeExample.scala:10: object mllib is not a member of > package org.apache.spark > [error] import org.apache.spark.mllib.linalg.Vector > ........ > C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:4: > object classification is not a member of package org.apache.spark.ml > [error] import org.apache.spark.ml.classification.{ > RandomForestClassifier, RandomForestClassificationModel } > [error] ^ > [error] > C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:6: > object feature is not a member of package org.apache.spark.ml > [error] import org.apache.spark.ml.feature.{ StringIndexer, IndexToString, > VectorIndexer, VectorAssembler } > [error] ^ > [error] > C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:7: > object evaluation is not a member of package org.apache.spark.ml > [error] import org.apache.spark.ml.evaluation.{ RegressionEvaluator, > MulticlassClassificationEvaluator } > [error] ^ > [error] > C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:8: > object classification is not a member of package org.apache.spark.ml > [error] import org.apache.spark.ml.classification._ > [error] ^ > [error] > C:\Users\marco\SparkExamples\src\main\scala\DecisionTreeExampleML.scala:9: > object tuning is not a member of package org.apache.spark.ml > [error] import org.apache.spark.ml.tuning.{ CrossValidator, > ParamGridBuilder } > [error] ^ > [error] C:\Users\marco\SparkExamples\src\main\scala\ > DecisionTreeExampleML.scala:10: object tuning is not a member of package > org.apache.spark.ml > [error] import org.apache.spark.ml.tuning.{ ParamGridBuilder, > TrainValidationSplit } > [error] ^ > [error] C:\Users\marco\SparkExamples\src\main\scala\ > DecisionTreeExampleML.scala:16: object Pipeline is not a member of > package org.apache.spark.ml > [error] import org.apache.spark.ml.{ Pipeline, PipelineModel } > [error] > .... > and many more > > Could anyone assist or pass me the right dependencies? Or am i using wrong > resolvers? > > kr > marco > -- Donald Drake Drake Consulting http://www.drakeconsulting.com/ https://twitter.com/dondrake <http://www.MailLaunder.com/> 800-733-2143