I'm trying to submit a simple test code through spark-submit. first portion
of the code works fine, but some calls to breeze vector library fails:
14/10/22 17:36:02 INFO CacheManager: Partition rdd_1_0 not found, computing
it
14/10/22 17:36:02 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError:
breeze.linalg.SparseVector$mcD$sp.<init>([I[DILbreeze/storage/DefaultArrayValue;)V
at spark.kmeans.SparseVector.toBreeze(Vectors.scala:182)
at spark.kmeans.MyTest$$anonfun$1.apply(MyTest.scala:56)
at spark.kmeans.MyTest$$anonfun$1.apply(MyTest.scala:56)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:236)
I ran this code with
~/tools/spark-1.1.0-bin-hadoop2.4/bin/spark-submit --class
spark.kmeans.MyTest --master local target/reco-v1-jar-with-dependencies.jar
my system scala version is 2.10.4
the relevant sections in my pom is: (I did not build with sbt, since it
gave me some problems)
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.0.2</version>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>1.0.2</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.4</version>
</dependency>
Thanks a lot
Yang