So weird. Is there anything wrong with the way I made the pom file (I labelled them as <scope>provided</scope>)?
Is there missing jar I forget to add in “--jar”? See the trace below: Exception in thread "main" java.lang.NoClassDefFoundError: breeze/storage/DefaultArrayValue at smartapp.smart.sparkwithscala.textMingApp.main(textMingApp.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: breeze.storage.DefaultArrayValue at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 10 more 15/11/18 17:15:15 INFO util.Utils: Shutdown hook called From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Wednesday, 18 November 2015 4:01 PM To: Jack Yang Cc: user@spark.apache.org Subject: Re: spark with breeze error of NoClassDefFoundError Looking in local maven repo, breeze_2.10-0.7.jar contains DefaultArrayValue : jar tvf /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | grep !$ jar tvf /Users/tyu/.m2/repository//org/scalanlp/breeze_2.10/0.7/breeze_2.10-0.7.jar | grep DefaultArrayValue 369 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$mcZ$sp$class.class 309 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$mcJ$sp.class 2233 Wed Mar 19 11:18:32 PDT 2014 breeze/storage/DefaultArrayValue$DoubleDefaultArrayValue$.class Can you show the complete stack trace ? FYI On Tue, Nov 17, 2015 at 8:33 PM, Jack Yang <j...@uow.edu.au<mailto:j...@uow.edu.au>> wrote: Hi all, I am using spark 1.4.0, and building my codes using maven. So in one of my scala, I used: import breeze.linalg._ val v1 = new breeze.linalg.SparseVector(commonVector.indices, commonVector.values, commonVector.size) val v2 = new breeze.linalg.SparseVector(commonVector2.indices, commonVector2.values, commonVector2.size) println (v1.dot(v2) / (norm(v1) * norm(v2)) ) in my pom.xml file, I used: <dependency> <groupId>org.scalanlp</groupId> <artifactId>breeze-math_2.10</artifactId> <version>0.4</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.scalanlp</groupId> <artifactId>breeze_2.10</artifactId> <version>0.11.2</version> <scope>provided</scope> </dependency> When submit, I included breeze jars (breeze_2.10-0.11.2.jar breeze-math_2.10-0.4.jar breeze-natives_2.10-0.11.2.jar breeze-process_2.10-0.3.jar) using “--jar” arguments, although I doubt it is necessary to do that. however, the error is Exception in thread "main" java.lang.NoClassDefFoundError: breeze/storage/DefaultArrayValue Any thoughts? Best regards, Jack