spark-submit results in NoClassDefFoundError

2014-10-29 Thread Tobias Pfeiffer 
Hi,

I am trying to get my Spark application to run on YARN and by now I have
managed to build a fat jar as described on 
http://markmail.org/message/c6no2nyaqjdujnkq (which is the only really
usable  manual on how to get such a jar file). My code runs fine using sbt
test and sbt run, but when running

~/spark-1.1.0-bin-hadoop2.4/bin/spark-submit \
  --class my.spark.MyClass --master local[3] \
  target/scala-2.10/myclass-assembly-1.0.jar

I get:

Spark assembly has been built with Hive, including Datanucleus jars on
classpath
Exception in thread main java.lang.NoClassDefFoundError:
com/typesafe/scalalogging/slf4j/Logger
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)
at java.lang.Class.getMethod0(Class.java:2856)
at java.lang.Class.getMethod(Class.java:1668)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:325)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
com.typesafe.scalalogging.slf4j.Logger
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 7 more
ABRT problem creation: 'success'

It seems to run into an error before it does anything with my jar?

I am using
  com.typesafe.scala-logging %% scala-logging-slf4j% 2.1.2
instead of
  com.typesafe %% scalalogging-slf4j% 1.1.0
in my SBT file, could that be a reason?

Thanks
Tobias

Re: spark-submit results in NoClassDefFoundError

2014-10-29 Thread Tobias Pfeiffer 
Hi again,

On Thu, Oct 30, 2014 at 11:50 AM, Tobias Pfeiffer t...@preferred.jp wrote:

 Spark assembly has been built with Hive, including Datanucleus jars on
 classpath
 Exception in thread main java.lang.NoClassDefFoundError:
 com/typesafe/scalalogging/slf4j/Logger


It turned out scalalogging was not included in the fat jar due to 
https://github.com/sbt/sbt-assembly/issues/116.


 I am using
   com.typesafe.scala-logging %% scala-logging-slf4j% 2.1.2
 instead of
   com.typesafe %% scalalogging-slf4j% 1.1.0
 in my SBT file, could that be a reason?


So yes, that was the reason, in a way... however, I decided to include
scala in the fat jar instead of modifying all my logging code...

Tobias