No, I am not. I run it with sbt «sbt "run-main Branchmark"». I thought it
was the same thing since I am passing all the configurations through the
application code. Is that the problem?
On Thu, Jun 4, 2015 at 6:26 PM, Sandy Ryza <sandy.r...@cloudera.com> wrote:
> Hi Saiph,
>
> Are you launching using spark-submit?
>
> -Sandy
>
> On Thu, Jun 4, 2015 at 10:20 AM, Saiph Kappa <saiph.ka...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I've been running my spark streaming application in standalone mode
>> without any worries. Now, I've been trying to run it on YARN (hadoop 2.7.0)
>> but I am having some problems.
>>
>> Here are the config parameters of my application:
>> «
>> val sparkConf = new SparkConf()
>>
>> sparkConf.setMaster("yarn-client")
>> sparkConf.set("spark.yarn.am.memory", "2g")
>> sparkConf.set("spark.executor.instances", "2")
>>
>> sparkConf.setAppName("Benchmark")
>>
>> sparkConf.setJars(Array("target/scala-2.10/benchmark-app_2.10-0.1-SNAPSHOT.jar"))
>> sparkConf.set("spark.executor.memory", "4g")
>> sparkConf.set("spark.serializer",
>> "org.apache.spark.serializer.KryoSerializer")
>> sparkConf.set("spark.executor.extraJavaOptions", " -XX:+UseCompressedOops
>> -XX:+UseConcMarkSweepGC " +
>> "-XX:+AggressiveOpts -XX:FreqInlineSize=300 -XX:MaxInlineSize=300 ")
>> if (sparkConf.getOption("spark.master") == None) {
>> sparkConf.setMaster("local[*]")
>> }
>> »
>>
>> The jar I'm including there only contains the application classes.
>>
>>
>> Here is the log of the application: http://pastebin.com/7RSktezA
>>
>> Here is the userlog on hadoop/YARN:
>> «
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/spark/Logging
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>> at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at
>> org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:596)
>> at
>> org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
>> Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> ... 14 more
>> »
>>
>> I tried to add the spark core jar to ${HADOOP_HOME}/lib but the error
>> persists. Am I doing something wrong?
>>
>> Thanks.
>>
>
>
