spark-submit is the recommended way of launching Spark applications on
YARN, because it takes care of submitting the right jars as well as setting
up the classpath and environment variables appropriately.
-Sandy
On Thu, Jun 4, 2015 at 10:30 AM, Saiph Kappa <saiph.ka...@gmail.com> wrote:
> No, I am not. I run it with sbt «sbt "run-main Branchmark"». I thought it
> was the same thing since I am passing all the configurations through the
> application code. Is that the problem?
>
> On Thu, Jun 4, 2015 at 6:26 PM, Sandy Ryza <sandy.r...@cloudera.com>
> wrote:
>
>> Hi Saiph,
>>
>> Are you launching using spark-submit?
>>
>> -Sandy
>>
>> On Thu, Jun 4, 2015 at 10:20 AM, Saiph Kappa <saiph.ka...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I've been running my spark streaming application in standalone mode
>>> without any worries. Now, I've been trying to run it on YARN (hadoop 2.7.0)
>>> but I am having some problems.
>>>
>>> Here are the config parameters of my application:
>>> «
>>> val sparkConf = new SparkConf()
>>>
>>> sparkConf.setMaster("yarn-client")
>>> sparkConf.set("spark.yarn.am.memory", "2g")
>>> sparkConf.set("spark.executor.instances", "2")
>>>
>>> sparkConf.setAppName("Benchmark")
>>>
>>> sparkConf.setJars(Array("target/scala-2.10/benchmark-app_2.10-0.1-SNAPSHOT.jar"))
>>> sparkConf.set("spark.executor.memory", "4g")
>>> sparkConf.set("spark.serializer",
>>> "org.apache.spark.serializer.KryoSerializer")
>>> sparkConf.set("spark.executor.extraJavaOptions", "
>>> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC " +
>>> "-XX:+AggressiveOpts -XX:FreqInlineSize=300 -XX:MaxInlineSize=300
>>> ")
>>> if (sparkConf.getOption("spark.master") == None) {
>>> sparkConf.setMaster("local[*]")
>>> }
>>> »
>>>
>>> The jar I'm including there only contains the application classes.
>>>
>>>
>>> Here is the log of the application: http://pastebin.com/7RSktezA
>>>
>>> Here is the userlog on hadoop/YARN:
>>> «
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/spark/Logging
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> at
>>> org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:596)
>>> at
>>> org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
>>> Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>> ... 14 more
>>> »
>>>
>>> I tried to add the spark core jar to ${HADOOP_HOME}/lib but the error
>>> persists. Am I doing something wrong?
>>>
>>> Thanks.
>>>
>>
>>
>
