[
https://issues.apache.org/jira/browse/SPARK-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219297#comment-14219297
]
Sean Owen commented on SPARK-4492:
----------------------------------
OK, it may then be that you package some of spark, but not the YARN module. For
better or worse it is not intended to be 'embedded' in another app this way, so
I think this is not supported.
That said you can do it without much trouble, I've found, if you do not need
most of what the scripts do for you, and can replicate the bits you do need
easily.
The recipe is roughly:
1) Don't embed Hadoop, Spark libraries in your app
2) Make a SparkConf in your app and configure it there
3) Run your binary on a cluster and add in the cluster's Spark/Hadoop jars to
the classpath
You can package Spark too but then you have trouble matching versions exactly.
> Exception when following SimpleApp tutorial java.lang.ClassNotFoundException:
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-4492
> URL: https://issues.apache.org/jira/browse/SPARK-4492
> Project: Spark
> Issue Type: Bug
> Reporter: sam
>
> When I follow the example here
> https://spark.apache.org/docs/1.0.2/quick-start.html and run with "java -cp
> my.jar my.main.Class" with master set to "yarn-client" I get the below
> exception.
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:228)
> at com.barclays.SimpleApp$.main(SimpleApp.scala:11)
> at com.barclays.SimpleApp.main(SimpleApp.scala)
> Caused by: org.apache.spark.SparkException: Unable to load YARN support
> at
> org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:106)
> at
> org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:101)
> at
> org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
> ... 3 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:169)
> at
> org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:102)
> ... 5 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
