[
https://issues.apache.org/jira/browse/SPARK-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001161#comment-14001161
]
Patrick Wendell commented on SPARK-1870:
----------------------------------------
The jars may not be present on the classpath because we add them through
dynamic classloading and not by modifying the system classpath.
What happens if you also call sc.addJar(X) with the filename of the jar inside
your application? In the future it might be nice to automatically call this for
you, but I think for now you need to do it yourself in YARN mode. Here are the
relevant docs:
http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/running-on-yarn.html#adding-additional-jars
These were written by [~sandyr].
> Jars specified via --jars in spark-submit are not added to executor classpath
> for YARN
> --------------------------------------------------------------------------------------
>
> Key: SPARK-1870
> URL: https://issues.apache.org/jira/browse/SPARK-1870
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.0.0
> Reporter: Xiangrui Meng
> Priority: Critical
>
> With `spark-submit`, jars specified via `--jars` are added to distributed
> cache in `yarn-cluster` mode. The executor should add cached jars to
> classpath. However,
> {code}
> sc.parallelize(0 to 10, 10).map { i =>
> System.getProperty("java.class.path")
> }.collect().foreach(println)
> {code}
> shows only system jars, `app.jar`, and `spark.jar` but not other jars in the
> distributed cache.
> The workaround is using assembly jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
