[
https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366251#comment-15366251
]
Thomas Graves edited comment on SPARK-16265 at 7/7/16 3:10 PM:
---------------------------------------------------------------
Have you tried shipping your jre as gzip file in the distributed cache and
pointing to it with: spark.yarn.appMasterEnv.JAVA_HOME=<path-to-java> and
spark.executorEnv.JAVA_HOME=<path-to-java>?
Not positive that works but you should try it first.
I'd rather not get into the business of installing software with Spark. It
then starts to extend to things like I want to install python version x, I want
to install R version Y, etc. If there is a generic way we can provide support
for this I'm fine with it. For instance we already can ship most things in
distributed cache and generically set any env variable, that makes it possible
to configure many things.
Also as stated above generally in the yarn environment java would be one of
those things installed on the clusters. yes we try to keep things isolated and
you should ship as much stuff as possible with your application but at this
point we also rely on the hadoop libraries and other things that you could have
incompatibilities with. obviously docker or something like that could give you
an entire environment.
was (Author: tgraves):
Have you tried shipping your jre as gzip file in the distributed cache and
pointing to it with: spark.yarn.appMasterEnv.JAVA_HOME=<path-to-java> and
spark.executorEnv.JAVA_HOME=<path-to-java>?
Not positive that works but you should try it first.
I'd rather not get into the business of installing software with Spark. It
then starts to extend to things like I want to install python version x, I want
to install R version Y, etc.
Also as stated above generally in the yarn environment java would be one of
those things installed on the clusters. yes we try to keep things isolated and
you should ship as much stuff as possible with your application but at this
point we also rely on the hadoop libraries and other things that you could have
incompatibilities with. obviously docker or something like that could give you
an entire environment.
> Add option to SparkSubmit to ship driver JRE to YARN
> ----------------------------------------------------
>
> Key: SPARK-16265
> URL: https://issues.apache.org/jira/browse/SPARK-16265
> Project: Spark
> Issue Type: Improvement
> Affects Versions: 1.6.2
> Reporter: Andrew Duffy
>
> Add an option to {{SparkSubmit}} to allow the driver to package up it's
> version of the JRE to be shipped to a YARN cluster. This allows deploying
> Spark applications to a YARN cluster in which its required Java version need
> not match one of the versions already installed on the YARN cluster, useful
> in situations in which the Spark Application developer does not have
> administrative access over the YARN cluster (ex. school or corporate
> environment) but still wants to use certain language features in their code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
