[
https://issues.apache.org/jira/browse/SPARK-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nan Zhu updated SPARK-2404:
---------------------------
Description:
in spark-class and spark-submit
the SPARK_HOME is set to the present working directory, causing the value of
already defined SPARK_HOME being overwritten
we should not overwrite that if SPARK_HOME has been defined
Our scenario
we have a login portal for all the team members to use the spark-cluster,
everyone gets an account and home directory
spark-1.0 is copied in the root path "/", every account gets a soft link to the
/spark-1.0 in it's home directory
spark 1.0 is deployed in a cluster, where the user name is different with any
accounts except one, say "nanzhu", in login portal,
so when the user runs spark-shell, it always tries to run
/home/user_account/spark-1.0/bin/compute-class.sh, which does not exist.
We set a global SPARK_HOME to /home/nanzhu/spark-1.0 globally which is
consistent with the remote cluster setup, but unfortunately, this is
overwritten by the spark-class and spark-submit ....
was:
in spark-class and spark-submit
the SPARK_HOME is set to the present working directory, causing the value of
already defined SPARK_HOME being overwritten
we should not overwrite that if SPARK_HOME has been defined
Our scenario
we have a login portal for all the team members to use the spark-cluster,
everyone gets an account and home directory
spark-1.0 is copied in the root path "/", every account gets a software link to
the /spark-1.0 in it's home directory
spark 1.0 is deployed in a cluster, where the user name is different with any
accounts except one, say "nanzhu", in login portal,
so when the user runs spark-shell, it always tries to run
/home/user_account/spark-1.0/bin/compute-class.sh, which does not exist.
We set a global SPARK_HOME to /home/nanzhu/spark-1.0 globally which is
consistent with the remote cluster setup, but unfortunately, this is
overwritten by the spark-class and spark-submit ....
> spark-submit and spark-class may overwrite the already defined SPARK_HOME
> -------------------------------------------------------------------------
>
> Key: SPARK-2404
> URL: https://issues.apache.org/jira/browse/SPARK-2404
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Nan Zhu
> Assignee: Nan Zhu
> Fix For: 1.0.1
>
>
> in spark-class and spark-submit
> the SPARK_HOME is set to the present working directory, causing the value of
> already defined SPARK_HOME being overwritten
> we should not overwrite that if SPARK_HOME has been defined
> Our scenario
> we have a login portal for all the team members to use the spark-cluster,
> everyone gets an account and home directory
> spark-1.0 is copied in the root path "/", every account gets a soft link to
> the /spark-1.0 in it's home directory
> spark 1.0 is deployed in a cluster, where the user name is different with any
> accounts except one, say "nanzhu", in login portal,
> so when the user runs spark-shell, it always tries to run
> /home/user_account/spark-1.0/bin/compute-class.sh, which does not exist.
> We set a global SPARK_HOME to /home/nanzhu/spark-1.0 globally which is
> consistent with the remote cluster setup, but unfortunately, this is
> overwritten by the spark-class and spark-submit ....
--
This message was sent by Atlassian JIRA
(v6.2#6252)
