[
https://issues.apache.org/jira/browse/SPARK-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648234#comment-14648234
]
David Chin edited comment on SPARK-967 at 7/30/15 8:34 PM:
-----------------------------------------------------------
I won't create a pull request unless asked to, but I have a solution for this.
I am running Spark in standalone mode within a Univa Grid Engine cluster. As
such, configs and logs, etc should be specific to each UGE job, identified by
an integer job ID.
Currently, any environment variables on the master are not passed along by the
sbin/start-slaves.sh invocation of ssh. I put in a fix on my local version,
which works. However, this is still less than ideal in that UGE's job
accounting cannot keep track of resource usage by jobs not under its process
tree. Not sure, yet, what the correct solution is. I thought I saw a feature
request to allow other remote shell programs besides ssh, but I can't find it
now.
Please see my version of sbin/start-slaves.sh here, forked from current master:
https://github.com/prehensilecode/spark/blob/master/sbin/start-slaves.sh
was (Author: prehensilecode):
I won't create a pull request unless asked to, but I have a solution for this.
I am running Spark in standalone mode within a Univa Grid Engine cluster. As
such, configs and logs, etc should be specific to each UGE job, identified by
an integer job ID.
Currently, any environment variables on the master are not passed along by the
sbin/start-slaves.sh invocation of ssh. I put in a fix on my local version,
which works. However, this is still less than ideal in that UGE's job
accounting cannot keep track of resource usage by jobs not under its process
tree. Not sure, yet, what the correct solution is. I thought I saw a feature
request to allow other remote shell programs besides ssh, but I can't find it
now.
Please see my version of sbin/start-slaves.sh here:
https://github.com/prehensilecode/spark/blob/master/sbin/start-slaves.sh
> start-slaves.sh uses local path from master on remote slave nodes
> -----------------------------------------------------------------
>
> Key: SPARK-967
> URL: https://issues.apache.org/jira/browse/SPARK-967
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 0.8.0, 0.8.1, 0.9.0
> Reporter: Evgeniy Tsvigun
> Priority: Trivial
> Labels: script, starter
>
> If a slave node has home path other than master, start-slave.sh fails to
> start a worker instance, for other nodes behaves as expected, in my case:
> $ ./bin/start-slaves.sh
> node05.dev.vega.ru: bash: line 0: cd: /usr/home/etsvigun/spark/bin/..: No
> such file or directory
> node04.dev.vega.ru: org.apache.spark.deploy.worker.Worker running as
> process 4796. Stop it first.
> node03.dev.vega.ru: org.apache.spark.deploy.worker.Worker running as
> process 61348. Stop it first.
> I don't mention /usr/home anywhere, the only environment variable I set is
> $SPARK_HOME, relative to $HOME on every node, which makes me think some
> script takes `pwd` on master and tries to use it on slaves.
> Spark version: fb6875dd5c9334802580155464cef9ac4d4cc1f0
> OS: FreeBSD 8.4
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
