Rolling upgrade Spark cluster

2014-12-16 Thread Kenichi Maehashi 
Hi,

I have a Spark cluster using standalone mode. Spark Master is
configured as High Availablity mode.
Now I am going to upgrade Spark from 1.0 to 1.1, but don't want to
interrupt the currently running jobs.

(1) Are there any way to perform a rolling upgrade (while running a job)?
(2) If not, when using YARN as a cluster manager, can I perform a
rolling upgrade?

Thanks,

Kenichi

-- 
Kenichi Maehashi webmas...@kenichimaehashi.com

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Slave Node Management in Standalone Cluster

2014-11-18 Thread Kenichi Maehashi 
Hi,

I'm operating Spark in standalone cluster configuration (3 slaves) and
have some question.

1. How can I stop a slave on the specific node?
   Under `sbin/` directory, there are
`start-{all,master,slave,slaves}` and `stop-{all,master,slaves}`, but
no `stop-slave`. Are there any way to stop the specific (e.g., the
2nd) slave via command line?

2. How can I check cluster status from command line?
   Are there any way to confirm that all Master / Workers are up and
working without using Web UI?

Thanks in advance!

-- 
Kenichi Maehashi

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Slave Node Management in Standalone Cluster

2014-11-18 Thread Kenichi Maehashi 
Hi Akhil and Kousuke,

Thank you for your quick response.
Monitoring through JSON API seems straightforward and cool.

Thanks again!




2014-11-18 19:06 GMT+09:00 Kousuke Saruta saru...@oss.nttdata.co.jp:
 Hi Kenichi

 1. How can I stop a slave on the specific node?
Under `sbin/` directory, there are
 `start-{all,master,slave,slaves}` and `stop-{all,master,slaves}`, but
 no `stop-slave`. Are there any way to stop the specific (e.g., the
 2nd) slave via command line?


 You can use sbin/spark-daemon.sh on the machine where the worker you'd like
 to stop runs.
 First, you find PID of the worker you'd like to stop and second, you find
 PID file of the worker.
 The PID file is on /tmp/ by default and the file name is like as follows.

 xxx.org.apache.spark.deploy.worker.Worker-WorkerID.pid

 After you find the PID file, you run the following command.

 sbin/spark-daemon.sh stop org.apache.spark.worker.Worker WorkerID

 2. How can I check cluster status from command line?
Are there any way to confirm that all Master / Workers are up and
 working without using Web UI?


 AFAIK, there are no command line tools for checking statuses of standalone
 cluster.
 Instead of that, you can use special URL like as follows.

 http://master or worker's hostname:webui-port/json

 You can get Master and Worker status as JSON format data.

 - Kousuke


 (2014/11/18 0:27), Kenichi Maehashi wrote:

 Hi,

 I'm operating Spark in standalone cluster configuration (3 slaves) and
 have some question.

 1. How can I stop a slave on the specific node?
 Under `sbin/` directory, there are
 `start-{all,master,slave,slaves}` and `stop-{all,master,slaves}`, but
 no `stop-slave`. Are there any way to stop the specific (e.g., the
 2nd) slave via command line?

 2. How can I check cluster status from command line?
 Are there any way to confirm that all Master / Workers are up and
 working without using Web UI?

 Thanks in advance!




-- 
Kenichi Maehashi

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org