hi, I'm working on spark standalone system on ec2, and I'm having problems on resizing the cluster (meaning - adding or removing slaves).
In the basic ec2 scripts (http://spark.apache.org/docs/latest/ec2-scripts.html), there's only script for lunching the cluster, not adding slaves to it. On the spark-standalone page (http://spark.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts), I can see only options for stopping and starting slaves, not adding them. What I try to do now (as a bad workaround...), is the following: 1) Go to the ec2 UI, create image from the current slave 2) Lunch new instance based on this image 3) Copy the public DNS of this slave 4) SSH to the master, and edit the file "/root/spark-ec2/ec2-variables.sh", and add the DNS to the "export SLAVES" variables 5) Running the script /root/spark-ec2/setup.sh After doing the above steps, I can see the new slave in the UI (8080 port) of the master. However, this solution is bad for many reasons: 1) It requires many manual steps 2) It requires stopping and starting the cluster 3) There's no auto-detection in case slave stopped and many other reasons... oes anybody have another idea on how to add/remove slaves for standalone on a simple and safe way? thanks, nizan -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Adding-slaves-on-spark-standalone-on-ec2-tp23064.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
