Re: Trouble with spark-ec2 script: --ebs-vol-size
Should I take it from the lack of replies that the --ebs-vol-size feature doesn't work? -Ben -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trouble-with-spark-ec2-script-ebs-vol-size-tp9619p9934.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Trouble with spark-ec2 script: --ebs-vol-size
please add From: Ben Horner [via Apache Spark User List] ml-node+s1001560n9934...@n3.nabble.commailto:ml-node+s1001560n9934...@n3.nabble.com Date: Wednesday, July 16, 2014 at 8:47 AM To: Ben Horner ben.hor...@atigeo.commailto:ben.hor...@atigeo.com Subject: Re: Trouble with spark-ec2 script: --ebs-vol-size Should I take it from the lack of replies that the --ebs-vol-size feature doesn't work? -Ben If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Trouble-with-spark-ec2-script-ebs-vol-size-tp9619p9934.html To unsubscribe from Trouble with spark-ec2 script: --ebs-vol-size, click herehttp://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=9619code=YmVuLmhvcm5lckBhdGlnZW8uY29tfDk2MTl8MTk2OTU5NjEyOQ==. NAMLhttp://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trouble-with-spark-ec2-script-ebs-vol-size-tp9619p9935.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Trouble with spark-ec2 script: --ebs-vol-size
Hi Ben, It worked for me, but only when using the default region. Using --region=us-west-2 resulted in errors about security groups. Chris On Wed, Jul 16, 2014 at 8:53 AM, Ben Horner ben.hor...@atigeo.com wrote: please add From: Ben Horner [via Apache Spark User List] [hidden email] http://user/SendEmail.jtp?type=nodenode=9935i=0 Date: Wednesday, July 16, 2014 at 8:47 AM To: Ben Horner [hidden email] http://user/SendEmail.jtp?type=nodenode=9935i=1 Subject: Re: Trouble with spark-ec2 script: --ebs-vol-size Should I take it from the lack of replies that the --ebs-vol-size feature doesn't work? -Ben -- If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Trouble-with-spark-ec2-script-ebs-vol-size-tp9619p9934.html To unsubscribe from Trouble with spark-ec2 script: --ebs-vol-size, click here. NAML http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- View this message in context: Re: Trouble with spark-ec2 script: --ebs-vol-size http://apache-spark-user-list.1001560.n3.nabble.com/Trouble-with-spark-ec2-script-ebs-vol-size-tp9619p9935.html Sent from the Apache Spark User List mailing list archive http://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
Trouble with spark-ec2 script: --ebs-vol-size
Hello, I'm using the spark-0.9.1-bin-hadoop1 distribution, and the ec2/spark-ec2 script within it to spin up a cluster. I tried running my processing just using the default (ephemeral) HDFS configuration, but my job errored out, saying that there was no space left. So now I'm trying to increase the size of HDFS on the cluster. My launch command: ec2/spark-ec2 -k ... -i ... -z us-east-1d -s 4 -t m3.2xlarge --ebs-vol-size=250 -m r3.2xlarge launch ... My understanding is that I should get a cluster, where each slave node has an ebs backed drive with 250 GB of storage, with a persistent HDFS set to use these slave drives. I turn off the ephemeral HDFS on the cluster master: ephemeral-hdfs/bin/stop-all.sh Then I turn on the persistent HDFS on the cluster master: persistent-hdfs/bin/start-all.sh Once I discovered the proper URL to hit the persistent name node page (not the ephemeral standard 50070 port): http://master:60070/dfshealth.jsp The page shows 4 nodes as expected, but the configured capacity shows as 31.5 GB, not the expected 1 TB (250 GB x 4) Please help! Don't be shy to let me know if I've made mis-steps, or if I'm not understanding things correctly! Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trouble-with-spark-ec2-script-ebs-vol-size-tp9619.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
