Hello all, There is a bug in the spark-ec2 script (perhaps due to a change in the Amazon AMI).
The --ebs-vol-size option directs the spark-ec2 script to add an EBS volume of the specified size, and mount it at /vol for a persistent HDFS. To do this, it uses mkfs.xfs which is not available (though mkfs is). To work around this, I was able to run "yum install xfsprogs" on the master and each slave, and then use the --resume option with the script, and the persistent HDFS actually worked! This has been a frustrating experience, but I've used the spark-ec2 script for several months now, and it's incredibly helpful. I hope this post helps towards fixing the problem! Thanks, -Ben P.S. This is the full initial command I used, in case this is isolated to particular instance types or anything: ec2/spark-ec2 -k ... -i ... -z us-east-1d -s 4 -t m3.2xlarge --ebs-vol-size=250 -m r3.2xlarge launch ... P.P.S. Ganglia is still broken, and has been for a while... -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/BUG-in-spark-ec2-script-ebs-vol-size-and-workaround-tp10217.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
