Re: [ERROR] Insufficient Space
Awesome. -Vadim ᐧ On Fri, Jun 19, 2015 at 8:30 PM, Kelly, Jonathan jonat...@amazon.com wrote: Yep, I'm on the EMR team at Amazon, and I was at the Spark Summit. ;-) So of course I'm biased toward EMR, even over EC2. I'm not sure if there's a way to resize an EC2 Spark cluster, or at least if there is, it probably still requires manual work on the new nodes. This would be the advantage of EMR over EC2, as we take care of all of that configuration. ~ Jonathan From: Vadim Bichutskiy vadim.bichuts...@gmail.com Date: Friday, June 19, 2015 at 5:21 PM To: Jonathan Kelly jonat...@amazon.com Cc: user user@spark.apache.org Subject: Re: [ERROR] Insufficient Space Thanks Jonathan. I should totally move to EMR. Spark on EMR was announced at Spark Summit! There's no easy way to resize the cluster on EC2. You basically have to destroy it and launch a new one. Right? -Vadim ᐧ On Fri, Jun 19, 2015 at 3:41 PM, Kelly, Jonathan jonat...@amazon.com wrote: Would you be able to use Spark on EMR rather than on EC2? EMR clusters allow easy resizing of the cluster, and EMR also now supports Spark 1.3.1 as of EMR AMI 3.8.0. See http://aws.amazon.com/emr/spark ~ Jonathan From: Vadim Bichutskiy vadim.bichuts...@gmail.com Date: Friday, June 19, 2015 at 7:15 AM To: user user@spark.apache.org Subject: [ERROR] Insufficient Space Hello Spark Experts, I've been running a standalone Spark cluster on EC2 for a few months now, and today I get this error: IOError: [Errno 28] No space left on device Spark assembly has been built with Hive, including Datanucleus jars on classpath OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file I guess I need to resize the cluster. What's the best way to do that? Thanks, Vadim ᐧ
Re: [ERROR] Insufficient Space
Vadim, You could edit /etc/fstab, then issue mount -o remount to give more shared memory online. Didn't know Spark uses shared memory. Hope this helps. On Fri, Jun 19, 2015, 8:15 AM Vadim Bichutskiy vadim.bichuts...@gmail.com wrote: Hello Spark Experts, I've been running a standalone Spark cluster on EC2 for a few months now, and today I get this error: IOError: [Errno 28] No space left on device Spark assembly has been built with Hive, including Datanucleus jars on classpath OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file I guess I need to resize the cluster. What's the best way to do that? Thanks, Vadim ᐧ
Re: [ERROR] Insufficient Space
Thanks Jonathan. I should totally move to EMR. Spark on EMR was announced at Spark Summit! There's no easy way to resize the cluster on EC2. You basically have to destroy it and launch a new one. Right? -Vadim ᐧ On Fri, Jun 19, 2015 at 3:41 PM, Kelly, Jonathan jonat...@amazon.com wrote: Would you be able to use Spark on EMR rather than on EC2? EMR clusters allow easy resizing of the cluster, and EMR also now supports Spark 1.3.1 as of EMR AMI 3.8.0. See http://aws.amazon.com/emr/spark ~ Jonathan From: Vadim Bichutskiy vadim.bichuts...@gmail.com Date: Friday, June 19, 2015 at 7:15 AM To: user user@spark.apache.org Subject: [ERROR] Insufficient Space Hello Spark Experts, I've been running a standalone Spark cluster on EC2 for a few months now, and today I get this error: IOError: [Errno 28] No space left on device Spark assembly has been built with Hive, including Datanucleus jars on classpath OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file I guess I need to resize the cluster. What's the best way to do that? Thanks, Vadim ᐧ
[ERROR] Insufficient Space
Hello Spark Experts, I've been running a standalone Spark cluster on EC2 for a few months now, and today I get this error: IOError: [Errno 28] No space left on device Spark assembly has been built with Hive, including Datanucleus jars on classpath OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file I guess I need to resize the cluster. What's the best way to do that? Thanks, Vadim ᐧ
Re: [ERROR] Insufficient Space
Would you be able to use Spark on EMR rather than on EC2? EMR clusters allow easy resizing of the cluster, and EMR also now supports Spark 1.3.1 as of EMR AMI 3.8.0. See http://aws.amazon.com/emr/spark ~ Jonathan From: Vadim Bichutskiy vadim.bichuts...@gmail.commailto:vadim.bichuts...@gmail.com Date: Friday, June 19, 2015 at 7:15 AM To: user user@spark.apache.orgmailto:user@spark.apache.org Subject: [ERROR] Insufficient Space Hello Spark Experts, I've been running a standalone Spark cluster on EC2 for a few months now, and today I get this error: IOError: [Errno 28] No space left on device Spark assembly has been built with Hive, including Datanucleus jars on classpath OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file I guess I need to resize the cluster. What's the best way to do that? Thanks, Vadim [https://mailfoogae.appspot.com/t?sender=admFkaW0uYmljaHV0c2tpeUBnbWFpbC5jb20%3Dtype=zerocontentguid=eb8b2c49-6420-4983-b2d7-df3e87c4e321]ᐧ
