Awesome. -Vadim ᐧ
On Fri, Jun 19, 2015 at 8:30 PM, Kelly, Jonathan <jonat...@amazon.com> wrote: > Yep, I'm on the EMR team at Amazon, and I was at the Spark Summit. ;-) > So of course I'm biased toward EMR, even over EC2. I'm not sure if there's > a way to resize an EC2 Spark cluster, or at least if there is, it probably > still requires manual work on the new nodes. This would be the advantage of > EMR over EC2, as we take care of all of that configuration. > > ~ Jonathan > > From: Vadim Bichutskiy <vadim.bichuts...@gmail.com> > Date: Friday, June 19, 2015 at 5:21 PM > To: Jonathan Kelly <jonat...@amazon.com> > Cc: user <user@spark.apache.org> > Subject: Re: [ERROR] Insufficient Space > > Thanks Jonathan. I should totally move to EMR. Spark on EMR was > announced at Spark Summit! > > There's no easy way to resize the cluster on EC2. You basically have to > destroy it and launch a new one. Right? > > -Vadim > ᐧ > > On Fri, Jun 19, 2015 at 3:41 PM, Kelly, Jonathan <jonat...@amazon.com> > wrote: > >> Would you be able to use Spark on EMR rather than on EC2? EMR clusters >> allow easy resizing of the cluster, and EMR also now supports Spark 1.3.1 >> as of EMR AMI 3.8.0. See http://aws.amazon.com/emr/spark >> >> ~ Jonathan >> >> From: Vadim Bichutskiy <vadim.bichuts...@gmail.com> >> Date: Friday, June 19, 2015 at 7:15 AM >> To: user <user@spark.apache.org> >> Subject: [ERROR] Insufficient Space >> >> Hello Spark Experts, >> >> I've been running a standalone Spark cluster on EC2 for a few months >> now, and today I get this error: >> >> "IOError: [Errno 28] No space left on device >> Spark assembly has been built with Hive, including Datanucleus jars on >> classpath >> OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory >> file" >> >> I guess I need to resize the cluster. What's the best way to do that? >> >> Thanks, >> Vadim >> ᐧ >> > >
