Spark master keeps running out of RAM
Why would the spark master run out of RAM if I have too many slaves? Is this a flaw in the coding? I'm just a user of spark. The developer that set this up left the company, so I'm starting from the top here. So I noticed if I spawn lots of jobs, my spark master ends up crashing due to low memory. It makes sense to me the master would just be a brain/controller to dish out jobs and the resources on the slaves is what would get used up, not the master. Thanks for any ideas/concepts/info. Appreciate it much
Someone explain the /mnt and /mnt2 folders on spark-ec2 slaves
I have a 40GB ephemeral disk on /mnt and another one on /mnt2 The person that set this up has left. I'm aware of having maybe 1 ebs disk, but I guess this was launched with 2 ebs volumes using the --ebsxyz? Or are those two instance storages part of the AMI? tnx
Testing spark with AWS spot instances
I'm very new to apache spark. I'm just a user not a developer. I'm running a cluster with many spot instances. Am I correct in understanding that spark can handle an unlimited number of spot instance failures and restarts? Sometimes all the spot instances will dissapear without warning, and then they come back. Can I trust spark to pickup all jobs where it left off? I'm noticing some instability with my system. I'm suspecting it could be disk or RAM issues. When I add a lot of slaves I run low on RAM on my master. Maybe that's part of the problem. But jut want to confirm my understanding.
Re: Slaves died, but jobs not picked up by standby slaves
Never mind. What I was missing was waiting long enough :-O. Sry bout that. On Thu, Mar 24, 2016 at 11:20 AM, Dillian Murphey <crackshotm...@gmail.com> wrote: > Had 15 slaves. > > Added 10 more. > > Shut down some slaves in the first bunch of 15. > > The 10 slaves I added are sitting there idle. Spark did not assign idle > cores to pick up the slack. > > What am I missing? > > Thanks for any help. Confused. :-i >
Slaves died, but jobs not picked up by standby slaves
Had 15 slaves. Added 10 more. Shut down some slaves in the first bunch of 15. The 10 slaves I added are sitting there idle. Spark did not assign idle cores to pick up the slack. What am I missing? Thanks for any help. Confused. :-i
Re: Adding more slaves to a running cluster
It appears start-slave.sh works on a running cluster. I'm surprised I can't find more info on this. Maybe I'm not looking hard enough? Using AWS and spot instances is incredibly more efficient, which begs for the need of dynamically adding more nodes while the cluster is up, yet everything I've found so far seems to indicate it isn't supported yet. But yet here I am with 1.5 and it at least appears to be working. Am I missing something? On Tue, Nov 24, 2015 at 4:40 PM, Dillian Murphey <crackshotm...@gmail.com> wrote: > What's the current status on adding slaves to a running cluster? I want > to leverage spark-ec2 and autoscaling groups. I want to launch slaves as > spot instances when I need to do some heavy lifting, but I don't want to > bring down my cluster in order to add nodes. > > Can this be done by just running start-slave.sh?? > > What about using Mesos? > > I just want to create an AMI for a slave and on some trigger launch it and > have it automatically add itself to the cluster. > > thanks >
Adding more slaves to a running cluster
What's the current status on adding slaves to a running cluster? I want to leverage spark-ec2 and autoscaling groups. I want to launch slaves as spot instances when I need to do some heavy lifting, but I don't want to bring down my cluster in order to add nodes. Can this be done by just running start-slave.sh?? What about using Mesos? I just want to create an AMI for a slave and on some trigger launch it and have it automatically add itself to the cluster. thanks
