> > > Tim What metrics do you watch for demand and supply for Spark? Do you just > watch node resources or you actually look at some Spark JMX stats?
Sorry about the confusion. Here, I was referring to us autoscaling within our own framework in Mantis, not for Spark. The scheduler knows the demand Vs supply from its own assignments and therefore is able to know the idle resources, etc. You can find some details in http://www.slideshare.net/spodila/aws-reinvent-2014-talk-scheduling-using-apache-mesos-in-the-cloud On Fri, Jun 5, 2015 at 1:37 AM, Tim Chen <[email protected]> wrote: > Hi Sharma, > > > What metrics do you watch for demand and supply for Spark? Do you just > watch node resources or you actually look at some Spark JMX stats? > > Tim > > On Thu, Jun 4, 2015 at 10:35 PM, Sharma Podila <[email protected]> > wrote: > >> We Autoscale our Mesos cluster in EC2 from within our framework. Scaling >> up can be easy via watching demand Vs supply. However, scaling down >> requires bin packing the tasks tightly onto as few servers as possible. >> Do you have any specific ideas on how you would leverage Mantis/Mesos for >> Spark based jobs? Fenzo, the scheduler part of Mantis, could be another >> point of leverage, which could give a framework the ability to autoscale >> the cluster among other benefits. >> >> >> >> On Thu, Jun 4, 2015 at 1:06 PM, Dmitry Goldenberg < >> [email protected]> wrote: >> >>> Thanks, Vinod. I'm really interested in how we could leverage something >>> like Mantis and Mesos to achieve autoscaling in a Spark-based data >>> processing system... >>> >>> On Jun 4, 2015, at 3:54 PM, Vinod Kone <[email protected]> wrote: >>> >>> Hey Dmitry. At the current time there is no built-in support for Mesos >>> to autoscale nodes in the cluster. I've heard people (Netflix?) do it out >>> of band on EC2. >>> >>> On Thu, Jun 4, 2015 at 9:08 AM, Dmitry Goldenberg < >>> [email protected]> wrote: >>> >>>> A Mesos noob here. Could someone point me at the doc or summary for the >>>> cluster autoscaling capabilities in Mesos? >>>> >>>> Is there a way to feed it events and have it detect the need to bring >>>> in more machines or decommission machines? Is there a way to receive >>>> events back that notify you that machines have been allocated or >>>> decommissioned? >>>> >>>> Would this work within a certain set of >>>> "preallocated"/pre-provisioned/"stand-by" machines or will Mesos go and >>>> grab machines from the cloud? >>>> >>>> What are the integration points of Apache Spark and Mesos? What are >>>> the true advantages of running Spark on Mesos? >>>> >>>> Can Mesos autoscale the cluster based on some signals/events coming out >>>> of Spark runtime or Spark consumers, then cause the consumers to run on the >>>> updated cluster, or signal to the consumers to restart themselves into an >>>> updated cluster? >>>> >>>> Thanks. >>>> >>> >>> >> >
