Hi RJ, please check my answers in line. Thanks,
Guangya On Wed, Sep 9, 2015 at 1:24 PM, RJ Nowling <[email protected]> wrote: > Hi Guangya, > > My use case is actually trying to run Spark (in coarse grain mode) with > multiple users. I wanted ways to better ensure fair scheduling across > users. Spark provides very few primitives so I was hoping I could use Mesos > to limit resources per user and control how the cluster is partitioned. For > example, I may prefer that a Spark jobs share multiple machines without > using all resources on a single machine for fault tolerance. > For this scenario, you may want to schedule those offered resource again in framework level, you can leverage fenzo or what ever to enhance the scheduler part for spark to achieve your goal. > > I'm also considering the case of running multiple frameworks. In this > case, frameworks would have to coordinate to enforce user quotas and such. > It seems that this would be better solved somewhere below the framework > level. > For this scenario, there is an epic for "quota management" which can fill your requirement but it is still undergoing and not available now. epic: https://issues.apache.org/jira/browse/MESOS-1791 Design doc: https://docs.google.com/document/d/16iRNmziasEjVOblYp5bbkeBZ7pnjNlaIzPQqMTHQ-9I/edit?pli=1#heading=h.9g7fqjh6652v > > RJ > > > > On Sep 8, 2015, at 11:47 PM, Guangya Liu <[email protected]> wrote: > > Hi RJ, > > I think that your final goal is that you want to use framework running on > top of mesos to execute some tasks. Such logic should be in the framework > part. The netflix open sourced a framework scheduler library named as > fenzo, you may want to take a look at this one to see if it can help you. > > > http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-for-apache-mesos.html > https://github.com/Netflix/Fenzo > > Thanks, > > Guangya > > ------------------------------ > Date: Tue, 8 Sep 2015 23:09:36 -0500 > Subject: Re: Setting maximum per-node resources in offers > From: [email protected] > To: [email protected] > > Thanks, Klaus. > > I think I was probably misunderstanding the role of the allocator in Mesos > versus the scheduler in the framework sitting on top of Mesos. Probably > out of scope for Mesos to divide up resources as I was suggesting. > > On Tue, Sep 8, 2015 at 10:48 PM, Klaus Ma <[email protected]> wrote: > > If it's the only framework, you will receive all nodes from Mesos as > offers. You can re-schedule those resources to run tasks on each node. > > > On 2015年09月09日 03:03, RJ Nowling wrote: > > Hi all, > > I have a smallish cluster with a lot of cores and RAM per node. I want to > support multiple users so I'd like to set up Mesos to provide a maximum of > 8 cores per node in the resource offers. Resource offers should include > multiple nodes to reach the requirements of the user. For example, if the > user requests 32 cores, I would like 8 cores from each of 4 nodes. > > Is this possible? Or can someone suggest alternatives? > > Thanks, > RJ > > > -- > Klaus Ma (马达), PMP® | http://www.cguru.net > > >
