Hi Guangya, My use case is actually trying to run Spark (in coarse grain mode) with multiple users. I wanted ways to better ensure fair scheduling across users. Spark provides very few primitives so I was hoping I could use Mesos to limit resources per user and control how the cluster is partitioned. For example, I may prefer that a Spark jobs share multiple machines without using all resources on a single machine for fault tolerance.
I'm also considering the case of running multiple frameworks. In this case, frameworks would have to coordinate to enforce user quotas and such. It seems that this would be better solved somewhere below the framework level. RJ > On Sep 8, 2015, at 11:47 PM, Guangya Liu <[email protected]> wrote: > > Hi RJ, > > I think that your final goal is that you want to use framework running on top > of mesos to execute some tasks. Such logic should be in the framework part. > The netflix open sourced a framework scheduler library named as fenzo, you > may want to take a look at this one to see if it can help you. > > http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-for-apache-mesos.html > https://github.com/Netflix/Fenzo > > Thanks, > > Guangya > > Date: Tue, 8 Sep 2015 23:09:36 -0500 > Subject: Re: Setting maximum per-node resources in offers > From: [email protected] > To: [email protected] > > Thanks, Klaus. > > I think I was probably misunderstanding the role of the allocator in Mesos > versus the scheduler in the framework sitting on top of Mesos. Probably out > of scope for Mesos to divide up resources as I was suggesting. > > On Tue, Sep 8, 2015 at 10:48 PM, Klaus Ma <[email protected]> wrote: > If it's the only framework, you will receive all nodes from Mesos as offers. > You can re-schedule those resources to run tasks on each node. > > > On 2015年09月09日 03:03, RJ Nowling wrote: > Hi all, > > I have a smallish cluster with a lot of cores and RAM per node. I want to > support multiple users so I'd like to set up Mesos to provide a maximum of 8 > cores per node in the resource offers. Resource offers should include > multiple nodes to reach the requirements of the user. For example, if the > user requests 32 cores, I would like 8 cores from each of 4 nodes. > > Is this possible? Or can someone suggest alternatives? > > Thanks, > RJ > > -- > Klaus Ma (马达), PMP® | http://www.cguru.net > >
