I see. This is good to know for current development work. Thanks for clarifying, Guangya and Kevin.
Elizabeth Lingg > On Jun 22, 2016, at 3:02 AM, Guangya Liu <gyliu...@gmail.com> wrote: > > Hi Elizabeth, > > Just FYI, there is a JIRA tracing the resource revocation here > https://issues.apache.org/jira/browse/MESOS-4967 > > And I'm also working on the short term solution of excluding the scarce > resources from allocator (https://reviews.apache.org/r/48906/), with this > feature and Kevin's GPU_RESOURCES capability, the mesos can handle scarce > resources well. > > Thanks, > > Guangya > > On Wed, Jun 22, 2016 at 4:45 AM, Kevin Klues <klue...@gmail.com> wrote: > >> As an FYI, preliminary support to work around this issue for GPUs will >> appear in the 1.0 release >> https://reviews.apache.org/r/48914/ >> >> This doesn't solve the problem of scarce resources in general, but it >> will at least keep non-GPU workloads from starving out GPU-based >> workloads on GPU capable machines. The downside of this approach is >> that only GPU aware frameworks will be able to launch stuff on GPU >> capable machines (meaning some of their resources could go unused >> unnecessarily). We decided this tradeoff is acceptable for now. >> >> Kevin >> >> On Tue, Jun 21, 2016 at 1:40 PM, Elizabeth Lingg >> <elizabeth_li...@apple.com> wrote: >>> Thanks, looking forward to discussion and review on your document. The >> main use case I see here is that some of our frameworks will want to >> request the GPU resources, and we want to make sure that those frameworks >> are able to successfully launch tasks on agents with those resources. We >> want to be certain that other frameworks that do not require GPU’s will not >> request all other resources on those agents (i.e. cpu, disk, memory) which >> would mean the GPU resources are not allocated and the frameworks that >> require them will not receive them. As Ben Mahler mentioned, "(2) Because >> we do not have revocation yet, if a framework decides to consume the >> non-GPU resources on a GPU machine, it will prevent the GPU workloads from >> running!” This will occur for us in clusters where we have higher >> utilization as well as different types of workloads running. Smart task >> placement then becomes more relevant (i.e. we want to be able to schedule >> with scarce resources successfully and we may have considerations like not >> scheduling too many I/O bound workloads on a single host or more stringent >> requirements for scheduling persistent tasks). >>> >>> Elizabeth Lingg >>> >>> >>> >>>> On Jun 20, 2016, at 7:24 PM, Guangya Liu <gyliu...@gmail.com> wrote: >>>> >>>> Had some discussion with Ben M, for the following two solutions: >>>> >>>> 1) Ben M: Create sub-pools of resources based on machine profile and >>>> perform fair sharing / quota within each pool plus a framework >>>> capability GPU_AWARE >>>> to enable allocator filter out scarce resources for some frameworks. >>>> 2) Guangya: Adding new sorters for non scarce resources plus a framework >>>> capability GPU_AWARE to enable allocator filter out scarce resources for >>>> some frameworks. >>>> >>>> Both of the above two solutions are meaning same thing and there is no >>>> difference between those two solutions: Create sub-pools of resources >> will >>>> need to introduce different sorters for each sub-pools, so I will merge >>>> those two solutions to one. >>>> >>>> Also had some dicsussion with Ben for AlexR's solution of implementing >>>> "requestResource", this API should be treated as an improvement to the >>>> issues of doing resource allocation pessimistically. (e.g. we >> offer/decline >>>> the GPUs to 1000 frameworks before offering it to the GPU framework that >>>> wants it). And the "requestResource" is providing *more information* to >>>> mesos. Namely, it gives us awareness of demand. >>>> >>>> Even though for some cases, we can use the "requestResource" to get all >> of >>>> the scarce resources, and then once those scarce resources are in use, >> then >>>> the WDRF sorter will sorter non scarce resources as normal, but the >> problem >>>> is that we cannot guarantee that the framework which have >> "requestResource" >>>> can always consume all of the scarce resources before those scarce >> resource >>>> allocated to other frameworks. >>>> >>>> I'm planning to draft a document based on solution 1) "Create sub-pools" >>>> for the long term solution, any comments are welcome! >>>> >>>> Thanks, >>>> >>>> Guangya >>>> >>>> On Sat, Jun 18, 2016 at 11:58 AM, Guangya Liu <gyliu...@gmail.com> >> wrote: >>>> >>>>> Thanks Du Fan. So you mean that we should have some clear rules in >>>>> document or somewhere else to tell or guide cluster admin which >> resources >>>>> should be classified as scarce resources, right? >>>>> >>>>> On Sat, Jun 18, 2016 at 2:38 AM, Du, Fan <fan...@intel.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On 2016/6/17 7:57, Guangya Liu wrote: >>>>>> >>>>>>> @Fan Du, >>>>>>> >>>>>>> Currently, I think that the scarce resources should be defined by >> cluster >>>>>>> admin, s/he can specify those scarce resources via a flag when master >>>>>>> start >>>>>>> up. >>>>>>> >>>>>> >>>>>> This is not what I mean. >>>>>> IMO, it's not cluster admin's call to decide what resources should be >>>>>> marked as scarce , they can carry out the operation, but should be >> advised >>>>>> on based on the clear rule: to what extend the resource is scarce >> compared >>>>>> with other resources, and it will affect wDRF by causing starvation >> for >>>>>> frameworks which holds scarce resources, that's my point. >>>>>> >>>>>> To my best knowledge here, a quantitative study of how wDRF behaves in >>>>>> scenario of one/multiple scarce resources first will help to verify >> the >>>>>> proposed approach, and guide the user of this functionality. >>>>>> >>>>>> >>>>>> >>>>>> Regarding to the proposal of generic scarce resources, do you have any >>>>>>> thoughts on this? I can see that giving framework developers the >> options >>>>>>> of >>>>>>> define scarce resources may bring trouble to mesos, it is better to >> let >>>>>>> mesos define those scarce but not framework developer. >>>>>>> >>>>>> >>>>> >>> >> >> >> >> -- >> ~Kevin >>
