Re: New scheduler API proposal: unsuppress and clear_filter
Hi Benjamin: Thanks for the great feedback. I like the idea of giving frameworks more meaningful and fined grained control over which filters to remove, especially this is likely to help adoption. For example, letting the framework send an optional agentID which instructs Mesos to only clear filters on that agent might help a task launch with agent constraint. However, when it comes to framework sent desired resource profiles, we should give more thoughts. There is always the question that to what degree do we support the various meta-data in the resource schema. I feel the current schema is too complex for expressing resource needs, let alone respecting it in the allocator (even just for the purpose of removing filters). We probably want to first introduce a more concise format (such as resourceQuantity) for all purposes of specifying desired resource profiles (clear filters, quota guarantee, min_allocatable_resources and etc) and start from there. I suggest to just add the optional agentID atm and we can always add support for specifying resource requirements in the future. And since its semantic is far away from "requesting resources", I suggest keeping the name of CLEAR(or REMOVE)_FILTERS. What do you think? -Meng On Tue, Dec 4, 2018 at 1:50 AM Benjamin Bannier < benjamin.bann...@mesosphere.io> wrote: > Hi Meng, > > thanks for the proposal, I agree that the way these two aspects are > currently entangled is an issue (e.g., for master/allocator performance > reasons). At the same time, the workflow we currently expect frameworks to > follow is conceptually not hard to grasp, > > (1) If framework has work then > (i) put framework in unsuppressed state, > (ii) decline not matching offers with a long filter duration. > (2) If an offer matches, accept. > (3) If there is no more work, suppress. GOTO (1). > > Here the framework does not need to track its filters across allocation > cycles (they are an unexposed implementation detail of the hierarchical > allocator anyway) which e.g., allows metaschedulers like Marathon or Apache > Aurora to decouple the scheduling of different workloads. A downside of > this interface is that > > * there is little incentive for frameworks to use SUPPRESS in addition to > filters, and > * unsupression is all-or-nothing, forcing the master to send potentially > all unused resources to one framework, even if it is only interested in a > fraction. This can cause, at least temporal, non-optimal allocation > behavior. > > It seems to me that even though adding UNSUPPRESS and CLEAR_FILTERS would > give frameworks more control, it would only be a small improvement. In > above framework workflow we would allow a small improvement if the > framework knows that a new workload matches a previously running workflow > (i.e., it can infer that no filters for the resources it is interested in > is active) so that it can issue UNSUPPRESS instead of CLEAR_FILTERS. > Incidentally, there seems little local benefit for frameworks to use these > new calls as they’d mostly help the master and I’d imagine we wouldn’t want > to imply that clearing filters would unsuppress the framework. This seems > too little to me, and we run the danger that frameworks would just always > pair UNSUPPRESS and CLEAR_FILTERS (or keep using REVIVE) to simplify their > workflow. If we’d model the interface more along framework needs, there > would be clear benefit which would help adoption. > > A more interesting call for me would be REQUEST_RESOURCES. It maps very > well onto framework needs (e.g., “I want to launch a task requiring these > resources”), and clearly communicates a requirement to the master so that > it e.g., doesn’t need to remove all filters for a framework. It also seems > to fit the allocator model pretty well which doesn’t explicitly expose > filters. I believe implementing it should not be too hard if we'd restrict > its semantics to only communicate to the master that a framework _is > interested in a certain resource_ without promising that the framework > _will get them in any amount of time_ (i.e., no need to rethink DRF > fairness semantics in the hierarchical allocator). I also feel that if we > have REQUEST_RESOURCES we would have some freedom to perform further > improvements around filters in the master/allocator (e.g., filter > compatification, work around increasing the default filter duration, …). > > > A possible zeroth implementation for REQUEST_RESOURCES with the > hierarchical allocator would be to have it remove any filters containing > the requested resource and likely to unsuppress the framework. A > REQUEST_RESOURCES call would hold an optional resource and an optional > AgentID; the case where both are empty would map onto CLEAR_FILTERS. > > > That being said, it might still be useful to in the future expose a > low-level knob for framework allowing them to explicitly manage their > filters. > > > Cheers, > > Benjamin > > > On Dec 4, 2018, at 5:44 AM, Meng Zhu wrote:
Re: New scheduler API proposal: unsuppress and clear_filter
Hi Meng, thanks for the proposal, I agree that the way these two aspects are currently entangled is an issue (e.g., for master/allocator performance reasons). At the same time, the workflow we currently expect frameworks to follow is conceptually not hard to grasp, (1) If framework has work then (i) put framework in unsuppressed state, (ii) decline not matching offers with a long filter duration. (2) If an offer matches, accept. (3) If there is no more work, suppress. GOTO (1). Here the framework does not need to track its filters across allocation cycles (they are an unexposed implementation detail of the hierarchical allocator anyway) which e.g., allows metaschedulers like Marathon or Apache Aurora to decouple the scheduling of different workloads. A downside of this interface is that * there is little incentive for frameworks to use SUPPRESS in addition to filters, and * unsupression is all-or-nothing, forcing the master to send potentially all unused resources to one framework, even if it is only interested in a fraction. This can cause, at least temporal, non-optimal allocation behavior. It seems to me that even though adding UNSUPPRESS and CLEAR_FILTERS would give frameworks more control, it would only be a small improvement. In above framework workflow we would allow a small improvement if the framework knows that a new workload matches a previously running workflow (i.e., it can infer that no filters for the resources it is interested in is active) so that it can issue UNSUPPRESS instead of CLEAR_FILTERS. Incidentally, there seems little local benefit for frameworks to use these new calls as they’d mostly help the master and I’d imagine we wouldn’t want to imply that clearing filters would unsuppress the framework. This seems too little to me, and we run the danger that frameworks would just always pair UNSUPPRESS and CLEAR_FILTERS (or keep using REVIVE) to simplify their workflow. If we’d model the interface more along framework needs, there would be clear benefit which would help adoption. A more interesting call for me would be REQUEST_RESOURCES. It maps very well onto framework needs (e.g., “I want to launch a task requiring these resources”), and clearly communicates a requirement to the master so that it e.g., doesn’t need to remove all filters for a framework. It also seems to fit the allocator model pretty well which doesn’t explicitly expose filters. I believe implementing it should not be too hard if we'd restrict its semantics to only communicate to the master that a framework _is interested in a certain resource_ without promising that the framework _will get them in any amount of time_ (i.e., no need to rethink DRF fairness semantics in the hierarchical allocator). I also feel that if we have REQUEST_RESOURCES we would have some freedom to perform further improvements around filters in the master/allocator (e.g., filter compatification, work around increasing the default filter duration, …). A possible zeroth implementation for REQUEST_RESOURCES with the hierarchical allocator would be to have it remove any filters containing the requested resource and likely to unsuppress the framework. A REQUEST_RESOURCES call would hold an optional resource and an optional AgentID; the case where both are empty would map onto CLEAR_FILTERS. That being said, it might still be useful to in the future expose a low-level knob for framework allowing them to explicitly manage their filters. Cheers, Benjamin On Dec 4, 2018, at 5:44 AM, Meng Zhu wrote: > > See my comments inline. > > On Mon, Dec 3, 2018 at 5:43 PM Vinod Kone wrote: > >> Thanks Meng for the explanation. >> >> I imagine most frameworks do not remember what stuff they filtered much >> less figure out how previously filtered stuff can satisfy new operations. >> That sounds complicated! >> > > Frameworks do not need to remember what filters they currently have. Only > knowing > the resource profiles of the current vs. the previous operation would help > a lot. > But yeah, even this may be too much complexity. > >> >> But I like your example. So a suggestion we could make to frameworks could >> be to use CLEAR_FILTERS when they have new work, e.g., scale up/down, new >> app (they might want to use this even if they aren't suppressed!); and to >> use UNSUPPRESS when they are rescheduling old work? >> > > Yeah, these are the general guideline. > > I want to echo and reemphasize that CLEAR_FILTERS is orthogonal to > suppression. > Framework should consider clearing filters regardless of suppression. > > Ideally, when there is new different work, old irelavent filters should be > cleared. This helps > framework to get more offers and makes the allocator run faster (filter > could take up > bulk of the allocation time when they build up). On the flip side, calling > CLEAR_FILTERS too often > might also have performance implications (esp. if the master/allocator > actors are already
