Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-10 Thread Meng Zhu
Thanks Ben. Some thoughts below: >From a scheduler's perspective the difference between the two models is: > > (1) expressing "how much more" you need > (2) expressing an offer "matcher" > > So: > > (1) covers the middle part of the demand quantity spectrum we currently > have: unsuppressed ->

Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-10 Thread Benjamin Mahler
I think we're agreed: -There are no schedulers modeling the existing per-agent time-based filters that mesos is tracking, and we shouldn't go in a direction that encourages frameworks to try to model and manage these. So, we should be very careful in considering something like CLEAR_FILTERS.

Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-10 Thread Benjamin Bannier
Hi Ben et al., I'd expect frameworks to *always* know how to accept or decline offers in general. More involved frameworks might know how to suppress offers. I don't expect that any framework models filters and their associated durations in detail (that's why I called them a Mesos

Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-05 Thread Benjamin Mahler
Thanks for bringing REQUEST_RESOURCES up for discussion, it's one of the mechanisms that we've been considering for further scaling pessimistic offers before we make the migration to optimistic offers. It's also been referred to as "demand" rather than "request", but for the sake of this

Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-04 Thread Meng Zhu
Hi Benjamin: Thanks for the great feedback. I like the idea of giving frameworks more meaningful and fined grained control over which filters to remove, especially this is likely to help adoption. For example, letting the framework send an optional agentID which instructs Mesos to only clear

Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-04 Thread Benjamin Bannier
Hi Meng, thanks for the proposal, I agree that the way these two aspects are currently entangled is an issue (e.g., for master/allocator performance reasons). At the same time, the workflow we currently expect frameworks to follow is conceptually not hard to grasp, (1) If framework has work

Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-03 Thread Meng Zhu
See my comments inline. On Mon, Dec 3, 2018 at 5:43 PM Vinod Kone wrote: > Thanks Meng for the explanation. > > I imagine most frameworks do not remember what stuff they filtered much > less figure out how previously filtered stuff can satisfy new operations. > That sounds complicated! >

Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-03 Thread Vinod Kone
Thanks Meng for the explanation. I imagine most frameworks do not remember what stuff they filtered much less figure out how previously filtered stuff can satisfy new operations. That sounds complicated! But I like your example. So a suggestion we could make to frameworks could be to use

Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-03 Thread Meng Zhu
Hi Vinod: Yeah, `CLEAR_FILTERS` sounds good. UNSUPPRESS should be used whenever currently suppressed framework wants to resume getting offers after a previous SUPPRESS call. As for `CLEAR_FILTERS`, the short (but not very useful) suggestion is to call it whenever the framework wants to clear

Re: New scheduler API proposal: unsuppress and clear_filter

2018-12-03 Thread Vinod Kone
Hi Meng, What would be the recommendation for framework authors on when to use UNSUPPRESS vs CLEAR_FILTER? Also, should it CLEAR_FILTERS instead of CLEAR_FILTER? On Mon, Dec 3, 2018 at 2:26 PM Meng Zhu wrote: > Hi: > > tl;dr: We are proposing to add two new V1 scheduler APIs: unsuppress and >