Folks, We have updated the design doc [1] based on numerous comments on the v1.
There is one substantial design question left: what entity should decide whether a quota can be granted: the Master, an allocator, or a separate "Quota Manager". I have described pros and cons of each in the doc, please look inside for more information. Apart of that, I have made the following amendments: * Clarified the absence of authz and how this mitigate that with firewall; * Updated confusing naming: quota is a pair of guaranteed resource and limit; * Added a safety design principal; * Updated section explaining granting quota decision; * Updated QuotaInfo protobuf; * Added an alternative quota implementation via dynamic reservations; * Updated HTTP api to be more REST-like; * Added more ideas to the wip Allocator section. Please have a look and check whether your concerns have been addressed. [1]: https://docs.google.com/document/d/16iRNmziasEjVOblYp5bbkeBZ7pnjNlaIzPQqMTHQ-9I/edit# On Fri, Jul 10, 2015 at 8:52 AM, Jörg Schad <[email protected]> wrote: > I would like to propose one alternative MVP proposal for the actual quota > implementation. > > Instead of making changes to the allocator we could have an allocator > agnostic “Quota manager” which builds on top the existing dynamic > reservations. > > Beyond MVP, we would still allow for allocator based implementations for > more complex quota mechanism, but this quota manager could also offer quota > support for allocators which are not aware of quota (TBD). > > > General Flow: > > - Quota manager leverages dynamic reservations to fulfill quota requests. > It basically continuously tries to match the desired state (quota requests) > with the actual state (dynamic reservations). > > - it receives quota requests from master > > - using (approximate) slave usage information it selects target slave/agent > > - requests to dynamic reservation (HTTP endpoint?), if fails retry > > - Master should notify QM of slave failures and reregistrations > > - in case of slave failures needs to request new reservation > > - in case of slave reregistrations might need to unreserve > > Required state of “Quota Manager” > > - current dynamic reservations > > - current granted quota > > - approximate slave available resources (in order to decide on which slave > to /reserve, but doesn’t matter if slightly out of sync -> in worst case > /reserve fails and we have to retry) > > Failover > > - current granted quota are persisted in registry by master > > - current dynamic reservations are reconstructed from slave reregistration > > Master needs to propagate information about slave failures to Quota > Manager, as it might need to create new dynamic reservation in that case > > > Advantages compared to allocator based implementation: > > - no need to change allocator(s) for MVP > > - quota support for external quota-agnostic allocators > > - re-using existing mechanisms > > - minimal implementation effort for MVP > > - almost free support for quota chunks (even in MVP) as dynamic > reservations are per slave > > Disadvantages > > - allocator based quota implementation still needed for more elaborate > implementations > > - as dynamic reservations do not account towards fair share, so wouldn’t > quota based on this implementation. In my opinion this is not a real > problem as a) we did not really define the semantics of quota and b) fair > share is a allocator internal (i.e. DRF internal) notion so other allocator > implementations are free to do that differently anyhow. > > Looking forward to feedback! > Jörg > > On Thu, Jul 9, 2015 at 11:52 PM, Tomás Senart <[email protected]> wrote: > > > What about "Global Reservations"? > > > > On Thu, Jul 9, 2015 at 3:25 PM, Marco Massenzio <[email protected]> > > wrote: > > > > > I've added my twocent in the doc - my vote goes for "Guaranteed > > Allocation" > > > - not as catchy as "Quota" (and will make classes' naming a challenge!) > > but > > > maybe more helpful in the long-term. > > > > > > Anyone has a better suggestion, please do... I can't really say I'm > > > super-excited by Guaranteed Allocation myself! > > > > > > *Marco Massenzio* > > > *Distributed Systems Engineer* > > > > > > On Thu, Jul 9, 2015 at 1:48 AM, Alex Rukletsov <[email protected]> > > > wrote: > > > > > > > And you're not the only one who were confused by the terminology! One > > of > > > > the alternatives that didn't make it to the public doc was > > "cluster-wide > > > > dynamic reservations". The reason we preferred "quota" to " ... > > > > reservation" is because the latter is already overloaded with > meanings > > in > > > > Mesos world (static reservations, dynamic reservations). I have hoped > > the > > > > Terminology section would have helped to avoid the confusion, but I > see > > > it > > > > doesn't. We'll think about how we can solve the problem, we > definitely > > > > don't want to create one more "libprocess process represented as a > > thread > > > > in an OS process" ; ). > > > > > > > > I see your point regarding authorization, you're not alone here > either > > : > > > ). > > > > Some folks mentioned that the lack of authz is a blocker and will > > prevent > > > > them from upgrading the cluster. I would propose to treat MVP as > > > > experimental feature: use it at your own risk or disable endpoints > > > related > > > > to quota and hence the entire feature. Does it make sense? > > > > > > > > On Wed, Jul 8, 2015 at 7:10 PM, James Peach <[email protected]> > wrote: > > > > > > > > > > > > > > > On Jul 4, 2015, at 3:15 AM, Alex Rukletsov <[email protected]> > > > > wrote: > > > > > > > > > > > > Folks, > > > > > > > > > > > > Jörg and I are working on adding *quota* support to Mesos. Quota > > can > > > be > > > > > > described as cluster-wide dynamic reservation. I would like to > > share > > > > the > > > > > > design doc [1] to gather community feedback early in the design > > > phase. > > > > > > > > > > The most confusing part of this document to me was the 'quota' > > > > > terminology. Quotas normally refer to administrative limits (esp. > > disk > > > > > quotas with hard and soft limits), not reserving resources. Since > > what > > > > you > > > > > are describing is an extension to the resource reservation system, > it > > > > would > > > > > be clearer if it was described in those terms. > > > > > > > > > > I was also concerned that access control / authorization is not > > planned > > > > > for the initial implementation. I think that if Mesos is to have an > > > > > authorization policy, it should be applied uniformly following the > > > > > principle of least surprise. > > > > > > > > > > > The doc is work in progress, especially the part related to quota > > > > support > > > > > > in the allocator. We think we can start working on adding quota > > > support > > > > > to > > > > > > Mesos Master while fleshing out the design for how quota is > handled > > > by > > > > > the > > > > > > built-in allocator. > > > > > > > > > > > > While working on the design, we faced some challenges and design > > > > > questions. > > > > > > One of them is what decisions should be deferred to allocator and > > > what > > > > > can > > > > > > be decided by the Master. We elaborate on this in the doc. > > > > > > > > > > > > Looking forward to your feedback! > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/16iRNmziasEjVOblYp5bbkeBZ7pnjNlaIzPQqMTHQ-9I/edit?usp=sharing > > > > > > > > > > > > > > > > > > > >
