I would like to propose one alternative MVP proposal for the actual quota
implementation.
Instead of making changes to the allocator we could have an allocator
agnostic “Quota manager” which builds on top the existing dynamic
reservations.
Beyond MVP, we would still allow for allocator based implementations for
more complex quota mechanism, but this quota manager could also offer quota
support for allocators which are not aware of quota (TBD).
General Flow:
- Quota manager leverages dynamic reservations to fulfill quota requests.
It basically continuously tries to match the desired state (quota requests)
with the actual state (dynamic reservations).
- it receives quota requests from master
- using (approximate) slave usage information it selects target slave/agent
- requests to dynamic reservation (HTTP endpoint?), if fails retry
- Master should notify QM of slave failures and reregistrations
- in case of slave failures needs to request new reservation
- in case of slave reregistrations might need to unreserve
Required state of “Quota Manager”
- current dynamic reservations
- current granted quota
- approximate slave available resources (in order to decide on which slave
to /reserve, but doesn’t matter if slightly out of sync -> in worst case
/reserve fails and we have to retry)
Failover
- current granted quota are persisted in registry by master
- current dynamic reservations are reconstructed from slave reregistration
Master needs to propagate information about slave failures to Quota
Manager, as it might need to create new dynamic reservation in that case
Advantages compared to allocator based implementation:
- no need to change allocator(s) for MVP
- quota support for external quota-agnostic allocators
- re-using existing mechanisms
- minimal implementation effort for MVP
- almost free support for quota chunks (even in MVP) as dynamic
reservations are per slave
Disadvantages
- allocator based quota implementation still needed for more elaborate
implementations
- as dynamic reservations do not account towards fair share, so wouldn’t
quota based on this implementation. In my opinion this is not a real
problem as a) we did not really define the semantics of quota and b) fair
share is a allocator internal (i.e. DRF internal) notion so other allocator
implementations are free to do that differently anyhow.
Looking forward to feedback!
Jörg
On Thu, Jul 9, 2015 at 11:52 PM, Tomás Senart <[email protected]> wrote:
> What about "Global Reservations"?
>
> On Thu, Jul 9, 2015 at 3:25 PM, Marco Massenzio <[email protected]>
> wrote:
>
> > I've added my twocent in the doc - my vote goes for "Guaranteed
> Allocation"
> > - not as catchy as "Quota" (and will make classes' naming a challenge!)
> but
> > maybe more helpful in the long-term.
> >
> > Anyone has a better suggestion, please do... I can't really say I'm
> > super-excited by Guaranteed Allocation myself!
> >
> > *Marco Massenzio*
> > *Distributed Systems Engineer*
> >
> > On Thu, Jul 9, 2015 at 1:48 AM, Alex Rukletsov <[email protected]>
> > wrote:
> >
> > > And you're not the only one who were confused by the terminology! One
> of
> > > the alternatives that didn't make it to the public doc was
> "cluster-wide
> > > dynamic reservations". The reason we preferred "quota" to " ...
> > > reservation" is because the latter is already overloaded with meanings
> in
> > > Mesos world (static reservations, dynamic reservations). I have hoped
> the
> > > Terminology section would have helped to avoid the confusion, but I see
> > it
> > > doesn't. We'll think about how we can solve the problem, we definitely
> > > don't want to create one more "libprocess process represented as a
> thread
> > > in an OS process" ; ).
> > >
> > > I see your point regarding authorization, you're not alone here either
> :
> > ).
> > > Some folks mentioned that the lack of authz is a blocker and will
> prevent
> > > them from upgrading the cluster. I would propose to treat MVP as
> > > experimental feature: use it at your own risk or disable endpoints
> > related
> > > to quota and hence the entire feature. Does it make sense?
> > >
> > > On Wed, Jul 8, 2015 at 7:10 PM, James Peach <[email protected]> wrote:
> > >
> > > >
> > > > > On Jul 4, 2015, at 3:15 AM, Alex Rukletsov <[email protected]>
> > > wrote:
> > > > >
> > > > > Folks,
> > > > >
> > > > > Jörg and I are working on adding *quota* support to Mesos. Quota
> can
> > be
> > > > > described as cluster-wide dynamic reservation. I would like to
> share
> > > the
> > > > > design doc [1] to gather community feedback early in the design
> > phase.
> > > >
> > > > The most confusing part of this document to me was the 'quota'
> > > > terminology. Quotas normally refer to administrative limits (esp.
> disk
> > > > quotas with hard and soft limits), not reserving resources. Since
> what
> > > you
> > > > are describing is an extension to the resource reservation system, it
> > > would
> > > > be clearer if it was described in those terms.
> > > >
> > > > I was also concerned that access control / authorization is not
> planned
> > > > for the initial implementation. I think that if Mesos is to have an
> > > > authorization policy, it should be applied uniformly following the
> > > > principle of least surprise.
> > > >
> > > > > The doc is work in progress, especially the part related to quota
> > > support
> > > > > in the allocator. We think we can start working on adding quota
> > support
> > > > to
> > > > > Mesos Master while fleshing out the design for how quota is handled
> > by
> > > > the
> > > > > built-in allocator.
> > > > >
> > > > > While working on the design, we faced some challenges and design
> > > > questions.
> > > > > One of them is what decisions should be deferred to allocator and
> > what
> > > > can
> > > > > be decided by the Master. We elaborate on this in the doc.
> > > > >
> > > > > Looking forward to your feedback!
> > > > >
> > > > > [1]:
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/16iRNmziasEjVOblYp5bbkeBZ7pnjNlaIzPQqMTHQ-9I/edit?usp=sharing
> > > >
> > > >
> > >
> >
>
