[
https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668964#comment-15668964
]
Carlo Curino commented on YARN-5881:
------------------------------------
[~seanpo03] thanks for raising this JIRA. This is indeed a very important
topic. Beside cloud settings, where clearly users care to get a fixed amount of
resources they paid for and not a relative amount, this is also important in
some on-prem settings, where certain production queues have fixed jobs running
on a schedule that must run (mission critical), and required fixed amount of
resources. In the reservation subsystem as you know we deal with this as
reservations (dynamic leaf queues), but the same concept should be applied more
generally to the queue structure. This will also be important to support
services with gang-semantics.
Beside the general engineering involved, I see a non-trivial issue related of
what to do when capacity fluctuate up/down. I assume you will have buffers to
accommodate modest fluctuations, but what happens if we loose enough capacity
to drop below the amount of absolutely configured queues. You could prioritize
certain queues over others, uniformly shrink all queues, etc.
Few questions to answers:
# do we allow for a mix of absolutely and relatively configured queues?
# how are capacity fluctuations managed?
# how is "over-capacity" resources distributed? (I can imagine to
instantaneously cast both capacity in the relative domain and perform standard
calculations)
# same as above for preemption actions.
# can we do this cleanly in CapacityScheduler? (as I mention in other JIRAs
the interaction between many of the tunables is become very unclear)
Overall I think this is very important, and even solving part of the problem
under some simplifying assumption might be ok.
> Enable configuration of queue capacity in terms of absolute resources
> ---------------------------------------------------------------------
>
> Key: YARN-5881
> URL: https://issues.apache.org/jira/browse/YARN-5881
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Sean Po
> Assignee: Sean Po
>
> Currently, Yarn RM supports the configuration of queue capacity in terms of a
> proportion to cluster capacity. In the context of Yarn being used as a public
> cloud service, it makes more sense if queues can be configured absolutely.
> This will allow administrators to set usage limits more concretely and
> simplify customer expectations for cluster allocation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
