[
https://issues.apache.org/jira/browse/YARN-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524129#comment-16524129
]
Haibo Chen commented on YARN-1013:
----------------------------------
To add a bit of context to YARN-1015 (or YARN-1011 as a whole), we are trying
to just handle G resource requests with their enforcement flag set to false.
This indicates to the scheduler that it can be satisfied with an Opportunistic
container to start with and the Opportunistic container will later be promoted
to Guaranteed container automatically by the scheduler. If the user does not
want O containers to start with, they set the enforcement flag to true. *O
resource requests are handled by existing OpportunisticAMProcessor.*
{quote}YARN-6794 randomly promotes O container even if user doesn't care about
container execution type.
{quote}
This is not quite right. YARN-6794 promotes O containers in the order they are
allocated. This is to ensure that users can treat YARN-1011 as
eager-scheduling.
Let's say the scheduler receives two requests in order, R1 (G, false) and R2
(G, true) (G is the resourceRequest execution type, and true/false is the the
enforcement flag). If there is no oversubscription, R1 will be satisfied with
a G container at t1 and R2 will be satisfied with a G container at t2 (t1 <
t2). What YARN-1011 does effectively, from the users' point of view, is that R1
will be satisfied with a O container at t0 (t0 < t1), be promoted at time t1,
and R2 will still be satisfied with a G container at t2. You see essentially,
R1 can run as O container from t0 to t1, but it would run as G container right
at the same moment, t1, as if oversubscription never takes place. YARN-6794
has to follow an order to promote O containers so that when there are multiple
(G, false) requests, it still maintains the same order in which the (G, false)
requests would have been satisfied with G containers, if oversubscription had
been turned off.
YARN-8179 is pretty straightforward, and it's probably not as straightforward
to do YARN-1013 with all the other CS features in play, but the fair scheduler
implementation (YARN-1015) tries to take into account of queue weight, resource
usage and application demand, so that Opportunistic resources are allocated the
same way as Guaranteed resources (fair share for Opportunistic resources as
well).
> CS should watch resource utilization of containers and allocate speculative
> containers if appropriate
> -----------------------------------------------------------------------------------------------------
>
> Key: YARN-1013
> URL: https://issues.apache.org/jira/browse/YARN-1013
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Arun C Murthy
> Assignee: Weiwei Yang
> Priority: Major
>
> CS should watch resource utilization of containers (provided by NM in
> heartbeat) and allocate speculative containers (at lower OS priority) if
> appropriate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
