[ 
https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083223#comment-15083223
 ] 

Karthik Kambatla edited comment on YARN-1011 at 1/5/16 3:28 PM:
----------------------------------------------------------------

We would run an opportunistic container on a node only if the actual 
utilization is less than the allocation by a margin bigger than the allocation 
of said opportunistic container. We reactively preempt the opportunistic 
container if the actual utilization goes over a threshold. To address spikes in 
usage where our reactive measures are too slow to kick in, we run the 
opportunistic containers at a strictly lower priority. 

bq. the app got opportunistic containers and their perf wasnt the same as 
normal containers - so it ran slower. 
As soon as we realize the perf is slower because the node has higher usage than 
we had anticipated, we preempt the container and retry allocation (guaranteed 
or opportunistic depending on the new cluster state). So, it shouldn't run 
slower for longer than our monitoring interval. Is this assumption okay? 

bq. However, things get complicated because a node with an opportunistic 
container may continue to run its normal containers while space frees up for 
guaranteed capacity on other nodes.
The opportunistic container will continue to run on this node so long as it is 
getting the resources it needs. If there is any sort of resource contention, it 
is preempted and is up for allocation on one of the free nodes. 

bq. This would require that the system upgrade opportunistic containers in the 
same order as it would allocate containers.
bq. IMO, the NM cannot make a local choice about upgrading its opportunistic 
containers because this is effectively a resource allocation decision and only 
the RM has the info to do that.
The RM schedules the next highest priority "task" for which it couldn't find a 
guaranteed container as an opportunistic container. This task continues to run 
as long as it is not getting enough resources. If there is no resource 
contention, the task continues to run. If guaranteed resources free up on the 
node it is running, isn't it fair to promote the container to Guaranteed. After 
all, if the resources unused were not hidden behind other containers' 
allocation and actually available as guaranteed capacity on that node 
initially, the RM would just have scheduled a guaranteed container in the first 
place.

I should probably clarify that the proposal here targets those cases where 
users' estimates are significantly off reality and there are enough free 
resources per node to run additional task(s) without causing any resource 
contention. Even though this is the norm, we want to guard against spikes in 
usage to avoid perf regressions. In practice, I expect admins to come up with a 
reasonable threshold for over-subscription: e.g. 0.8 - we use only 
oversubscribe upto 80% of capacity advertised through 
{{yarn.nodemanger.resource.*}}. Thinking more about this, this threshold should 
have an upper limit - 0.95? 



was (Author: kasha):
We would run an opportunistic container on a node only if the actual 
utilization is less than the allocation by a margin bigger than the allocation 
of said opportunistic container. We reactively preempt the opportunistic 
container if the actual utilization goes over a threshold. To address spikes in 
usage where our reactive measures are too slow to kick in, we run the 
opportunistic containers at a strictly lower priority. 

bq. the app got opportunistic containers and their perf wasnt the same as 
normal containers - so it ran slower. 
As soon as we realize the perf is slower because the node has higher usage than 
we had anticipated, we preempt the container and retry allocation (guaranteed 
or opportunistic depending on the new cluster state). So, it shouldn't run 
slower for longer than our monitoring interval. Is this assumption okay? 

bq. However, things get complicated because a node with an opportunistic 
container may continue to run its normal containers while space frees up for 
guaranteed capacity on other nodes.
The opportunistic container will continue to run on this node so long as it is 
getting the resources it needs. If there is any sort of resource contention, it 
is preempted and is up for allocation on one of the free nodes. 

bq. This would require that the system upgrade opportunistic containers in the 
same order as it would allocate containers.
bq. IMO, the NM cannot make a local choice about upgrading its opportunistic 
containers because this is effectively a resource allocation decision and only 
the RM has the info to do that.
The RM schedules the next highest priority "task" for which it couldn't find a 
guaranteed container as an opportunistic container. This task continues to run 
as long as it is not getting enough resources. If there is no resource 
contention, the task continues to run. If guaranteed resources free up on the 
node it is running, isn't it fair to promote the container to Guaranteed. After 
all, if the resources unused were not hidden behind other containers' 
allocation and actually available as guaranteed capacity on that node 
initially, the RM would just have scheduled a guaranteed container in the first 
place.

I should probably clarify that the proposal here targets those cases where 
users' estimates are significantly off reality and there are enough free 
resources per node to run additional task(s) without causing any resource 
contention. Even though this is the norm, we want to guard against spikes in 
usage to avoid perf regressions. In practice, I expect admins to come up with a 
reasonable threshold for over-subscription: e.g. 0.8 - we use only 
oversubscribe upto 80% of capacity advertised through 
{{yarn.nodemanger.resource.*}}


> [Umbrella] Schedule containers based on utilization of currently allocated 
> containers
> -------------------------------------------------------------------------------------
>
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are 
> utilized.
> RM can, and should, get to a point where it measures utilization of allocated 
> containers and, if appropriate, allocate more (speculative?) containers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to