Bikas Saha commented on YARN-4108:

These problems will be hard to solve without involving the scheduler in the 
decision cycle. The preemption policy can determine how much to preempt from a 
queue at a macro level. But the actual containers to preempt would be selected 
by the scheduler.
That is where using the global node picture will help. For a given container 
request, if we can scan its nodes (if any) and make  either an allocation or 
preemption decision. 
Else, if we are doing container allocation on node heartbeat, then just like 
delay scheduling logic, we can mark a node for preemption but not preempt it 
and associate that node with the container request for which preemption is 
needed (request.nodeToPreempt). And we can cycle through all nodes like this 
and change the request->node association when we find better nodes to preempt. 
After cycling through all nodes, if when we again reach a node such that it 
matches the request.nodeToPreempt then we can execute the decision of actually 
preempting the node. If there are no nodes that can satisfy the request (e.g. 
request wants node A but preemptedQueue has no containers on node A) then 
scheduler should be able to callback to the preemption module and notify it so 
that some other queue can be picked to preempt.

> CapacityScheduler: Improve preemption to preempt only those containers that 
> would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> This is sibling JIRA for YARN-2154. We should make sure container preemption 
> is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality 
> (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I 
> don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), 
> cross applicaiton preemption (such as priority-based (YARN-1963) / 
> fairness-based (YARN-3319)).

This message was sent by Atlassian JIRA

Reply via email to