Sunil G commented on YARN-4108:

Adding to the locality preemption improvement suggestion from [~jlowe]

While analyzing and preparing a POC for application priority preemption (lower 
priority apps to be preempted to give space for higher priority apps within a 
queue), we ran into similar lines of problem. There were few lower priority 
apps, and the demand from a higher priority app may be best served by selecting 
the to-be-preempted-containers from those nodes which are locally best for that 
higher priority app.
One of possible solution here is in selecting the apps/containers to preempt so 
that it can match the specific need of application. In ProportionalCPP, we 
consider the demand as a sum of resource demand from all needy apps from a 
queue which is less served. If we resolve/work on the demand per application 
level, then we can consider other characteristics such as node locality, 
user-limit etc from the potential to-be-preempted apps from target queue. 

But this needs a change in approach with existing one.

Coming to the approach from Wangda, overall it looks fine. But i feel we may 
end up killing or preempt some containers from all applications in queue (worst 
case). Now we select application to be preempted by low priroity or newly 
submitted etc. 
To be precise with an eg, If an app need to launch a container on node1, it may 
need to free up 4GB. This 4Gb will be fetched now within a node. So a high 
priority app also can loose a container (and some low priority container still 
running in other node). So point from Jason is very important, we may need to 
peek through nodes for coming to a scheduler allocation, And a best candidate 
node to be selected to preempt (a node where newly submitted app is running 
more OR low priority app is running more etc).

> CapacityScheduler: Improve preemption to preempt only those containers that 
> would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> This is sibling JIRA for YARN-2154. We should make sure container preemption 
> is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality 
> (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I 
> don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), 
> cross applicaiton preemption (such as priority-based (YARN-1963) / 
> fairness-based (YARN-3319)).

This message was sent by Atlassian JIRA

Reply via email to