MENG DING commented on YARN-4108:

Hi, [~leftnoteasy]

I also feel that the logic you proposed is a good staring point overall. Just 
want to confirm that I understand it correctly. For pending asks with hard 
locality requirements, I think this logic works the best. However, for other 
pending asks, are we able to achieve optimal preemption (i.e. sufficiently 
preemptable resources with the lowest cost of preemption as per [~jlowe])? For 
example, just because {{node.available + preemptable > 
application.next_request}} doesn't necessarily mean that the preemption cost is 
the lowest on this node. Maybe we need to have a combination of reservation 
continuous looking + delayed scheduling mechanism to ensure that we have done 
calculation of preemption cost on enough hosts for the pending ask. But then I 
feel this approach might be too expensive ...

> CapacityScheduler: Improve preemption to preempt only those containers that 
> would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> This is sibling JIRA for YARN-2154. We should make sure container preemption 
> is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality 
> (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I 
> don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), 
> cross applicaiton preemption (such as priority-based (YARN-1963) / 
> fairness-based (YARN-3319)).

This message was sent by Atlassian JIRA

Reply via email to