Wangda Tan commented on YARN-4108:

Hi [~sunilg],

Thanks for looking at this,
bq. 1. With different PreemptionType, are we planning to handle preemption 
across queue, within queue (fifo/priority), within user etc? YARN-2009 was 
trying to handling the preemption within a queue adhering to priority. 
I planned to support preemption within leaf queue with this approach, but I 
think it needs more effort so may not be able to fit in the same JIRA. However, 
since this effort is in early stage, I don't want this blocks other 
enhancements to existing preemption implementation such as YARN-2009.

bq. >> I would see it will be a great help if flexibility is provided with some 
interface to sort containers eventhough a great deal of validation is done. 
I totally agree with you, I think flexibility of supporting different kinds of 
selecting/calculation algorithm is the P0 item for the feature. We cannot 
easily tune preemption algorithm without that.

bq. Could we get a choice to kill container based on data locality, I could see 
the changes but cudnt see how its achieved in preemption manager end.
The POC patch implemented part of it. We should be able to preempt container 
based on data locally, since currently it make preemption decisions based on 
"real" resource request. So preempted container is considered everything in 
existing allocation logic, such as locality delay, limits (queue/user), etc.

> CapacityScheduler: Improve preemption to preempt only those containers that 
> would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-4108-design-doc-v1.pdf, YARN-4108.poc.1.patch
> This is sibling JIRA for YARN-2154. We should make sure container preemption 
> is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality 
> (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I 
> don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), 
> cross applicaiton preemption (such as priority-based (YARN-1963) / 
> fairness-based (YARN-3319)).

This message was sent by Atlassian JIRA

Reply via email to