Wangda Tan commented on YARN-2009:


bq. ... then I don't think idealAssigned can be calculated independently from 
each other ... 
Actually I was thinking the same thing which we should compute idealAssigned 
for each user when I was reviewing YARN-2069. But I realized we may not need, 
let me explain it a little bit:
Computed user-limit resource in existing CS is using as higher bound of how 
much that each user should get, there's no "lower bound user limit resource" in 

I think all of us agree that behavior of preemption should be consistent with 
behavior of scheduling, any mismatch between the two could lead to excessive 

When FIFO (and also FIFO + PRIORITY) policy is enabled, an example of existing 
CS's behavior is:
Queue's user-limit-percent = 33
Queue's used=guaranteed=max=12. 
There're 3 users (A,B,C) in the queue, order of applications are A/B/C
Applications from user-A/C are asking for more resource, and application of 
userB is satisfied already.

So the computed user-limit-resource will be 6.

Assume resource usages of A/B/C are 5/6/1, and A/C have 1 pending resource, 

The actual user-ideal-assignment when doing scheduling is 6/6/0 !
(A can get the 1 additional resource, and B will not changed, C can get nothing 
after that)

So in another words, user-limit is just a cap in additional to FIFO (or 
FIFO+Priority) order:

Back to the preemption patch, the pseudo code to compute application ideal 
allocation consider user limit will be:
void compute-ideal-allocation-for-apps(List<Application> apps) {
    user-limit-resource = queue.get-user-limit-resource();

    // initial all value to 0
    Map<String, Resource> user-to-allocated;

    for app in sort-by-fifo-or-priority(apps) {
       if (user-to-allocated.get(app.user) < user-limit-resource) {
            app.allocated = min(app.used + pending, user-limit-resource - 
            user-to-allocated.get(app.user) += app.allocated;
       } else {
             // skip this app because user-limit reached

Please let me know about your thoughts.


> Priority support for preemption in ProportionalCapacityPreemptionPolicy
> -----------------------------------------------------------------------
>                 Key: YARN-2009
>                 URL: https://issues.apache.org/jira/browse/YARN-2009
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Devaraj K
>            Assignee: Sunil G
>         Attachments: YARN-2009.0001.patch
> While preempting containers based on the queue ideal assignment, we may need 
> to consider preempting the low priority application containers first.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to