[ 
https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504978#comment-15504978
 ] 

Wangda Tan commented on YARN-2009:
----------------------------------

[~eepayne],

bq. ... then I don't think idealAssigned can be calculated independently from 
each other ... 
Actually I was thinking the same thing which we should compute idealAssigned 
for each user when I was reviewing YARN-2069. But I realized we may not need, 
let me explain it a little bit:
Computed user-limit resource in existing CS is using as higher bound of how 
much that each user should get, there's no "lower bound user limit resource" in 
reality. 

I think all of us agree that behavior of preemption should be consistent with 
behavior of scheduling, any mismatch between the two could lead to excessive 
preemption.

When FIFO (and also FIFO + PRIORITY) policy is enabled, an example of existing 
CS's behavior is:
{code}
Queue's user-limit-percent = 33
Queue's used=guaranteed=max=12. 
There're 3 users (A,B,C) in the queue, order of applications are A/B/C
Applications from user-A/C are asking for more resource, and application of 
userB is satisfied already.

So the computed user-limit-resource will be 6.

Assume resource usages of A/B/C are 5/6/1, and A/C have 1 pending resource, 

The actual user-ideal-assignment when doing scheduling is 6/6/0 !
(A can get the 1 additional resource, and B will not changed, C can get nothing 
after that)
{code} 

So in another words, user-limit is just a cap in additional to FIFO (or 
FIFO+Priority) order:

Back to the preemption patch, the pseudo code to compute application ideal 
allocation consider user limit will be:
{code}
void compute-ideal-allocation-for-apps(List<Application> apps) {
    user-limit-resource = queue.get-user-limit-resource();

    // initial all value to 0
    Map<String, Resource> user-to-allocated;

    for app in sort-by-fifo-or-priority(apps) {
       if (user-to-allocated.get(app.user) < user-limit-resource) {
            app.allocated = min(app.used + pending, user-limit-resource - 
user-to-allocated.get(app.user));
            user-to-allocated.get(app.user) += app.allocated;
       } else {
             // skip this app because user-limit reached
       }
    }
}
{code}

Please let me know about your thoughts.

Thanks,

> Priority support for preemption in ProportionalCapacityPreemptionPolicy
> -----------------------------------------------------------------------
>
>                 Key: YARN-2009
>                 URL: https://issues.apache.org/jira/browse/YARN-2009
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Devaraj K
>            Assignee: Sunil G
>         Attachments: YARN-2009.0001.patch
>
>
> While preempting containers based on the queue ideal assignment, we may need 
> to consider preempting the low priority application containers first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to