Sunil G commented on YARN-4781:

Thanks [~eepayne] for bringing this thread alive..

bq.Then, while all 4 users have running apps, User5 comes along and can't get 
any resources, they see that User1 is using 62% more resources than everyone 
else, and wonders why they can't get any resources.

Ideally when app3 finishes, quota will go for apps which are started first or 
less consumed resources compared to others. However my concern is over case 
where we need to worry about priority with FairOrdering. Both are kind of 
mutually exclusive feature to me.


Coming to the approach, i guess we need to ensure that FairOrdering + 
USERLIMIT_FIRST is the combination. In this combination, we will be able to do 
preemption for apps which under allocated as per FairOrdering policy.


The patch is simpler enough to cover the comparator change. Thinking out load, 
could we use Fair plugin instead of FifoIntraQueuePreemptionPlugin, and make an 
abstract class to move all common code. Then Fair plugin could be auto chosen 
based on ordering policy selected.

> Support intra-queue preemption for fairness ordering policy.
> ------------------------------------------------------------
>                 Key: YARN-4781
>                 URL: https://issues.apache.org/jira/browse/YARN-4781
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Major
>         Attachments: YARN-4781.001.patch
> We introduced fairness queue policy since YARN-3319, which will let large 
> applications make progresses and not starve small applications. However, if a 
> large application takes the queue’s resources, and containers of the large 
> app has long lifespan, small applications could still wait for resources for 
> long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to 
> preempt resources of queue with fairness policy enabled.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to