[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-5892:
-----------------------------
Attachment: YARN-5892.008.patch
There are a couple of notable things about this patch:
- A user's weight can be less than 1.0. When a user's weight is < 1.0, {{Max
Resources}} looks as expected on the scheduler GUI when all users are
requesting resources. However, when user(s) with weithg < 1 are asking and
others are not, the {{Max Resource}} value is not accurate for the non-active
users. For example, in a queue with 20GB, {{User2}} has a weight of 0.2 and
{{User5}} has a weight of 1.0. When both are active, the weights and used
resources for both are part of the {{Max Resources}} calculations, as below:
||*User Name*||*Max Resource*||*Weight*||*Used Resource*||
|User2|<memory:1536, vCores:1>|0.2|<memory:7168, vCores:14>|
|User5|<memory:7680, vCores:1>|1.0|<memory:2048, vCores:4>|
However, when {{User5}} stops requsting resources, its {{Max Resources}}
becomes a factor of {{User2}}'s {{Max Resources}}:
||*User Name*||*Max Resource*||*Weight*||*Used Resource*||
|User2|<memory:7168, vCores:1>|0.2|<memory:7168, vCores:14>|
|User5|<memory:35840, vCores:1>|1.0|<memory:2560, vCores:5>|
Note that the queue only has 20GB. This is odd, but I don't view this as a
problem since it is evident which users are asking for resources (see attached
screenshot) and the {{Max Resources}} is only applicable when a user is active.
- User Limit Factor is also affected by user weights. If {{User1}} has a weight
of 0.5 and a queue's User Limit Factor is 1.0, {{User1}} can only use half of
the queue.
- A user weighted at 0 will be granted the AM container, but no more.
- We have an addition use case that this feature doesn't cover, but it will
need to wait for another JIRA. We need to allow a user to be weighted very
small but that will not be limited by the weighted user limit factor so that it
can use the whole queue.
> Capacity Scheduler: Support user-specific minimum user limit percent
> --------------------------------------------------------------------
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch,
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch,
> YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch,
> YARN-5892.008.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent</name>
> <value>25</value>
> </property>
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent</name>
> <value>75</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
