[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886351#comment-15886351
]
Wangda Tan commented on YARN-5892:
----------------------------------
Thanks [~eepayne] for the elaborations,
I have a discussion with [~jlowe]/[~curino] during community sync up and have a
separate offline discussion with [~sunilg].
I was thinking solutions like dynamic creation of user-queue and assign quotas
to these queues, etc. But it seems doesn't fit well for FIFO + user-quota. Also
mentioned by [~jlowe], dynamic queue exposes implementation details to end user.
[~curino] proposed to use reservation system to solve the problem, but I'm not
sure how, Carlo could you please add your thoughts?
In my mind, the MULP (minimum-user-limit-percentage) is a quota applied to
active users, which gives enough resource for app (or user which the app
belongs to) to run. If there're more active users than (100 / MULP), scheduler
tries to give resource to first (100 / MULP) users. If #active_users < (100 /
MULP), available resource will be equally divided to these users.
So I preferred to keep the semantic more similar to existing one, I propose to
introduce a weight of users instead of overriding the MULP: scheduler will
continue assign MULP% shares to each "unit users", but different user can have
different weight to adjust quota based on share of "unit users". Also, the
weights of users can be used independent from MULP: because in the future we
may want to replace concept of user limit by different ones. (Like setting
quota for each user, give weighted fair share to users, etc.)
Also, regarding to config, there's a collision for
{{<queue-name>.<user-name>.MULP}} when queue's name equals to {{<user-name>}}.
How about adding one more layer to specify settings of users, like:
{code}
... <leaf-queue-name>.user-settings.<user-name>.weight = x
... <leaf-queue-name>.user-settings.<user-name>.weight = y
...
{code}?
> Capacity Scheduler: Support user-specific minimum user limit percent
> --------------------------------------------------------------------
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: YARN-5892.001.patch, YARN-5892.002.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent</name>
> <value>25</value>
> </property>
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent</name>
> <value>75</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
