[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15971666#comment-15971666
]
Wangda Tan commented on YARN-5892:
----------------------------------
Thanks [~eepayne] for updating the patch, got some time to check the latest
patch and caught up with discussions.
There're several concerns regarding to user-limit < 1:
1) When there're several active users with weight < 1. For example in a queue,
UL = 1, and u1.weight=0.1, u2.weight=0.2. If there're only two apps, a1 from u1
and a2 from u2 running in the queue and both are requesting a lot of resource,
ideally we should get u1 has 33% of the queue's guaranteed capacity and u2 has
66% of the queue's guaranteed capacity. However in this implementation (just
from my understanding, haven't tested yet). a1 can get all queue's resource
(because #active-user-applied-weights = 1/0.3) while a2 got starved.
2) I would like to prevent setting user's weight to <= 0. It doesn't have clear
semantics and could cause many other computation issues - printing a warning is
not enough to me.
Generally speaking, set user weight < 1 is a reasonable requirement however I
don't think we're ready for that. It looks there're bunch of things we need to
do to make #2 and related preemption logic works properly. I would suggest to
not allow this for now and add a note in the doc saying that this could be
supported in the future to avoid incompatible behavior changes. Thoughts?
Beyond that, I suggest to make #active-users-times-weight can updated in O(1)
for every changes to active users set or any active user's weight get updated.
We saw in some extreme cases a queue can have several hundreds of users so
change to active users set could be quite frequent.
Also, weight of users applies to hard limit of user (user limit factor) as
well. This is a gray area to me, since it may cause some issue of resource
planning (one more factor apply to maximum resource of user). Would like to
hear thoughts from [~jlowe]/[~sunilg] as well.
> Capacity Scheduler: Support user-specific minimum user limit percent
> --------------------------------------------------------------------
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch,
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch,
> YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch,
> YARN-5892.008.patch, YARN-5892.009.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent</name>
> <value>25</value>
> </property>
> <property>
>
> <name>yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent</name>
> <value>75</value>
> </property>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
