[
https://issues.apache.org/jira/browse/YARN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648418#comment-14648418
]
Nathan Roberts commented on YARN-3945:
--------------------------------------
Hi [~leftnoteasy]. Regarding minimum_user_limit_percent.
- I totally agree it is very confusing.
- I don't think we can change it in any significant way at this point without a
major configuration switch that clearly indicates you're getting different
behavior. I'm sure admins have built up clusters with this tuned in very
specific ways, a significant change wouldn't be compatible with their
expectations.
bq. User-limit is not a fairness mechanism to balance resources between users,
instead, it can lead to bad imbalance. One example is, if we set user-limit =
50, and there are 10 users running, we cannot manage how much resource can be
used by each user.
I don't really agree with this. It may not be doing an ideal job but I think
the intent is to introduce fairness between users. It's a progression from 0
being the most fair, and 100+ being more fifo. In your example it's trying to
get everyone 50% which isn't likely to happen so in this case it's going to
operate mostly fifo. If the intent is to be much more fair across the 10 users,
then a much smaller value would be appropriate.
bq. meaningful since #active-user is changing every minute, it is not a
predictable formula.
Since the scheduler can't predict what an application is going to request in
the future, I don't see how a predictable formula is even possible (ignoring
the possibility of taking away resources via in-queue preemption). It's not
great, but being fair to currently requesting users makes some bit of sense.
bq. Instead we may need to consider some notion like fair sharing:
user-limit-factor becomes max-resource-limit of each user, and
user-limit-percentage becomes something like guaranteed-concurrent-#user, when
#user > guaranteed-concurrent-#user, rest users can only get idle shares.
user-limit-factor is the max-resource-limit of each user today, right? The
second one seems very hard to track. It seems like one of the initial users can
stay in the "guaranteed" set as long as he keeps requesting resources. This
doesn't seem very fair to the users only getting idle shares.
> maxApplicationsPerUser is wrongly calculated
> --------------------------------------------
>
> Key: YARN-3945
> URL: https://issues.apache.org/jira/browse/YARN-3945
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
> Affects Versions: 2.7.1
> Reporter: Naganarasimha G R
> Assignee: Naganarasimha G R
> Attachments: YARN-3945.20150728-1.patch, YARN-3945.20150729-1.patch
>
>
> maxApplicationsPerUser is currently calculated based on the formula
> {{maxApplicationsPerUser = (int)(maxApplications * (userLimit / 100.0f) *
> userLimitFactor)}} but description of userlimit is
> {quote}
> Each queue enforces a limit on the percentage of resources allocated to a
> user at any given time, if there is demand for resources. The user limit can
> vary between a minimum and maximum value.{color:red} The the former (the
> minimum value) is set to this property value {color} and the latter (the
> maximum value) depends on the number of users who have submitted
> applications. For e.g., suppose the value of this property is 25. If two
> users have submitted applications to a queue, no single user can use more
> than 50% of the queue resources. If a third user submits an application, no
> single user can use more than 33% of the queue resources. With 4 or more
> users, no user can use more than 25% of the queues resources. A value of 100
> implies no user limits are imposed. The default is 100. Value is specified as
> a integer.
> {quote}
> configuration related to minimum limit should not be made used in a formula
> to calculate max applications for a user
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
