[
https://issues.apache.org/jira/browse/YARN-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576900#comment-16576900
]
Zian Chen commented on YARN-8509:
---------------------------------
Hi [~eepayne], sure, let address these two questions,
1) the summation is for each user, we calculate the minimum of these two
expression, one is the pending resource for this user per partition, another
one is user limit (which is queue_capacity * user_limit_factor) - user used
resource per partition
2) I think there is some misunderstanding here. First of all, after the title
been changed, this Jira is not intend to only support balancing of queues after
satisfied. It intend to change the general strategy of how user limit is been
calculated in preemption scenario.
So the queue capacities I mentioned here for the example is an initial state,
which is like this,
|| ||queue-a||queue-b||queue-c||queue-b||
|Guaranteed|30|30|30|10|
|Used|10|40|50|0|
|Pending|6|30|30|0|
this configuration should able to happen if we set user_limit_percent to 50 and
user_limit_factor to 1.0f, 3.0f, 3.0f and 2.0f respectively. But within current
equation, this initial state won't happen.
user_limit =
min(max(current_capacity)/ #active_users,
current_capacity * user_limit_percent),
queue_capacity * user_limit_factor)
in above case, queue-b's queue_capacity * user_limit_factor is 90GB while
max(current_capacity)/ #active_users, current_capacity * user_limit_percent) is
40GB, this will make user-limit-factor don't make any effect at all, and
headroom becomes zero for queue-b.
So the point is, we should let user-limit to reach at most queue_capacity *
user_limit_factor
> Total pending resource calculation in preemption should use user-limit factor
> instead of minimum-user-limit-percent
> -------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8509
> URL: https://issues.apache.org/jira/browse/YARN-8509
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Reporter: Zian Chen
> Assignee: Zian Chen
> Priority: Major
> Attachments: YARN-8509.001.patch, YARN-8509.002.patch,
> YARN-8509.003.patch
>
>
> In LeafQueue#getTotalPendingResourcesConsideringUserLimit, we calculate total
> pending resource based on user-limit percent and user-limit factor which will
> cap pending resource for each user to the minimum of user-limit pending and
> actual pending. This will prevent queue from taking more pending resource to
> achieve queue balance after all queue satisfied with its ideal allocation.
>
> We need to change the logic to let queue pending can go beyond userlimit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
