[
https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824442#comment-15824442
]
Wangda Tan commented on YARN-5889:
----------------------------------
Thanks [~sunilg] for updating the patch.
Some minor comments:
1) I made a mistake of my previous comment: recalculateQueueUsageRatio should
belong to LQ, and it invokes UM#calculateUserUsageRatio.
2) UM#latestVersionOfUserCount is a little bit confusing, since it doesn't only
consider user count, how about rename it to latestVersionOfUsersState? (And
update comments, getters/setters accordingly).
3) ActiveUM#isUserAddedOrRemoved, parameter is not required since it is always
false.
4) Locks of UM:
- Add writeLock for following code block:
{code}
long latestVersionOfUserCount = getLatestVersionOfUserCount();
if (isRecomputeNeeded(user, userLimitPerSchedulingMode,
latestVersionOfUserCount, schedulingMode)) {
// recompute
userLimitPerSchedulingMode = reComputeUserLimits(rc, userName,
nodePartition, clusterResource, schedulingMode, true);
// update user count to cache so that we can avoid recompute if no major
// changes.
user.setLocalVersionOfUserCount(latestVersionOfUserCount);
}
{code}
For getComputedResourceLimitFor*Users.
This can make sure update of UL can be sequentially done and version increases
while updating UL can always trigger another update.
5) Regarding to UserToPartitionRecord and computeUserLimit:
My gut feeling is that we can optimize/simplify this part a little bit, let's
revisit it after this patch: This patch could lead to wrong REST UL computation
(returns all user limit instead of active user limit), and it recomputes UL for
every user instead of every partition.
> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>
> Key: YARN-5889
> URL: https://issues.apache.org/jira/browse/YARN-5889
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Sunil G
> Assignee: Sunil G
> Attachments: YARN-5889.0001.patch,
> YARN-5889.0001.suggested.patchnotes, YARN-5889.0002.patch,
> YARN-5889.0003.patch, YARN-5889.0004.patch, YARN-5889.v0.patch,
> YARN-5889.v1.patch, YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with
> a write lock. To improve performance, this tickets is focussing on moving
> user-limit calculation out of heartbeat allocation flow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
