[
https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812187#comment-15812187
]
Wangda Tan commented on YARN-5889:
----------------------------------
Thanks [~sunilg] for providing the patch and reviews from [~eepayne].
Quickly scanned the patch,
If everybody agrees with approach in my comment:
https://issues.apache.org/jira/browse/YARN-5889?focusedCommentId=15749129&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15749129
{code}
active-user-limit = max(
resource-used-by-active-users / #active-users,
queue-capacity * MULP
)
{code}
We can:
1) Update {{resource-used-by-active-users}} when any user enters/exits set of
active users.
2) Update #active-users (using approaches like ActiveUsersManager approach)
Both of #1/#2 can be done within the same thread, and since we will have an
identical UL for all the active users, I think this can be done without adding
a new thread. Please correct me if I'm wrong.
It looks like Sunil and Eric have different implementation suggestions, I have
looked at details of the two approaches. However I think the goals are:
a. Can always get up-to-date UL
b. Avoid re-computation of UL as much as possible.
Any approach can achieve above goals is good to me.
And in addition, we can add a UsersManager to LQ to manager all user-related
information, such as user-limit / active-user-limit / #active-users.
Ideally, LQ/PCPP should get activeUserLimit / userLimit from UsersManager
And for last, we can look at logics for #active-users per partition, now we
have an identical #active-users for all partition, which causes some problems.
Since we will make a major changes to logics around this, it could be a good
chance to fix the problem as well.
> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>
> Key: YARN-5889
> URL: https://issues.apache.org/jira/browse/YARN-5889
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Sunil G
> Assignee: Sunil G
> Attachments: YARN-5889.0001.patch,
> YARN-5889.0001.suggested.patchnotes, YARN-5889.v0.patch, YARN-5889.v1.patch,
> YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with
> a write lock. To improve performance, this tickets is focussing on moving
> user-limit calculation out of heartbeat allocation flow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
