[
https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745552#comment-15745552
]
Jason Lowe commented on YARN-5889:
----------------------------------
bq. To solve the problem, we need to compute user limit considering non-active
users. If a non-active user uses less than minimum-user-limit, we can continue
distribute its available quotas to other active users; in the other hand, if a
non-active user uses more than minimum-user-limit, we could also get resource
from the user.
As I understand it, for allocation purposes we need to compute the user limit
where the number of users is the number of users who have at least one
application that is requesting resources. For preemption purposes, we need to
compute the user limit where the number of users is the total number of users
who have at least one app in the queue (whether they have apps that are
requesting or not).
bq. This computation is more expensive, it should be O(N), N is number of
applications in the queue.
I don't see how this is related to the number of applications. If you look at
how the user limit is calculated, there are no terms in that calculation that
have anything to do with how many apps the user has. Am I missing something?
If so, maybe an example where number of applications or order of applications
would help clarify.
I still think a simple flag to indicate a user limit needs to be recomputed
would go a long way here. We are already tracking total resources associated
with each user in each queue in the User structure and caching the user limit
there as well. We could add a flag to indicate the cached user limit needs to
be recalculated, and when we go to get the cached value it can on-the-fly
recalculate it if it is dirty. The calculation would only become dirty if one
of the following events occur:
- An allocation is assigned to one of the user's apps
- A container associated with one of the user's apps completes
- A user becomes active in the queue (i.e.: now has apps requesting resources)
- A user becomes inactive in the queue (i.e.: no longer has apps requesting
resources)
- The queue capacity changes (e.g.: nodes added/removed)
- The queue settings are refreshed
We can speed up the last four by having a queue-level sequence number that is
incremented when active users or queue changes. The User structure can cache
the sequence number used when the user limit is recalculated and compare with
the queue sequence number to know a user's limit needs recalculating. This can
also become the user-level flag when containers are allocated/completed by
setting the cached sequence number in the User structure to the queue's
sequence number minus one, which will force the sequence numbers to mismatch
and cause a recalculation when the user limit is requested.
The idea here is that we will only calculate the user limit lazily and only
when absolutely necessary. This will be much faster than what we do today and
not require asynchronous computation that breaks the constraints of the
scheduler. For purposes of doing preemptions, we can have the same concept
used for a separate cached user limit, one that considers all users instead of
only active ones.
When it comes time to calculate preemptions (which are only necessary when the
queue cannot get more resources), we can sort the users by how far they are
beyond their all-users user limit (either in absolute terms or by percentage).
This may or may not require a computation depending upon whether the cached
value is out of date. Then we can walk down the list starting with the user
most past their limit. We can stop traversing when we either preempt enough
resources or we get to a user that is below their limit.
> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>
> Key: YARN-5889
> URL: https://issues.apache.org/jira/browse/YARN-5889
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Sunil G
> Assignee: Sunil G
> Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch,
> YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with
> a write lock. To improve performance, this tickets is focussing on moving
> user-limit calculation out of heartbeat allocation flow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
