[
https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712097#comment-15712097
]
Eric Payne edited comment on YARN-5889 at 12/1/16 2:29 PM:
---
bq. Any change in resource (allocation and release of container etc) for a
given user could set a state variable. This will set off by the computation
thread if next cycle falls immediate.
Just so we're on the same page and what I'm suggesting is more clear.
I'm suggesting that we remove the computation thread
({{ComputeUserLimitAsyncThread}}), and when {{LeafQueue#assignContainers}} or
{{LeafQueue#releaseResource}} is called, we can set the flag in
{{UserToPartitionRecord}}. This flag should also be set when queues are
refreshed. {{getComputedUserLimit}} would then be called as it is in your
patch, and compute user limit resource if the flag is set.
bq. Its not ideal to ask allocation thread to hold till computation. So by
seeing this state variable, we might need to compute user-limit in same
allocation thread.
I recognize that my suggested approach does the calculation during the
allocation thread. My assertion would be that
# this calculation is being done currently in the allocation thread
# my suggested approach is an optimization that would decrease (sometimes
greatly) the number of times the calculation happens
# my suggested approach would be more accurate than with a separate computation
thread.
was (Author: eepayne):
bq. Any change in resource (allocation and release of container etc) for a
given user could set a state variable. This will set off by the computation
thread if next cycle falls immediate.
Just so we're on the same page and what I'm suggesting is more clear.
I'm suggesting that we remove the computation thread
({{ComputeUserLimitAsyncThread}}), and when {{LeafQueue#assignContainers}} or
{{LeafQueue#releaseResource}} is called, we can set the flag in
{{UserToPartitionRecord}}. This flag should also be set within queues are
refreshed. {{getComputedUserLimit}} would then be called as it is now, and
compute user limit resource if the flag is set.
bq. Its not ideal to ask allocation thread to hold till computation. So by
seeing this state variable, we might need to compute user-limit in same
allocation thread.
I recognize that my suggested approach does the calculation during the
allocation thread. My assertion would be that
# this calculation is being done currently in the allocation thread
# my suggested approach is an optimization that would decrease (sometimes
greatly) the number of times the calculation happens
# my suggested approach would be more accurate than with a separate computation
thread.
> Improve user-limit calculation in capacity scheduler
>
>
> Key: YARN-5889
> URL: https://issues.apache.org/jira/browse/YARN-5889
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch,
> YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with
> a write lock. To improve performance, this tickets is focussing on moving
> user-limit calculation out of heartbeat allocation flow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
