[
https://issues.apache.org/jira/browse/YARN-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266313#comment-16266313
]
Wilfred Spiegelenburg commented on YARN-7535:
---------------------------------------------
The code has changed in recent versions, there is no updateDemandForApp any
more after YARN-6172.
Demand for a queue as [~yufeigu] explained should be limited to the maximum the
queue can use. So the existing code should be left as is. Changing the
calculation would affect the minimum share starvation and some other
calculations that use the demand. Having the extra detail on how high demand
really is in a queue is could provide some more detail for tuning. The
{{FSAppAttempt}} does not cap it so we have the info already.
Some considerations:
- We could store the extra detail to the {{leafQueue}}. There would not really
be an overhead beside some extra local storage.
- Adding it to the {{parentQueue}} to get it for the whole hierarchy would be
possible but it does involve overhead. We would then also need to choose if we
want the unlimited demand from the child queue or the limited version
- The scheduler state dump is easily changed,
- Do we want to display this in the web UI? It might be confusing to show the
two numbers always and the state dump would be a much better place because it
can be seen over time instead of just one instance
> We should display origin value of demand in fair scheduler page
> ---------------------------------------------------------------
>
> Key: YARN-7535
> URL: https://issues.apache.org/jira/browse/YARN-7535
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
> Reporter: YunFan Zhou
> Assignee: YunFan Zhou
>
> The value of *demand* of leaf queue that we now view on the fair scheduler
> page shows only the value of *maxResources* when the demand value is greater
> than *maxResources*. It doesn't reflect the real situation. Most of the time,
> when we expand the queue, we often rely on seeing the current demand real
> value.
> {code:java}
> private void updateDemandForApp(FSAppAttempt sched, Resource maxRes) {
> sched.updateDemand();
> Resource toAdd = sched.getDemand();
> if (LOG.isDebugEnabled()) {
> LOG.debug("Counting resource from " + sched.getName() + " " + toAdd
> + "; Total resource consumption for " + getName() + " now "
> + demand);
> }
> demand = Resources.add(demand, toAdd);
> demand = Resources.componentwiseMin(demand, maxRes);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
