Jonathan Hung created YARN-6491:
-----------------------------------
Summary: Move totalMB and totalVirtualCores computation from
ClusterMetricsInfo to QueueMetrics
Key: YARN-6491
URL: https://issues.apache.org/jira/browse/YARN-6491
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jonathan Hung
Right now in ClusterMetricsInfo.java we do this:{noformat} if (rs instanceof
CapacityScheduler) {
this.totalMB = metrics.getTotalMB();
this.totalVirtualCores = metrics.getTotalVirtualCores();
} else {
this.totalMB = availableMB + allocatedMB;
this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores;
}{noformat}
We'd like to have totalMB and totalVirtualCores as QueueMetrics fields. But
since QueueMetrics is scheduler agnostic we can't really just move this. It
seems the way totalMB and totalVirtualCores is computed across FS and CS should
be standardized.
Right now CS does not include reservedMB in allocatedMB, while FS does (as far
as I can tell). At least in <= 2.7, when a container is reserved,
queueUsage.getUsed is incremented, which is the value used to determine if a
queue can be assigned (AbstractCSQueue#canAssignToThisQueue). So I think it
makes sense to increment allocatedMB when a container is reserved, and not
increment it when a reserved container is allocated, to reflect the fact that
if allocated + reserved > queueLimit, allocation will fail, so the allocatedMB
metric should also be > queueLimit.
Still not sure if the same is true >= 2.8. Would appreciate any input on this
(or any of the mentioned proposals).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
