[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287152#comment-17287152
]
Qi Zhu edited comment on YARN-10627 at 2/19/21, 4:18 PM:
---------------------------------------------------------
[~bteke] [~gandras] [~snemeth]
!image-2021-02-20-00-07-09-875.png|width=420,height=156!
When i deep into the code, and look the ui, when weight setted, the capacity
will be 0.
Only the weight is -1, we can make sure it is not weight mode.
So the code may change to, in leafqueue:
{code:java}
private String getExtendedCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
} else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
}
}
{code}
In abstractCSQueue:
{code:java}
protected String getCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight=" + queueCapacities.getWeight() + ", " +
"normalizedWeight=" + queueCapacities.getNormalizedWeight();
} else {
return "capacity=" + queueCapacities.getCapacity();
}
}
{code}
And we can add a test to confirm it.
Thanks.
was (Author: zhuqi):
[~bteke] [~gandras] [~snemeth]
!image-2021-02-20-00-07-09-875.png|width=420,height=156!
When i deep into the code, and look the ui, when weight setted, the capacity
will be 0.
Only the weight is -1, we can make sure it is not weight mode.
So the code may change to, such as:
{code:java}
private String getExtendedCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
} else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
}
}
{code}
And we can add a test to confirm it.
Thanks.
> Extend logging to give more information about weight mode
> ---------------------------------------------------------
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
> Reporter: Benjamin Teke
> Assignee: Benjamin Teke
> Priority: Major
> Attachments: YARN-10627.001.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " + getState()
> + " [= configuredState ]" + "\n" + "acls = " + aclsString
> + " [= configuredAcls ]" + "\n"
> + "nodeLocalityDelay = " + nodeLocalityDelay + "\n"
> + "rackLocalityAdditionalDelay = "
> + rackLocalityAdditionalDelay + "\n"
> + "labels=" + labelStrBuilder.toString() + "\n"
> + "reservationsContinueLooking = "
> + reservationsContinueLooking + "\n" + "preemptionDisabled = "
> + getPreemptionDisabled() + "\n" + "defaultAppPriorityPerQueue
> = "
> + defaultAppPriorityPerQueue + "\npriority = " + priority
> + "\nmaxLifetime = " + getMaximumApplicationLifetime()
> + " seconds" + "\ndefaultLifetime = "
> + getDefaultApplicationLifetime() + " seconds");
> {code}
> ..etc. These should be extended for easier debugging or general verifications.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
