[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-26 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291919#comment-17291919
 ] 

Peter Bacsko commented on YARN-10627:
-

Thanks for the patch [~bteke] and [~zhuqi] / [~gandras] for the review.

Committed to trunk.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-26 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291897#comment-17291897
 ] 

Hadoop QA commented on YARN-10627:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
18s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m  3s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 146 unchanged - 16 fixed = 146 total (was 162) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 50s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green}{color} | {color:green} the patch passed with JDK 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-26 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291759#comment-17291759
 ] 

Benjamin Teke commented on YARN-10627:
--

[~pbacsko] sure, fixed it.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-26 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291718#comment-17291718
 ] 

Peter Bacsko commented on YARN-10627:
-

Thanks for the patch [~bteke], could you take care of those checkstyle stuff? 
Thanks.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
> 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-25 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291001#comment-17291001
 ] 

Hadoop QA commented on YARN-10627:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
58s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 
31s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
8s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m 14s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  2m 
30s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
28s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
7s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
7s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 49s{color} | 
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/677/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
 | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 2 new + 146 unchanged - 16 fixed = 148 total (was 162) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 56s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-25 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290991#comment-17290991
 ] 

Hadoop QA commented on YARN-10627:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
40s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
22s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 44s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
44s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
42s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 43s{color} | 
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/678/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
 | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 2 new + 146 unchanged - 16 fixed = 148 total (was 162) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 39s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-25 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290894#comment-17290894
 ] 

Benjamin Teke commented on YARN-10627:
--

Thanks [~pbacsko] and [~gandras] for the review. Fixed the checkstyle issues, 
added the rm close methods, and removed the unnecessary GB string. The current 
tests include cases for capacity related log string generation (the used 
configurations are mixed mode, they have both capacity and weight in the 
hierarchy), this is why I didn't feel the need to introduce separate cases. Do 
you think it should be done?

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-24 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289945#comment-17289945
 ] 

Andras Gyori commented on YARN-10627:
-

Thank you [~bteke] for the patch. Apart from checkstyle issues and comments 
from Peter, I have some minor nit feedback:
 * GB is not used in TestCapacitySchedulerWeightMode
 * If it is not too cumbersome, perhaps creating the non weight mode 
counterpart of tests would be valuable as well, too see everything is working 
when weight mode is disabled.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-24 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289911#comment-17289911
 ] 

Peter Bacsko commented on YARN-10627:
-

# Still have 4 checkstyle issues, they're not serious, but if we're not in a 
rush, we should fix those.
 # testGetCapacityOrWeightStringUsingWeights / 
testGetCapacityOrWeightStringParentPctLeafWeights -> make sure MockRM is closed 
in a finally block

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-23 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289236#comment-17289236
 ] 

Hadoop QA commented on YARN-10627:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
51s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 53s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 40s{color} | 
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/665/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
 | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 4 new + 147 unchanged - 15 fixed = 151 total (was 162) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 49s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-22 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288809#comment-17288809
 ] 

Qi Zhu commented on YARN-10627:
---

Thanks [~bteke] update.

Patch LGTM, let's trigger CI again.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>  

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-22 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288509#comment-17288509
 ] 

Benjamin Teke commented on YARN-10627:
--

[~zhuqi]

Uploaded a new patchset with the added UTs.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-22 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288435#comment-17288435
 ] 

Hadoop QA commented on YARN-10627:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 
53s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
46s{color} | 
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/657/artifact/out/branch-mvninstall-root.txt{color}
 | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
23s{color} | 
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/657/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt{color}
 | {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
23s{color} | 
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/657/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt{color}
 | {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 21s{color} | 
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/657/artifact/out/buildtool-branch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
 | {color:orange} The patch fails to run checkstyle in 
hadoop-yarn-server-resourcemanager {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
22s{color} | 
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/657/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
 | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
1m 11s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
23s{color} | 
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/657/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt{color}
 | {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
23s{color} | 
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/657/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt{color}
 | {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  2m 
19s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
22s{color} | 
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/657/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
 | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
22s{color} | 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-19 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287339#comment-17287339
 ] 

Hadoop QA commented on YARN-10627:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
18s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 41s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m  1s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-19 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287213#comment-17287213
 ] 

Benjamin Teke commented on YARN-10627:
--

[~zhuqi], indeed, thanks for finding it. Fixed it with the second patch, will 
introduce some tests for it in patch 003.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-19 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287152#comment-17287152
 ] 

Qi Zhu commented on YARN-10627:
---

[~bteke] [~gandras]

!image-2021-02-20-00-07-09-875.png|width=420,height=156!

When i deep into the code, and look the ui, when weight setted, the capacity 
will be 0.

So the code may change to, such as:

 
{code:java}
private String getExtendedCapacityOrWeightString() {
  if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
  } else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
  }
}
{code}
And we can add a test to confirm it.

 

 

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>  

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-18 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286817#comment-17286817
 ] 

Qi Zhu commented on YARN-10627:
---

Thank you [~bteke] for the patch. LGTM +1.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   + " [= configuredState ]" + "\n" + "acls = " + aclsString
>   + " [= configuredAcls ]" 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-18 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286535#comment-17286535
 ] 

Andras Gyori commented on YARN-10627:
-

Thank you [~bteke] for the patch. I agree with the resolution, and it looks 
good to me +1.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   + " [= configuredState ]" + "\n" + 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-17 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285977#comment-17285977
 ] 

Hadoop QA commented on YARN-10627:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
29s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red}  0m  7s{color} 
| {color:red}{color} | {color:red} Unprocessed flag(s): 
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/628/artifact/out/Dockerfile
 |
| JIRA Issue | YARN-10627 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/13020579/YARN-10627.001.patch |
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/628/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>  

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-16 Thread Qi Zhu (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285189#comment-17285189
 ] 

Qi Zhu commented on YARN-10627:
---

Thanks [~bteke] for this issue.

I also think more information is helpful and import.

 

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   + " [= configuredState ]" + "\n" + "acls = " + aclsString
>   + " [= configuredAcls ]"