[jira] [Updated] (YARN-10626) Log resource allocation in NM log at container start time

2021-02-15 Thread Eric Badger (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated YARN-10626:
---
Attachment: YARN-10626.001.patch

> Log resource allocation in NM log at container start time
> -
>
> Key: YARN-10626
> URL: https://issues.apache.org/jira/browse/YARN-10626
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-10626.001.patch
>
>
> As far as I can tell, there are no resource allocation logs in the NM log for 
> the various containers that are scheduled. These can be useful when trying to 
> debug what resources were requested vs what resources were actually 
> allocated. This is especially useful when debugging upstream technology 
> changes to make sure that they are correctly interpreting and passing down 
> resource parameters



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10626) Log resource allocation in NM log at container start time

2021-02-15 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284922#comment-17284922
 ] 

Hadoop QA commented on YARN-10626:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m 10s{color} 
| {color:red}{color} | {color:red} YARN-10626 does not apply to trunk. Rebase 
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for 
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10626 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/13020492/YARN-10626.001.patch |
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/617/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Log resource allocation in NM log at container start time
> -
>
> Key: YARN-10626
> URL: https://issues.apache.org/jira/browse/YARN-10626
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-10626.001.patch
>
>
> As far as I can tell, there are no resource allocation logs in the NM log for 
> the various containers that are scheduled. These can be useful when trying to 
> debug what resources were requested vs what resources were actually 
> allocated. This is especially useful when debugging upstream technology 
> changes to make sure that they are correctly interpreting and passing down 
> resource parameters



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10627) Extend logging to give more information about weight mode

2021-02-15 Thread Benjamin Teke (Jira) 
Benjamin Teke created YARN-10627:


 Summary: Extend logging to give more information about weight mode
 Key: YARN-10627
 URL: https://issues.apache.org/jira/browse/YARN-10627
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn
Reporter: Benjamin Teke
Assignee: Benjamin Teke


In YARN-10504 weight mode was added, however the logged information about the 
created queues or the toString methods weren't updated accordingly. Some 
examples:

ParentQueue#setupQueueConfigs:

{code:java}
 LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
  + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
  + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
  + ", absoluteMaxCapacity=" + this.queueCapacities
  .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
  + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
  + ", reservationsContinueLooking=" + reservationsContinueLooking
  + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
  + ", priority=" + priority
  + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
{code}


ParentQueue#toString:

{code:java}
public String toString() {
return queueName + ": " +
"numChildQueue= " + childQueues.size() + ", " + 
"capacity=" + queueCapacities.getCapacity() + ", " +  
"absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
"usedResources=" + queueUsage.getUsed() + 
"usedCapacity=" + getUsedCapacity() + ", " + 
"numApps=" + getNumApplications() + ", " + 
"numContainers=" + getNumContainers();
 }
{code}

LeafQueue#setupQueueConfigs:

{code:java}
  LOG.info(
  "Initializing " + getQueuePath() + "\n" + "capacity = "
  + queueCapacities.getCapacity()
  + " [= (float) configuredCapacity / 100 ]" + "\n"
  + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
  + " [= parentAbsoluteCapacity * capacity ]" + "\n"
  + "maxCapacity = " + queueCapacities.getMaximumCapacity()
  + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = "
  + queueCapacities.getAbsoluteMaximumCapacity()
  + " [= 1.0 maximumCapacity undefined, "
  + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise 
]"
  + "\n" + "effectiveMinResource=" +
  getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
  + " , effectiveMaxResource=" +
  getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
  + "\n" + "userLimit = " + usersManager.getUserLimit()
  + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
  + usersManager.getUserLimitFactor()
  + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = "
  + maxApplications
  + " [= configuredMaximumSystemApplicationsPerQueue or"
  + " (int)(configuredMaximumSystemApplications * 
absoluteCapacity)]"
  + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
  + " [= (int)(maxApplications * (userLimit / 100.0f) * "
  + "userLimitFactor) ]" + "\n"
  + "maxParallelApps = " + getMaxParallelApps() + "\n"
  + "usedCapacity = " +
  + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory / "
  + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
  + "absoluteUsedCapacity = " + absoluteUsedCapacity
  + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
  + "maxAMResourcePerQueuePercent = " + maxAMResourcePerQueuePercent
  + " [= configuredMaximumAMResourcePercent ]" + "\n"
  + "minimumAllocationFactor = " + minimumAllocationFactor
  + " [= (float)(maximumAllocationMemory - minimumAllocationMemory) 
/ "
  + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
  + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
  + "numContainers = " + numContainers
  + " [= currentNumContainers ]" + "\n" + "state = " + getState()
  + " [= configuredState ]" + "\n" + "acls = " + aclsString
  + " [= configuredAcls ]" + "\n"
  + "nodeLocalityDelay = " + nodeLocalityDelay + "\n"
  + "rackLocalityAdditionalDelay = "
  + rackLocalityAdditionalDelay + "\n"
  + "labels=" + labelStrBuilder.toString() + "\n"
  + "reservationsContinueLooking = "
  + reservationsContinueLooking + "\n" + "preemptionDisabled = "
  + getPreemptionDisabled() + "\n" + "defaultAppPriorityPerQueue = "
  +

[jira] [Commented] (YARN-10626) Log resource allocation in NM log at container start time

2021-02-15 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284948#comment-17284948
 ] 

Hadoop QA commented on YARN-10626:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m  
7s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
20s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
31s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
21s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 18s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
20s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
17s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
36s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
21s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
21s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
15s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m  1s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
31s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
|

[jira] [Created] (YARN-10626) Log resource allocation in NM log at container start time

2021-02-15 Thread Eric Badger (Jira) 
Eric Badger created YARN-10626:
--

 Summary: Log resource allocation in NM log at container start time
 Key: YARN-10626
 URL: https://issues.apache.org/jira/browse/YARN-10626
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Eric Badger
Assignee: Eric Badger


As far as I can tell, there are no resource allocation logs in the NM log for 
the various containers that are scheduled. These can be useful when trying to 
debug what resources were requested vs what resources were actually allocated. 
This is especially useful when debugging upstream technology changes to make 
sure that they are correctly interpreting and passing down resource parameters



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10626) Log resource allocation in NM log at container start time

2021-02-15 Thread Eric Badger (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284957#comment-17284957
 ] 

Eric Badger commented on YARN-10626:


[~Jim_Brennan], could you review this when you get a chance? Due to the nature 
of a log change, I don't believe any unit tests are necessary for this patch

> Log resource allocation in NM log at container start time
> -
>
> Key: YARN-10626
> URL: https://issues.apache.org/jira/browse/YARN-10626
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-10626.001.patch, YARN-10626.002.patch
>
>
> As far as I can tell, there are no resource allocation logs in the NM log for 
> the various containers that are scheduled. These can be useful when trying to 
> debug what resources were requested vs what resources were actually 
> allocated. This is especially useful when debugging upstream technology 
> changes to make sure that they are correctly interpreting and passing down 
> resource parameters



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10626) Log resource allocation in NM log at container start time

2021-02-15 Thread Eric Badger (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated YARN-10626:
---
Attachment: YARN-10626.002.patch

> Log resource allocation in NM log at container start time
> -
>
> Key: YARN-10626
> URL: https://issues.apache.org/jira/browse/YARN-10626
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-10626.001.patch, YARN-10626.002.patch
>
>
> As far as I can tell, there are no resource allocation logs in the NM log for 
> the various containers that are scheduled. These can be useful when trying to 
> debug what resources were requested vs what resources were actually 
> allocated. This is especially useful when debugging upstream technology 
> changes to make sure that they are correctly interpreting and passing down 
> resource parameters



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10628) Add node usage metrics in SLS

2021-02-15 Thread VADAGA ANANYO RAO (Jira) 
VADAGA ANANYO RAO created YARN-10628:


 Summary: Add node usage metrics in SLS
 Key: YARN-10628
 URL: https://issues.apache.org/jira/browse/YARN-10628
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: scheduler-load-simulator
Affects Versions: 3.3.1
Reporter: VADAGA ANANYO RAO
Assignee: VADAGA ANANYO RAO
 Attachments: Nodes_memory_usage.png, Nodes_vcores_usage.png

Given the work around container packing going on in YARN schedulers, it would 
be beneficial to have charts showing the usage per node in SLS. This will help 
to improve container packing algorithms for more efficient packings.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10628) Add node usage metrics in SLS

2021-02-15 Thread VADAGA ANANYO RAO (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

VADAGA ANANYO RAO updated YARN-10628:
-
Attachment: YARN-10628.0001.patch

> Add node usage metrics in SLS
> -
>
> Key: YARN-10628
> URL: https://issues.apache.org/jira/browse/YARN-10628
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler-load-simulator
>Affects Versions: 3.3.1
>Reporter: VADAGA ANANYO RAO
>Assignee: VADAGA ANANYO RAO
>Priority: Major
> Attachments: Nodes_memory_usage.png, Nodes_vcores_usage.png, 
> YARN-10628.0001.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Given the work around container packing going on in YARN schedulers, it would 
> be beneficial to have charts showing the usage per node in SLS. This will 
> help to improve container packing algorithms for more efficient packings.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10622:

Attachment: (was: YARN-10622.001.patch)

> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be 
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
>   return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
>   leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
>   }
> {code}
> This, however does not take childless ParentQueues (which was introduced in 
> YARN-10596) into account. 
> A childless ParentQueue will throw a NPE in 
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
>   RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the 
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but 
> its name is present in the leafQueueNames as stated before.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10622:

Attachment: YARN-10622.001.patch

> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be 
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
>   return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
>   leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
>   }
> {code}
> This, however does not take childless ParentQueues (which was introduced in 
> YARN-10596) into account. 
> A childless ParentQueue will throw a NPE in 
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
>   RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the 
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but 
> its name is present in the leafQueueNames as stated before.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10622:

Attachment: YARN-10622.001.patch

> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be 
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
>   return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
>   leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
>   }
> {code}
> This, however does not take childless ParentQueues (which was introduced in 
> YARN-10596) into account. 
> A childless ParentQueue will throw a NPE in 
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
>   RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the 
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but 
> its name is present in the leafQueueNames as stated before.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10622:

Attachment: (was: YARN-10622.001.patch)

> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be 
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
>   return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
>   leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
>   }
> {code}
> This, however does not take childless ParentQueues (which was introduced in 
> YARN-10596) into account. 
> A childless ParentQueue will throw a NPE in 
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
>   RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the 
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but 
> its name is present in the leafQueueNames as stated before.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10622:

Attachment: (was: YARN-10622.001.patch)

> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be 
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
>   return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
>   leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
>   }
> {code}
> This, however does not take childless ParentQueues (which was introduced in 
> YARN-10596) into account. 
> A childless ParentQueue will throw a NPE in 
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
>   RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the 
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but 
> its name is present in the leafQueueNames as stated before.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10622:

Attachment: YARN-10622.001.patch

> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be 
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
>   return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
>   leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
>   }
> {code}
> This, however does not take childless ParentQueues (which was introduced in 
> YARN-10596) into account. 
> A childless ParentQueue will throw a NPE in 
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
>   RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the 
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but 
> its name is present in the leafQueueNames as stated before.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10622:

Attachment: YARN-10622.001.patch

> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be 
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
>   return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
>   leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
>   }
> {code}
> This, however does not take childless ParentQueues (which was introduced in 
> YARN-10596) into account. 
> A childless ParentQueue will throw a NPE in 
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
>   RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the 
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but 
> its name is present in the leafQueueNames as stated before.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284671#comment-17284671
 ] 

Hadoop QA commented on YARN-10622:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
41s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
51s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m  5s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 55s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
34s{color} | {color:green}{color} | {color:green} the

[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used

2021-02-15 Thread Andras Gyori (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284614#comment-17284614
 ] 

Andras Gyori commented on YARN-10532:
-

Thank you [~zhuqi], the latest patch looks good to me. If someone else has 
feedbacks, can you make sure to add some additional logging info in policy? In 
this current state, I only saw Removing queue entry, which could be elaborated 
a bit. My suggestion is to add a logging info when you send a deletion event to 
CS in the policy.

> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is 
> not being used
> 
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch, 
> YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, 
> YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, 
> YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, 
> YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, 
> YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, 
> YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, 
> image-2021-02-12-21-32-02-267.png
>
>
> It's better if we can delete auto-created queues when they are not in use for 
> a period of time (like 5 mins). It will be helpful when we have a large 
> number of auto-created queues (e.g. from 500 users), but only a small subset 
> of queues are actively used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10625) FairScheduler: add global flag to disable AM-preemption

2021-02-15 Thread Peter Bacsko (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284749#comment-17284749
 ] 

Peter Bacsko commented on YARN-10625:
-

[~bteke] / [~snemeth] please review this patch if you have some free cycles.

I think a doc update is also desirable, I'd do that in a follow-up JIRA.

> FairScheduler: add global flag to disable AM-preemption
> ---
>
> Key: YARN-10625
> URL: https://issues.apache.org/jira/browse/YARN-10625
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.3.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10625-001.patch
>
>
> YARN-9537 added a feature to disable AM preemption on a per queue basis.
> This is a nice enhancement, but it's very inconvenient if the cluster has a 
> lot of queues or queues dynamically created/deleted regularly (static queue 
> configuration changes).
> It's a legitimate use-case to have AM preemption turned off completely. To 
> make it easier, add property which acts as a global flag for this feature.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.

2021-02-15 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284704#comment-17284704
 ] 

Qi Zhu commented on YARN-10624:
---

Thanks for [~gandras] for your review.

[~ztang] [~snemeth]

If you could help commit it, when you are free, thanks.

> Support max queues limit configuration in new auto created queue, consistent 
> with old auto created.
> ---
>
> Key: YARN-10624
> URL: https://issues.apache.org/jira/browse/YARN-10624
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10624.001.patch, YARN-10624.002.patch
>
>
> Since old created leaf queue has the max leaf queues limit, i think we also 
> should support this in new auto created queue, both the auto created leaf and 
> the auto created parent need limits.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used

2021-02-15 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284706#comment-17284706
 ] 

Qi Zhu commented on YARN-10532:
---

Thanks  [~gandras]  for your patient review and valid suggestions, i will add 
more logging info in next patch.

 

> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is 
> not being used
> 
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch, 
> YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, 
> YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, 
> YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, 
> YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, 
> YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, 
> YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, 
> image-2021-02-12-21-32-02-267.png
>
>
> It's better if we can delete auto-created queues when they are not in use for 
> a period of time (like 5 mins). It will be helpful when we have a large 
> number of auto-created queues (e.g. from 500 users), but only a small subset 
> of queues are actively used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10625) FairScheduler: add global flag to disable AM-preemption

2021-02-15 Thread Peter Bacsko (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated YARN-10625:

Attachment: YARN-10625-001.patch

> FairScheduler: add global flag to disable AM-preemption
> ---
>
> Key: YARN-10625
> URL: https://issues.apache.org/jira/browse/YARN-10625
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.3.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10625-001.patch
>
>
> YARN-9537 added a feature to disable AM preemption on a per queue basis.
> This is a nice enhancement, but it's very inconvenient if the cluster has a 
> lot of queues or queues dynamically created/deleted regularly (static queue 
> configuration changes).
> It's a legitimate use-case to have AM preemption turned off completely. To 
> make it easier, add property which acts as a global flag for this feature.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Szilard Nemeth (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284739#comment-17284739
 ] 

Szilard Nemeth commented on YARN-10622:
---

Thanks [~gandras] for working on this,
Patch LGTM, committed to trunk.


> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be 
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
>   return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
>   leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
>   }
> {code}
> This, however does not take childless ParentQueues (which was introduced in 
> YARN-10596) into account. 
> A childless ParentQueue will throw a NPE in 
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
>   RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the 
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but 
> its name is present in the leafQueueNames as stated before.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10622) Fix preemption policy to exclude childless ParentQueues

2021-02-15 Thread Szilard Nemeth (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated YARN-10622:
--
Fix Version/s: 3.4.0

> Fix preemption policy to exclude childless ParentQueues
> ---
>
> Key: YARN-10622
> URL: https://issues.apache.org/jira/browse/YARN-10622
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10622.001.patch
>
>
> ProportionalCapacityPreemptionPolicy selects the potential LeafQueues to be 
> preempted by this logic:
> {code:java}
> private Set getLeafQueueNames(TempQueuePerPartition q) {
> // If its a ManagedParentQueue, it might not have any children
> if ((q.children == null || q.children.isEmpty())
> && !(q.parentQueue instanceof ManagedParentQueue)) {
>   return ImmutableSet.of(q.queueName);
> }
> Set leafQueueNames = new HashSet<>();
> for (TempQueuePerPartition child : q.children) {
>   leafQueueNames.addAll(getLeafQueueNames(child));
> }
> return leafQueueNames;
>   }
> {code}
> This, however does not take childless ParentQueues (which was introduced in 
> YARN-10596) into account. 
> A childless ParentQueue will throw a NPE in 
> FifoCandidatesSelector#selectCandidates:
> {code:java}
> LeafQueue leafQueue = preemptionContext.getQueueByPartition(queueName,
>   RMNodeLabelsManager.NO_LABEL).leafQueue;
> {code}
> TempQueuePerPartition has a leafQueue member variable, which is null, if the 
> queue is not a LeafQueue. In case of childless ParentQueue, it is null, but 
> its name is present in the leafQueueNames as stated before.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.

2021-02-15 Thread Szilard Nemeth (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated YARN-10624:
--
Fix Version/s: 3.4.0

> Support max queues limit configuration in new auto created queue, consistent 
> with old auto created.
> ---
>
> Key: YARN-10624
> URL: https://issues.apache.org/jira/browse/YARN-10624
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10624.001.patch, YARN-10624.002.patch
>
>
> Since old created leaf queue has the max leaf queues limit, i think we also 
> should support this in new auto created queue, both the auto created leaf and 
> the auto created parent need limits.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.

2021-02-15 Thread Szilard Nemeth (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284709#comment-17284709
 ] 

Szilard Nemeth commented on YARN-10624:
---

Thanks [~zhuqi] for working on this.
Patch LGTM, committed to trunk.
Thanks [~gandras] and [~bteke] for the reviews.

> Support max queues limit configuration in new auto created queue, consistent 
> with old auto created.
> ---
>
> Key: YARN-10624
> URL: https://issues.apache.org/jira/browse/YARN-10624
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10624.001.patch, YARN-10624.002.patch
>
>
> Since old created leaf queue has the max leaf queues limit, i think we also 
> should support this in new auto created queue, both the auto created leaf and 
> the auto created parent need limits.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10625) FairScheduler: add global flag to disable AM-preemption

2021-02-15 Thread Benjamin Teke (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284777#comment-17284777
 ] 

Benjamin Teke commented on YARN-10625:
--

[~pbacsko] thanks for the patch. LGTM +1 (non-binding). I agree with the doc 
update, as the global property doesn't overwrite the queue level ones, which 
can be non-trivial.

> FairScheduler: add global flag to disable AM-preemption
> ---
>
> Key: YARN-10625
> URL: https://issues.apache.org/jira/browse/YARN-10625
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.3.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10625-001.patch
>
>
> YARN-9537 added a feature to disable AM preemption on a per queue basis.
> This is a nice enhancement, but it's very inconvenient if the cluster has a 
> lot of queues or queues dynamically created/deleted regularly (static queue 
> configuration changes).
> It's a legitimate use-case to have AM preemption turned off completely. To 
> make it easier, add property which acts as a global flag for this feature.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10625) FairScheduler: add global flag to disable AM-preemption

2021-02-15 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284823#comment-17284823
 ] 

Hadoop QA commented on YARN-10625:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
23s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 
59s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 51s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 12s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
43s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} the