[jira] [Commented] (YARN-9615) Add dispatcher metrics to RM

2021-02-17 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286324#comment-17286324
 ] 

Qi Zhu commented on YARN-9615:
--

cc [~bteke] [~gandras]  [~pbacsko] 

If you any advice about this?

Thanks.

> Add dispatcher metrics to RM
> 
>
> Key: YARN-9615
> URL: https://issues.apache.org/jira/browse/YARN-9615
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-9615.001.patch, YARN-9615.poc.patch, 
> screenshot-1.png
>
>
> It'd be good to have counts/processing times for each event type in RM async 
> dispatcher and scheduler async dispatcher.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10632) Make maximum depth allowed configurable.

2021-02-17 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286302#comment-17286302
 ] 

Qi Zhu edited comment on YARN-10632 at 2/18/21, 6:53 AM:
-

cc [~bteke]  [~gandras]  [~snemeth] [~pbacsko] 

I think for queue auto creation v2, it is needed that we make the max depth 
configurable.

For example, i want the root the max depth to 5, but i want the max depth under 
some parent queue to be 1.

If you could help review this?

Thanks. 


was (Author: zhuqi):
cc [~bteke]  [~gandras]  [~snemeth] [~pbacsko] 

I think for some queue auto creation v2, it is needed that we make the max 
depth configurable.

For example, i want the root the max depth to 5, but i want the max depth under 
some parent queue to be 1.

If you could help review this?

Thanks. 

> Make maximum depth allowed configurable.
> 
>
> Key: YARN-10632
> URL: https://issues.apache.org/jira/browse/YARN-10632
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10632.001.patch
>
>
> Now the max depth allowed are fixed to 2. But i think this should be 
> configurable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10632) Make maximum depth allowed configurable.

2021-02-17 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286302#comment-17286302
 ] 

Qi Zhu commented on YARN-10632:
---

cc [~bteke]  [~gandras]  [~snemeth] [~pbacsko] 

I think for some queue auto creation v2, it is needed that we make the max 
depth configurable.

For example, i want the root the max depth to 5, but i want the max depth under 
some parent queue to be 1.

Thanks. 

> Make maximum depth allowed configurable.
> 
>
> Key: YARN-10632
> URL: https://issues.apache.org/jira/browse/YARN-10632
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10632.001.patch
>
>
> Now the max depth allowed are fixed to 2. But i think this should be 
> configurable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10632) Make maximum depth allowed configurable.

2021-02-17 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286302#comment-17286302
 ] 

Qi Zhu edited comment on YARN-10632 at 2/18/21, 6:30 AM:
-

cc [~bteke]  [~gandras]  [~snemeth] [~pbacsko] 

I think for some queue auto creation v2, it is needed that we make the max 
depth configurable.

For example, i want the root the max depth to 5, but i want the max depth under 
some parent queue to be 1.

If you could help review this?

Thanks. 


was (Author: zhuqi):
cc [~bteke]  [~gandras]  [~snemeth] [~pbacsko] 

I think for some queue auto creation v2, it is needed that we make the max 
depth configurable.

For example, i want the root the max depth to 5, but i want the max depth under 
some parent queue to be 1.

Thanks. 

> Make maximum depth allowed configurable.
> 
>
> Key: YARN-10632
> URL: https://issues.apache.org/jira/browse/YARN-10632
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10632.001.patch
>
>
> Now the max depth allowed are fixed to 2. But i think this should be 
> configurable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10632) Make maximum depth allowed configurable.

2021-02-17 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286301#comment-17286301
 ] 

Hadoop QA commented on YARN-10632:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m  
7s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red}  0m  7s{color} 
| {color:red}{color} | {color:red} Unprocessed flag(s): 
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/633/artifact/out/Dockerfile
 |
| JIRA Issue | YARN-10632 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/13020610/YARN-10632.001.patch |
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/633/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Make maximum depth allowed configurable.
> 
>
> Key: YARN-10632
> URL: https://issues.apache.org/jira/browse/YARN-10632
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10632.001.patch
>
>
> Now the max depth allowed are fixed to 2. But i think this should be 
> configurable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10633) setup yarn federation failed

2021-02-17 Thread yuguang (Jira) 
yuguang created YARN-10633:
--

 Summary: setup yarn federation failed
 Key: YARN-10633
 URL: https://issues.apache.org/jira/browse/YARN-10633
 Project: Hadoop YARN
  Issue Type: Bug
  Components: federation
Affects Versions: 3.2.2
Reporter: yuguang


Hi

I am trying to setup yarn federation mode. But after I add  below configuration 
in etc/hadoop/yarn-site.xml


yarn.federation.enabled
true


then when I run yarn node -list  . Get below error . Also the historyserver 
service can not be started either .

I am using hadoop-3.2.2 version . 

[root@yarna hadoop-3.2.2]# yarn node -list

2021-02-18 05:51:39,178 INFO service.AbstractService: Service 
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl failed in state 
STARTEDjava.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for 
length 0 at 
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.init(ConfiguredRMFailoverProxyProvider.java:62)
 at 
org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:175)
 at org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:130) at 
org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:103) at 
org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
 at 
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:233)
 at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) 
at 
org.apache.hadoop.yarn.client.cli.YarnCLI.createAndStartYarnClient(YarnCLI.java:55)
 at org.apache.hadoop.yarn.client.cli.NodeCLI.run(NodeCLI.java:110) at 
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at 
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at 
org.apache.hadoop.yarn.client.cli.NodeCLI.main(NodeCLI.java:62)Exception in 
thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds 
for length 0 at 
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.init(ConfiguredRMFailoverProxyProvider.java:62)
 at 
org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:175)
 at org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:130) at 
org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:103) at 
org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
 at 
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:233)
 at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) 
at 
org.apache.hadoop.yarn.client.cli.YarnCLI.createAndStartYarnClient(YarnCLI.java:55)
 at org.apache.hadoop.yarn.client.cli.NodeCLI.run(NodeCLI.java:110) at 
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at 
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at 
org.apache.hadoop.yarn.client.cli.NodeCLI.main(NodeCLI.java:62)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10633) setup yarn federation failed

2021-02-17 Thread yuguang (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286291#comment-17286291
 ] 

yuguang commented on YARN-10633:


If I delete the configuration, the historyserver can be started up  and yarn 
node -list command works fine. 

 

> setup yarn federation failed
> 
>
> Key: YARN-10633
> URL: https://issues.apache.org/jira/browse/YARN-10633
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: federation
>Affects Versions: 3.2.2
>Reporter: yuguang
>Priority: Major
>
> Hi
> I am trying to setup yarn federation mode. But after I add  below 
> configuration in etc/hadoop/yarn-site.xml
> 
> yarn.federation.enabled
> true
> 
> then when I run yarn node -list  . Get below error . Also the historyserver 
> service can not be started either .
> I am using hadoop-3.2.2 version . 
> [root@yarna hadoop-3.2.2]# yarn node -list
> 2021-02-18 05:51:39,178 INFO service.AbstractService: Service 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl failed in state 
> STARTEDjava.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for 
> length 0 at 
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.init(ConfiguredRMFailoverProxyProvider.java:62)
>  at 
> org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:175)
>  at org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:130) 
> at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:103) at 
> org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
>  at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:233)
>  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) 
> at 
> org.apache.hadoop.yarn.client.cli.YarnCLI.createAndStartYarnClient(YarnCLI.java:55)
>  at org.apache.hadoop.yarn.client.cli.NodeCLI.run(NodeCLI.java:110) at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at 
> org.apache.hadoop.yarn.client.cli.NodeCLI.main(NodeCLI.java:62)Exception in 
> thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds 
> for length 0 at 
> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.init(ConfiguredRMFailoverProxyProvider.java:62)
>  at 
> org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:175)
>  at org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:130) 
> at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:103) at 
> org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
>  at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:233)
>  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) 
> at 
> org.apache.hadoop.yarn.client.cli.YarnCLI.createAndStartYarnClient(YarnCLI.java:55)
>  at org.apache.hadoop.yarn.client.cli.NodeCLI.run(NodeCLI.java:110) at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at 
> org.apache.hadoop.yarn.client.cli.NodeCLI.main(NodeCLI.java:62)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10437) Destroy yarn service if any YarnException occurs during submitApp

2021-02-17 Thread Brahma Reddy Battula (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286273#comment-17286273
 ] 

Brahma Reddy Battula commented on YARN-10437:
-

[~dmmkr] thanks for reporting.. lgtm.. [~hemanthboyina] do you've any further 
comments..?

> Destroy yarn service if any YarnException occurs during submitApp
> -
>
> Key: YARN-10437
> URL: https://issues.apache.org/jira/browse/YARN-10437
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-native-services
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Minor
> Attachments: YARN-10437.001.patch, YARN-10437.002.patch
>
>
> If a user submits a yarn service with configuration such that it causes an 
> exception during application submission, the files related to the service are 
> not cleared from hdfs automatically. Ideally the files stored to hdfs cannot 
> be used in future to start or stop the service as the configuration itself is 
> invalid. So, we should destroy the service and remove the residual files in 
> hdfs, if any YarnException is thrown.
> For example if the user submits a service with configuring with "memory" more 
> than the maximum resource, the service fails but the files in hdfs are not 
> cleared.  But these files should be cleared.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10439) Yarn Service AM listens on all IP's on the machine

2021-02-17 Thread Brahma Reddy Battula (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286272#comment-17286272
 ] 

Brahma Reddy Battula commented on YARN-10439:
-

[~dmmkr] thanks for reporting.. Yes, it's security issue as this will open.

Changes lgtm..hold to commit till this weekend.

 

> Yarn Service AM listens on all IP's on the machine
> --
>
> Key: YARN-10439
> URL: https://issues.apache.org/jira/browse/YARN-10439
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: security, yarn-native-services
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Minor
> Attachments: YARN-10439.001.patch, YARN-10439.002.patch
>
>
> In ClientAMService.java, rpc server is created without passing hostname, due 
> to which the client listens on 0.0.0.0, which is a bad practise.
>  
> {{InetSocketAddress address = {color:#cc7832}new 
> {color}InetSocketAddress({color:#6897bb}0{color}){color:#cc7832};{color}}}
> {{{color:#9876aa}server {color}= 
> rpc.getServer(ClientAMProtocol.{color:#cc7832}class, this, 
> {color}address{color:#cc7832}, {color}conf{color:#cc7832},{color} 
> {color:#9876aa}context{color}.{color:#9876aa}secretManager{color}{color:#cc7832},
>  {color}{color:#6897bb}1{color}){color:#cc7832};{color}}}
>  
> Also, a new configuration must be added similar to 
> "yarn.app.mapreduce.am.job.client.port-range", so that client can configure 
> port range for yarn service AM to bind.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10441) Add support for hadoop.http.rmwebapp.scheduler.page.class

2021-02-17 Thread Brahma Reddy Battula (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286271#comment-17286271
 ] 

Brahma Reddy Battula commented on YARN-10441:
-

[~dmmkr] thanks reporting.. Changes looks good to me...will hold the commit 
till this weekend.

> Add support for hadoop.http.rmwebapp.scheduler.page.class
> -
>
> Key: YARN-10441
> URL: https://issues.apache.org/jira/browse/YARN-10441
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Major
> Attachments: YARN-10441.001.patch, YARN-10441.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-10361 the existing 
> configuration  of hadoop.http.rmwebapp.scheduler.page.class is updated to 
> yarn.http.rmwebapp.scheduler.page.class, which causes incompatibility with 
> old versions, It is better to make the old configuration deprecated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10466) Fix NullPointerException in yarn-services Component.java

2021-02-17 Thread Brahma Reddy Battula (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286267#comment-17286267
 ] 

Brahma Reddy Battula commented on YARN-10466:
-

[~dmmkr] thanks for reporting this. 

one minor nit: how about changing the log level to info, as this can be given 
hint(as usually loglevel  will not debug by default)..?

> Fix NullPointerException in  yarn-services Component.java
> -
>
> Key: YARN-10466
> URL: https://issues.apache.org/jira/browse/YARN-10466
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Minor
> Attachments: YARN-10466.001.patch
>
>
> Due to changes in 
> [YARN-10219|https://issues.apache.org/jira/browse/YARN-10219]   where the 
> constraint is initialised as null, there might be few scenarios in which NPE 
> can be thrown  in requestContainers method.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10125) In Federation, kill application from client does not kill Unmanaged AM's and containers launched by Unmanaged AM

2021-02-17 Thread Brahma Reddy Battula (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286265#comment-17286265
 ] 

Brahma Reddy Battula commented on YARN-10125:
-

[~dmmkr]  thanks for reporting. Even this should be handled.. @[Giovanni Matteo 
Fumarola and 
|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=giovanni.fumarola] 
[~subru] , any idea on this?

> In Federation, kill application from client does not kill Unmanaged AM's and 
> containers launched by Unmanaged AM
> 
>
> Key: YARN-10125
> URL: https://issues.apache.org/jira/browse/YARN-10125
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client, federation, router
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Major
> Attachments: YARN-10125.001.patch
>
>
> In Federation, killing an application from client using "bin/yarn application 
> -kill ", kills the containers only of the home subcluster, 
> the Unmanaged AM and the containers launched in other subcluster are not 
> being killed causing blocking of resources.
> The containers get killed after the task gets completed and The unmanaged AM 
> gets killed after 10 minutes of killing the application, killing any 
> remaining running containers in that subcluster.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-17 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286264#comment-17286264
 ] 

Hadoop QA commented on YARN-10609:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m  
9s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red}  0m  7s{color} 
| {color:red}{color} | {color:red} Unprocessed flag(s): 
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/631/artifact/out/Dockerfile
 |
| JIRA Issue | YARN-10609 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/13020601/YARN-10609.005.patch |
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/631/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch, YARN-10609.002.patch, 
> YARN-10609.003.patch, YARN-10609.004.patch, YARN-10609.005.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10632) Make maximum depth allowed configurable.

2021-02-17 Thread Qi Zhu (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qi Zhu updated YARN-10632:
--
Fix Version/s: (was: 3.4.0)

> Make maximum depth allowed configurable.
> 
>
> Key: YARN-10632
> URL: https://issues.apache.org/jira/browse/YARN-10632
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
>
> Now the max depth allowed are fixed to 2. But i think this should be 
> configurable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10487) Support getQueueUserAcls, listReservations, getApplicationAttempts, getContainerReport, getContainers, getResourceTypeInfo API's for Federation

2021-02-17 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286263#comment-17286263
 ] 

Hadoop QA commented on YARN-10487:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  7s{color} 
| {color:red}{color} | {color:red} YARN-10487 does not apply to trunk. Rebase 
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for 
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10487 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/13015118/YARN-10487.001.patch |
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/632/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Support getQueueUserAcls, listReservations, getApplicationAttempts, 
> getContainerReport, getContainers, getResourceTypeInfo API's for Federation
> ---
>
> Key: YARN-10487
> URL: https://issues.apache.org/jira/browse/YARN-10487
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: D M Murali Krishna Reddy
>Assignee: D M Murali Krishna Reddy
>Priority: Major
> Attachments: YARN-10487.001.patch
>
>
> Support getQueueUserAcls, listReservations, getApplicationAttempts, 
> getContainerReport, getContainers, getResourceTypeInfo API's for Federation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-17 Thread Qi Zhu (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qi Zhu updated YARN-10609:
--
Attachment: YARN-10609.005.patch

> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch, YARN-10609.002.patch, 
> YARN-10609.003.patch, YARN-10609.004.patch, YARN-10609.005.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-17 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286252#comment-17286252
 ] 

Qi Zhu commented on YARN-10609:
---

Thanks a lot [~bteke]  for last check, i have fixed it in latest patch.

[~gandras] [~snemeth] [~pbacsko]  

If you have any other advice?

Thanks.

> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch, YARN-10609.002.patch, 
> YARN-10609.003.patch, YARN-10609.004.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-17 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286250#comment-17286250
 ] 

Hadoop QA commented on YARN-10609:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
11s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red}  0m  7s{color} 
| {color:red}{color} | {color:red} Unprocessed flag(s): 
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/630/artifact/out/Dockerfile
 |
| JIRA Issue | YARN-10609 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/13020597/YARN-10609.004.patch |
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/630/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch, YARN-10609.002.patch, 
> YARN-10609.003.patch, YARN-10609.004.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-17 Thread Qi Zhu (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qi Zhu updated YARN-10609:
--
Attachment: YARN-10609.004.patch

> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch, YARN-10609.002.patch, 
> YARN-10609.003.patch, YARN-10609.004.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10617) Fifo and Fair intra-queue preemption goes on indefinitely when apps are in pending state due to max AM limit reached

2021-02-17 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286248#comment-17286248
 ] 

Hadoop QA commented on YARN-10617:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
15s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red}  0m  7s{color} 
| {color:red}{color} | {color:red} Unprocessed flag(s): 
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/629/artifact/out/Dockerfile
 |
| JIRA Issue | YARN-10617 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/13020461/YARN-10617.0001.patch 
|
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/629/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Fifo and Fair intra-queue preemption goes on indefinitely when apps are in 
> pending state due to max AM limit reached
> 
>
> Key: YARN-10617
> URL: https://issues.apache.org/jira/browse/YARN-10617
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacity scheduler
>Affects Versions: 3.1.1
>Reporter: VADAGA ANANYO RAO
>Assignee: VADAGA ANANYO RAO
>Priority: Major
> Attachments: YARN-10617.0001.patch
>
>
> This case occurs when:
> 1. an application gets submitted in a cluster running at max-AM limit.
> 2. The new job requests AM resource. So it has 1 pending request.
> 3. To fulfil this request, the preemption logic preempts 1 resource from a 
> running app.
> 4. Because the cluster is at max-AM limit, the scheduler re-assigns the 
> preempted container back to the running app.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-17 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285977#comment-17285977
 ] 

Hadoop QA commented on YARN-10627:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
29s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red}  0m  7s{color} 
| {color:red}{color} | {color:red} Unprocessed flag(s): 
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/628/artifact/out/Dockerfile
 |
| JIRA Issue | YARN-10627 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/13020579/YARN-10627.001.patch |
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/628/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>

[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode

2021-02-17 Thread Benjamin Teke (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.001.patch

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   + " [= configuredState ]" + "\n" + "acls = " + aclsString
>   + " [= configuredAcls ]" + "\n"
>   +

[jira] [Commented] (YARN-10628) Add node usage metrics in SLS

2021-02-17 Thread Szilard Nemeth (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285953#comment-17285953
 ] 

Szilard Nemeth commented on YARN-10628:
---

Hi [~ananyo_rao],
If you need reviews on SLS, feel free to ping me.

> Add node usage metrics in SLS
> -
>
> Key: YARN-10628
> URL: https://issues.apache.org/jira/browse/YARN-10628
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scheduler-load-simulator
>Affects Versions: 3.3.1
>Reporter: VADAGA ANANYO RAO
>Assignee: VADAGA ANANYO RAO
>Priority: Major
> Attachments: Nodes_memory_usage.png, Nodes_vcores_usage.png, 
> YARN-10628.0001.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Given the work around container packing going on in YARN schedulers, it would 
> be beneficial to have charts showing the usage per node in SLS. This will 
> help to improve container packing algorithms for more efficient packings.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-17 Thread Benjamin Teke (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285900#comment-17285900
 ] 

Benjamin Teke commented on YARN-10609:
--

[~zhuqi] thanks for the update. There may have been a mixup as the sentences 
with the "default value" and the "specified as float" are duplicated. If you 
remove those it's +1 from my side. Thanks!

User limit factor provides a way to control the max amount of resources that a 
single user can consume. It is the multiple of the queue's capacity. By default 
this is set to 1 which ensures that a single user can never take more than the 
queue's configured capacity irrespective of how idle the cluster is. Increasing 
it means a single user can use more than the minimum capacity of the cluster, 
while decreasing it results in lower maximum resources. -By default this is set 
to 1 which ensures that a single user can never take more than the queue's 
configured capacity irrespective of how idle the cluster is. Value is specified 
as a float.- Setting this to -1 will disable the feature. Value is specified as 
a float. Note: using the flexible auto queue creation 
(yarn.scheduler.capacity..auto-queue-creation-v2) with weights will 
automatically set this property to -1, as the dynamic queues will be created 
with the hardcoded weight of 1 and in idle cluster scenarios they should be 
able to use more resources than calculated.

> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch, YARN-10609.002.patch, 
> YARN-10609.003.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications

2021-02-17 Thread Andras Gyori (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285845#comment-17285845
 ] 

Andras Gyori commented on YARN-10513:
-

Uploaded a new revision, in which I extend the queue selection view as well. 
Also added the OrderingPolicy attribute to the queue.

> CS Flexible Auto Queue Creation RM UIv2 modifications
> -
>
> Key: YARN-10513
> URL: https://issues.apache.org/jira/browse/YARN-10513
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: Screenshot 2021-02-04 at 12.54.25.png, Screenshot 
> 2021-02-04 at 12.54.52.png, Screenshot 2021-02-04 at 12.55.10.png, Screenshot 
> 2021-02-08 at 10.34.32.png, Screenshot 2021-02-17 at 15.22.30.png, 
> YARN-10513.001.patch, YARN-10513.002.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications

2021-02-17 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10513:

Attachment: Screenshot 2021-02-17 at 15.22.30.png

> CS Flexible Auto Queue Creation RM UIv2 modifications
> -
>
> Key: YARN-10513
> URL: https://issues.apache.org/jira/browse/YARN-10513
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: Screenshot 2021-02-04 at 12.54.25.png, Screenshot 
> 2021-02-04 at 12.54.52.png, Screenshot 2021-02-04 at 12.55.10.png, Screenshot 
> 2021-02-08 at 10.34.32.png, Screenshot 2021-02-17 at 15.22.30.png, 
> YARN-10513.001.patch, YARN-10513.002.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications

2021-02-17 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10513:

Attachment: YARN-10513.002.patch

> CS Flexible Auto Queue Creation RM UIv2 modifications
> -
>
> Key: YARN-10513
> URL: https://issues.apache.org/jira/browse/YARN-10513
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: Screenshot 2021-02-04 at 12.54.25.png, Screenshot 
> 2021-02-04 at 12.54.52.png, Screenshot 2021-02-04 at 12.55.10.png, Screenshot 
> 2021-02-08 at 10.34.32.png, Screenshot 2021-02-17 at 15.22.30.png, 
> YARN-10513.001.patch, YARN-10513.002.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'

2021-02-17 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279673#comment-17279673
 ] 

Qi Zhu edited comment on YARN-10178 at 2/17/21, 12:50 PM:
--

The test failed is not related. These tests are all passed locally.

cc [~wangda]  [~gandras] [~ztang] [~ebadger]  [~bteke]  [~tuyu]

If you any other advice.

 


was (Author: zhuqi):
The test failed is not related. These tests are all passed locally.

cc [~wangda] [~ztang] [~ebadger]  [~bteke]  [~tuyu]

If you any other advice.

 

> Global Scheduler async thread crash caused by 'Comparison method violates its 
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10178.001.patch, YARN-10178.002.patch, 
> YARN-10178.003.patch, YARN-10178.004.patch, YARN-10178.005.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received 
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread, 
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException: 
> Comparison method violates its general contract!  
>at 
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has  few require 
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw 
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know 
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use 
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign 
> container,and construct a CSAssignment struct, and use 
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService  will tryCommit this CSAssignment,look tryCommit 
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
>   long commitStart = System.nanoTime();
>   ResourceCommitRequest request =
>   (ResourceCommitRequest) r;
>  
>   ...
>   boolean isSuccess = false;
>   if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for

[jira] [Created] (YARN-10632) Make maximum depth allowed configurable.

2021-02-17 Thread Qi Zhu (Jira) 
Qi Zhu created YARN-10632:
-

 Summary: Make maximum depth allowed configurable.
 Key: YARN-10632
 URL: https://issues.apache.org/jira/browse/YARN-10632
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Qi Zhu
Assignee: Qi Zhu
 Fix For: 3.4.0


Now the max depth allowed are fixed to 2. But i think this should be 
configurable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager

2021-02-17 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285803#comment-17285803
 ] 

Qi Zhu commented on YARN-10258:
---

Thank you [~gb.ana...@gmail.com] for your contribution. Patch LGTM. 

> Add metrics for 'ApplicationsRunning' in NodeManager
> 
>
> Key: YARN-10258
> URL: https://issues.apache.org/jira/browse/YARN-10258
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.1.3
>Reporter: ANANDA G B
>Assignee: ANANDA G B
>Priority: Minor
> Attachments: YARN-10258-001.patch, YARN-10258-002.patch
>
>
> Add metrics for 'ApplicationsRunning' in NodeManagers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10631) Document AM preemption related changes (YARN-9537 and YARN-10625)

2021-02-17 Thread Peter Bacsko (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated YARN-10631:

Summary: Document AM preemption related changes (YARN-9537 and YARN-10625)  
(was: Document AM-preemption related changes (YARN-9537 and YARN-10625))

> Document AM preemption related changes (YARN-9537 and YARN-10625)
> -
>
> Key: YARN-10631
> URL: https://issues.apache.org/jira/browse/YARN-10631
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> Preemption-related changes were introduced in YARN-9537 and YARN-10625.
> These also introduce new properties which are not documented for Fair 
> Scheduler. Extend the documentation with these enhancements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10631) Document AM-preemption related changes (YARN-9537 and YARN-10625)

2021-02-17 Thread Peter Bacsko (Jira) 
Peter Bacsko created YARN-10631:
---

 Summary: Document AM-preemption related changes (YARN-9537 and 
YARN-10625)
 Key: YARN-10631
 URL: https://issues.apache.org/jira/browse/YARN-10631
 Project: Hadoop YARN
  Issue Type: Task
Reporter: Peter Bacsko
Assignee: Peter Bacsko


Preemption-related changes were introduced in YARN-9537 and YARN-10625.

These also introduce new properties which are not documented for Fair 
Scheduler. Extend the documentation with these enhancements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10630) Ambiguous queue name resolution in Yarn UIv2

2021-02-17 Thread Andras Gyori (Jira) 
Andras Gyori created YARN-10630:
---

 Summary: Ambiguous queue name resolution in Yarn UIv2
 Key: YARN-10630
 URL: https://issues.apache.org/jira/browse/YARN-10630
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn-ui-v2
Reporter: Andras Gyori
Assignee: Andras Gyori


Yarn UIv2 uses queueName instead of queuePath (which was added in the scheduler 
response in YARN-10610), which makes the queue resolution ambiguous in case of 
identical queue short names (eg. root.a.b <-> root.b). This causes invalid 
behaviour in multiple places.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager

2021-02-17 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285701#comment-17285701
 ] 

Hadoop QA commented on YARN-10258:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
15s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red}  0m  7s{color} 
| {color:red}{color} | {color:red} Unprocessed flag(s): 
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/626/artifact/out/Dockerfile
 |
| JIRA Issue | YARN-10258 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/13020547/YARN-10258-002.patch |
| Console output | 
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/626/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |


This message was automatically generated.



> Add metrics for 'ApplicationsRunning' in NodeManager
> 
>
> Key: YARN-10258
> URL: https://issues.apache.org/jira/browse/YARN-10258
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.1.3
>Reporter: ANANDA G B
>Assignee: ANANDA G B
>Priority: Minor
> Attachments: YARN-10258-001.patch, YARN-10258-002.patch
>
>
> Add metrics for 'ApplicationsRunning' in NodeManagers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager

2021-02-17 Thread Bilwa S T (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285697#comment-17285697
 ] 

Bilwa S T commented on YARN-10258:
--

Thank you [~gb.ana...@gmail.com] for your contribution. Patch LGTM. there are 
few checkstyle issues. Please fix. Resubmitting patch to trigger build again

> Add metrics for 'ApplicationsRunning' in NodeManager
> 
>
> Key: YARN-10258
> URL: https://issues.apache.org/jira/browse/YARN-10258
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.1.3
>Reporter: ANANDA G B
>Assignee: ANANDA G B
>Priority: Minor
> Attachments: YARN-10258-001.patch, YARN-10258-002.patch
>
>
> Add metrics for 'ApplicationsRunning' in NodeManagers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10258) Add metrics for 'ApplicationsRunning' in NodeManager

2021-02-17 Thread Bilwa S T (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bilwa S T updated YARN-10258:
-
Attachment: YARN-10258-002.patch

> Add metrics for 'ApplicationsRunning' in NodeManager
> 
>
> Key: YARN-10258
> URL: https://issues.apache.org/jira/browse/YARN-10258
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.1.3
>Reporter: ANANDA G B
>Assignee: ANANDA G B
>Priority: Minor
> Attachments: YARN-10258-001.patch, YARN-10258-002.patch
>
>
> Add metrics for 'ApplicationsRunning' in NodeManagers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org