[jira] [Resolved] (YARN-10803) [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException

2021-06-09 Thread Akira Ajisaka (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved YARN-10803.
--
Fix Version/s: 3.3.2
   3.4.0
   Resolution: Fixed

Committed to trunk and branch-3.3.

> [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider 
> fails by ClassCastException
> 
>
> Key: YARN-10803
> URL: https://issues.apache.org/jira/browse/YARN-10803
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/178/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
> {quote}
> [ERROR] 
> testFailoverChange(org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider) 
> Time elapsed: 0.024 s <<< ERROR! java.lang.ClassCastException: class 
> org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy cannot be 
> cast to class org.apache.hadoop.yarn.client.RMProxy 
> (org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider$TestProxy and 
> org.apache.hadoop.yarn.client.RMProxy are in unnamed module of loader 'app') 
> at 
> org.apache.hadoop.yarn.client.TestRMFailoverProxyProvider.testFailoverChange(TestRMFailoverProxyProvider.java:141)
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-10772) Stable API GetApplicationsRequest#newInstance compatibility broken by YARN-8363

2021-06-09 Thread Akira Ajisaka (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka reassigned YARN-10772:


Assignee: (was: Akira Ajisaka)

> Stable API GetApplicationsRequest#newInstance compatibility broken by 
> YARN-8363
> ---
>
> Key: YARN-10772
> URL: https://issues.apache.org/jira/browse/YARN-10772
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: api
>Affects Versions: 3.2.0
>Reporter: Wei-Chiu Chuang
>Priority: Major
>
> YARN-8363 migrated our usage of commons-lang to commons-lang3 in 3.2.0.
>  
> Unfortunately, it changed the API signature of 
> {code:java}
> /**
>  * 
>  * The request from clients to get a report of Applications matching the
>  * giving application types in the cluster from the
>  * ResourceManager.
>  * 
>  *
>  * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest)
>  *
>  * Setting any of the parameters to null, would just disable that
>  * filter
>  *
>  * @param scope {@link ApplicationsRequestScope} to filter by
>  * @param users list of users to filter by
>  * @param queues list of scheduler queues to filter by
>  * @param applicationTypes types of applications
>  * @param applicationTags application tags to filter by
>  * @param applicationStates application states to filter by
>  * @param startRange range of application start times to filter by
>  * @param finishRange range of application finish times to filter by
>  * @param limit number of applications to limit to
>  * @return {@link GetApplicationsRequest} to be used with
>  * {@link ApplicationClientProtocol#getApplications(GetApplicationsRequest)}
>  */
> @Public
> @Stable
> public static GetApplicationsRequest newInstance(
> ApplicationsRequestScope scope,
> Set users,
> Set queues,
> Set applicationTypes,
> Set applicationTags,
> EnumSet applicationStates,
> Range startRange,
> Range finishRange,
> Long limit) { {code}
> The startRange and finishRange changed type from LongRange to Range.
> It could cause problems when migrating applications, for example, from Hadoop 
> 3.1 to 3.3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Resolved] (YARN-10808) TestHBaseTimelineStorageUtils fails by NoClassDefFoundError

2021-06-09 Thread Akira Ajisaka (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved YARN-10808.
--
Resolution: Duplicate

> TestHBaseTimelineStorageUtils fails by NoClassDefFoundError
> ---
>
> Key: YARN-10808
> URL: https://issues.apache.org/jira/browse/YARN-10808
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Akira Ajisaka
>Priority: Major
>  Labels: newbie
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/529/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client.txt
> {quote}
> [ERROR] 
> testWithHbaseConfAtHdfsFileSystem(org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils)
>   Time elapsed: 5.89 s  <<< ERROR!
> java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2618)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2632)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1498)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:977)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:576)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils.testWithHbaseConfAtHdfsFileSystem(TestHBaseTimelineStorageUtils.java:86)
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10816) Avoid doing delegation token ops when yarn.timeline-service.http-authentication.type=simple

2021-06-09 Thread Tarun Parimi (Jira) 
Tarun Parimi created YARN-10816:
---

 Summary: Avoid doing delegation token ops when 
yarn.timeline-service.http-authentication.type=simple
 Key: YARN-10816
 URL: https://issues.apache.org/jira/browse/YARN-10816
 Project: Hadoop YARN
  Issue Type: Bug
  Components: timelineclient
Affects Versions: 3.4.0
Reporter: Tarun Parimi
Assignee: Tarun Parimi


YARN-10339 introduced changes to ensure that PseudoAuthenticationHandler is 
used in TimelineClient when 
yarn.timeline-service.http-authentication.type=simple

PseudoAuthenticationHandler doesn't support delegation token ops like get, 
renew and cancel since those ops strictly require SPNEGO auth to work. We don't 
use timeline delegation tokens when simple auth is used.

Prior to YARN-10339, Timeline delegation tokens were unnecessarily used when 
yarn.timeline-service.http-authentication.type=simple, but hadoop security was 
enabled. After YARN-10339, the tokens are not used when 
yarn.timeline-service.http-authentication.type=simple.

In a rolling upgrade scenario, we can have a client  which doesn't have 
YARN-10339 changes submitting an application and requests a Timeline delegation 
token even when yarn.timeline-service.http-authentication.type=simple. RM on 
the other hand can have YARN-10339 changes and so will result in error while 
trying to renew the token with PseudoAuthenticationHandler. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9049) Add application submit data to state store

2021-06-09 Thread Song Jiacheng (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360505#comment-17360505
 ] 

Song Jiacheng commented on YARN-9049:
-

Hi, [~botong], [~bibinchundatt]

I have developed a patch which persisted ApplicationSubmissionContext in zk, 
but I wonder if there will be too much pressure on zk, cause as we know every 
NM holds a connect to zk. Moreover, query for ApplicationSubmissionContext  
queries zk directly, not concerned about the cache.

> Add application submit data to state store
> --
>
> Key: YARN-9049
> URL: https://issues.apache.org/jira/browse/YARN-9049
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin Chundatt
>Assignee: Bibin Chundatt
>Priority: Major
> Attachments: YARN-9049.001.path
>
>
> As per the discussion in YARN-8898 we need to persist trimmend 
> ApplicationSubmissionContext details to federation State Store.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10815) Handle Invalid event: PAUSE_CONTAINER at Container state as SCHEDULED

2021-06-09 Thread Minni Mittal (Jira) 
Minni Mittal created YARN-10815:
---

 Summary: Handle Invalid event: PAUSE_CONTAINER at Container state 
as SCHEDULED
 Key: YARN-10815
 URL: https://issues.apache.org/jira/browse/YARN-10815
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Minni Mittal
Assignee: Minni Mittal






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10815) Handle Invalid event: PAUSE_CONTAINER at Container state SCHEDULED

2021-06-09 Thread Minni Mittal (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Minni Mittal updated YARN-10815:

Summary: Handle Invalid event: PAUSE_CONTAINER at Container state SCHEDULED 
 (was: Handle Invalid event: PAUSE_CONTAINER at Container state as SCHEDULED)

> Handle Invalid event: PAUSE_CONTAINER at Container state SCHEDULED
> --
>
> Key: YARN-10815
> URL: https://issues.apache.org/jira/browse/YARN-10815
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Minni Mittal
>Assignee: Minni Mittal
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10801) Fix Auto Queue template to properly set all configuration properties

2021-06-09 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360227#comment-17360227
 ] 

Hadoop QA commented on YARN-10801:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m 
46s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m 
 7s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 52s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m  
6s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are 
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  1m 
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green}{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 126 unchanged - 1 fixed = 126 total (was 127) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 54s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
|

[jira] [Created] (YARN-10814) YARN shouldn't start with empty hadoop.http.authentication.signature.secret.file

2021-06-09 Thread Benjamin Teke (Jira) 
Benjamin Teke created YARN-10814:


 Summary: YARN shouldn't start with empty 
hadoop.http.authentication.signature.secret.file
 Key: YARN-10814
 URL: https://issues.apache.org/jira/browse/YARN-10814
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Benjamin Teke


YARN currently throws the following warning upon accessing any REST endpoint 
when the configured http secret file exists but is empty:

{code:java}
2021-03-03 20:46:16,616 WARN org.eclipse.jetty.server.HttpChannel: /jmx
java.lang.IllegalArgumentException: Empty key
at 
java.base/javax.crypto.spec.SecretKeySpec.(SecretKeySpec.java:95)
at 
org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93)
at 
org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
at 
org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
at 
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1681)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:501)
at 
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at 
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
at 
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
at 
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834) 
{code}

YARN itself works, apps can be

[jira] [Commented] (YARN-10813) Root queue capacity is not set when using node labels

2021-06-09 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360140#comment-17360140
 ] 

Hadoop QA commented on YARN-10813:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 
57s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to 
include any new or modified tests. Please justify why no new tests are needed 
for this patch. Also please list what manual steps were performed to verify 
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 
 7s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
23s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
6s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m 48s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 23m 
37s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are 
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  2m 
14s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 6s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
10s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
10s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m 24s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
|

[jira] [Commented] (YARN-10801) Fix Auto Queue template to properly set all configuration properties

2021-06-09 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360135#comment-17360135
 ] 

Hadoop QA commented on YARN-10801:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
45s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 
 4s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m 18s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 23m  
5s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are 
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  2m 
10s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
59s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
56s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 125 unchanged - 1 fixed = 125 total (was 126) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 34s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
|

[jira] [Commented] (YARN-10657) We should make max application per queue to support node label.

2021-06-09 Thread Andras Gyori (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360113#comment-17360113
 ] 

Andras Gyori commented on YARN-10657:
-

Thank you [~zhuqi]! I will get back with approaches soon.

> We should make max application per queue to support node label.
> ---
>
> Key: YARN-10657
> URL: https://issues.apache.org/jira/browse/YARN-10657
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10657.001.patch, YARN-10657.002.patch
>
>
> https://issues.apache.org/jira/browse/YARN-10641?focusedCommentId=17291708=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291708
> As we discussed in above comment:
> We should deep into the label related max applications per queue.
> I think when node label enabled in queue, max applications should consider 
> the max capacity of all labels.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Reopened] (YARN-10657) We should make max application per queue to support node label.

2021-06-09 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori reopened YARN-10657:
-

> We should make max application per queue to support node label.
> ---
>
> Key: YARN-10657
> URL: https://issues.apache.org/jira/browse/YARN-10657
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10657.001.patch, YARN-10657.002.patch
>
>
> https://issues.apache.org/jira/browse/YARN-10641?focusedCommentId=17291708=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291708
> As we discussed in above comment:
> We should deep into the label related max applications per queue.
> I think when node label enabled in queue, max applications should consider 
> the max capacity of all labels.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-10657) We should make max application per queue to support node label.

2021-06-09 Thread Qi Zhu (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qi Zhu reassigned YARN-10657:
-

Assignee: Andras Gyori

> We should make max application per queue to support node label.
> ---
>
> Key: YARN-10657
> URL: https://issues.apache.org/jira/browse/YARN-10657
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10657.001.patch, YARN-10657.002.patch
>
>
> https://issues.apache.org/jira/browse/YARN-10641?focusedCommentId=17291708=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291708
> As we discussed in above comment:
> We should deep into the label related max applications per queue.
> I think when node label enabled in queue, max applications should consider 
> the max capacity of all labels.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10657) We should make max application per queue to support node label.

2021-06-09 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360109#comment-17360109
 ] 

Qi Zhu commented on YARN-10657:
---

[~gandras] Of course you can take it, and i will help review. :D

Assigned it to you.

> We should make max application per queue to support node label.
> ---
>
> Key: YARN-10657
> URL: https://issues.apache.org/jira/browse/YARN-10657
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10657.001.patch, YARN-10657.002.patch
>
>
> https://issues.apache.org/jira/browse/YARN-10641?focusedCommentId=17291708=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291708
> As we discussed in above comment:
> We should deep into the label related max applications per queue.
> I think when node label enabled in queue, max applications should consider 
> the max capacity of all labels.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-10657) We should make max application per queue to support node label.

2021-06-09 Thread Qi Zhu (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qi Zhu reassigned YARN-10657:
-

Assignee: (was: Qi Zhu)

> We should make max application per queue to support node label.
> ---
>
> Key: YARN-10657
> URL: https://issues.apache.org/jira/browse/YARN-10657
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Priority: Major
> Attachments: YARN-10657.001.patch, YARN-10657.002.patch
>
>
> https://issues.apache.org/jira/browse/YARN-10641?focusedCommentId=17291708=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291708
> As we discussed in above comment:
> We should deep into the label related max applications per queue.
> I think when node label enabled in queue, max applications should consider 
> the max capacity of all labels.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10801) Fix Auto Queue template to properly set all configuration properties

2021-06-09 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10801:

Attachment: YARN-10801.005.patch

> Fix Auto Queue template to properly set all configuration properties
> 
>
> Key: YARN-10801
> URL: https://issues.apache.org/jira/browse/YARN-10801
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10801.001.patch, YARN-10801.002.patch, 
> YARN-10801.003.patch, YARN-10801.004.patch, YARN-10801.005.patch
>
>
> Currently Auto Queue templates set configuration properties only on 
> Configuration object passed in the constructor. Due to the fact, that a lot 
> of configuration values are ready from the Configuration object in csContext, 
> template properties are not set in every cases. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10801) Fix Auto Queue template to properly set all configuration properties

2021-06-09 Thread Qi Zhu (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360017#comment-17360017
 ] 

Qi Zhu commented on YARN-10801:
---

Thanks [~gandras] for update.

The latest patch LGTM.

> Fix Auto Queue template to properly set all configuration properties
> 
>
> Key: YARN-10801
> URL: https://issues.apache.org/jira/browse/YARN-10801
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10801.001.patch, YARN-10801.002.patch, 
> YARN-10801.003.patch, YARN-10801.004.patch
>
>
> Currently Auto Queue templates set configuration properties only on 
> Configuration object passed in the constructor. Due to the fact, that a lot 
> of configuration values are ready from the Configuration object in csContext, 
> template properties are not set in every cases. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10780) Optimise retrieval of configured node labels in CS queues

2021-06-09 Thread Andras Gyori (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360012#comment-17360012
 ] 

Andras Gyori commented on YARN-10780:
-

Some of the logic is replicated in YARN-10801, therefore I have decided to wait 
until it is merged to avoid conflicts.

> Optimise retrieval of configured node labels in CS queues
> -
>
> Key: YARN-10780
> URL: https://issues.apache.org/jira/browse/YARN-10780
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10780.001.patch, YARN-10780.002.patch
>
>
> CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with 
> respect to queue numbers (its O(n*m), where n is the number of queues and m 
> is the number of properties set by each queue). During CS reinit, the node 
> labels are often queried, however looking at the code:
> {code:java}
> for (Entry stringStringEntry : this) {
>   e = stringStringEntry;
>   String key = e.getKey();
>   if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
>   + DOT)) {
> // Find  in
> // .accessible-node-labels..property
> int labelStartIdx =
> key.indexOf(ACCESSIBLE_NODE_LABELS)
> + ACCESSIBLE_NODE_LABELS.length() + 1;
> int labelEndIndx = key.indexOf('.', labelStartIdx);
> String labelName = key.substring(labelStartIdx, labelEndIndx);
> configuredNodeLabels.add(labelName);
>   }
> }
> {code}
>  This method iterates through ALL properties set in the configuration. For 
> example in case of initialising 2500 queues, each having at least 2 
> properties:
> 2500 * 5000 ~= over 12 million iteration + additional properties
> There are some ways to resolve this issue while keeping backward 
> compatibility:
>  # Create a property like the original accessible-node-labels, which contains 
> predefined labels. If it is set, then getConfiguredNodeLabels get the value 
> of this property, otherwise it falls back to the old logic. I think 
> accessible-node-labels are not used for this purpose (though I have a feeling 
> that it should have been).
>  # Collect node labels for all queues at the beginning of parseQueue and only 
> iterate through the properties once. This will increase the space complexity 
> in exchange of not requiring intervention from user's perspective. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10801) Fix Auto Queue template to properly set all configuration properties

2021-06-09 Thread Andras Gyori (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10801:

Attachment: YARN-10801.004.patch

> Fix Auto Queue template to properly set all configuration properties
> 
>
> Key: YARN-10801
> URL: https://issues.apache.org/jira/browse/YARN-10801
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10801.001.patch, YARN-10801.002.patch, 
> YARN-10801.003.patch, YARN-10801.004.patch
>
>
> Currently Auto Queue templates set configuration properties only on 
> Configuration object passed in the constructor. Due to the fact, that a lot 
> of configuration values are ready from the Configuration object in csContext, 
> template properties are not set in every cases. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10801) Fix Auto Queue template to properly set all configuration properties

2021-06-09 Thread Andras Gyori (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17360006#comment-17360006
 ] 

Andras Gyori commented on YARN-10801:
-

Thank you [~zhuqi] for the suggestion, I agree with this approach and 
incorporated the suggestion.

> Fix Auto Queue template to properly set all configuration properties
> 
>
> Key: YARN-10801
> URL: https://issues.apache.org/jira/browse/YARN-10801
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10801.001.patch, YARN-10801.002.patch, 
> YARN-10801.003.patch, YARN-10801.004.patch
>
>
> Currently Auto Queue templates set configuration properties only on 
> Configuration object passed in the constructor. Due to the fact, that a lot 
> of configuration values are ready from the Configuration object in csContext, 
> template properties are not set in every cases. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10813) Root queue capacity is not set when using node labels

2021-06-09 Thread Andras Gyori (Jira) 
Andras Gyori created YARN-10813:
---

 Summary: Root queue capacity is not set when using node labels
 Key: YARN-10813
 URL: https://issues.apache.org/jira/browse/YARN-10813
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Andras Gyori
Assignee: Andras Gyori


CapacitySchedulerConfiguration#getNonLabeledQueueCapacity handles root in the 
following way:
{code:java}
if (absoluteResourceConfigured || configuredWeightAsCapacity(
configuredCapacity)) {
  // Return capacity in percentage as 0 for non-root queues and 100 for
  // root.From AbstractCSQueue, absolute resource will be parsed and
  // updated. Once nodes are added/removed in cluster, capacity in
  // percentage will also be re-calculated.
  return queue.equals("root") ? 100.0f : 0f;
}
{code}
CapacitySchedulerConfiguration#internalGetLabeledQueueCapacity on the other 
hand does not take root queue into consideration:
{code:java}
if (absoluteResourceConfigured || configuredWeightAsCapacity(
configuredCapacity)) {
  // Return capacity in percentage as 0 for non-root queues and 100 for
  // root.From AbstractCSQueue, absolute resource, and weight will be parsed
  // and updated separately. Once nodes are added/removed in cluster,
  // capacity is percentage will also be re-calculated.
  return defaultValue;
}

float capacity = getFloat(capacityPropertyName, defaultValue);
{code}
Due to this, labeled root capacity is 0, which is not set in in 
AbstractCSQueue#derivedCapacityFromAbsoluteConfigurations, because root is 
never in Absolute mode.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10657) We should make max application per queue to support node label.

2021-06-09 Thread Andras Gyori (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17359995#comment-17359995
 ] 

Andras Gyori commented on YARN-10657:
-

[~zhuqi] Unfortunately I have encountered a problem that stems from this issue. 
May I take it over? I have some idea on how to proceed this, but current 
maxApplication logic is wrong in so many ways and we need to address it.

> We should make max application per queue to support node label.
> ---
>
> Key: YARN-10657
> URL: https://issues.apache.org/jira/browse/YARN-10657
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10657.001.patch, YARN-10657.002.patch
>
>
> https://issues.apache.org/jira/browse/YARN-10641?focusedCommentId=17291708=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291708
> As we discussed in above comment:
> We should deep into the label related max applications per queue.
> I think when node label enabled in queue, max applications should consider 
> the max capacity of all labels.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10812) yarn service number of containers count is wrong when flexing

2021-06-09 Thread Bilwa S T (Jira) 
Bilwa S T created YARN-10812:


 Summary: yarn service number of containers count is wrong when 
flexing
 Key: YARN-10812
 URL: https://issues.apache.org/jira/browse/YARN-10812
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Bilwa S T
Assignee: Bilwa S T


Currently let's say there are 2 containers running in a service.

User ask for 2 more by flexing it and there is just resources available only 
for 1 container to run but still number of containers will be updated to 4 
which is wrong.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Resolved] (YARN-10805) Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project

2021-06-09 Thread Takanobu Asanuma (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takanobu Asanuma resolved YARN-10805.
-
Fix Version/s: 3.4.0
   Resolution: Fixed

> Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project
> --
>
> Key: YARN-10805
> URL: https://issues.apache.org/jira/browse/YARN-10805
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org