[jira] [Updated] (YARN-10201) Make AMRMProxyPolicy aware of SC load
[ https://issues.apache.org/jira/browse/YARN-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-10201: -- Attachment: (was: YARN-5597.v0.patch) > Make AMRMProxyPolicy aware of SC load > - > > Key: YARN-10201 > URL: https://issues.apache.org/jira/browse/YARN-10201 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy >Reporter: Young Chen >Assignee: Young Chen >Priority: Major > Attachments: YARN-10201.v0.patch > > > LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when > splitting resource requests. We propose changes to the policy so that it > receives feedback from SCs and can load balance requests across the federated > cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10201) Make AMRMProxyPolicy aware of SC load
[ https://issues.apache.org/jira/browse/YARN-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-10201: -- Attachment: (was: YARN-5597.v0.patch) > Make AMRMProxyPolicy aware of SC load > - > > Key: YARN-10201 > URL: https://issues.apache.org/jira/browse/YARN-10201 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy >Reporter: Young Chen >Assignee: Young Chen >Priority: Major > Attachments: YARN-10201.v0.patch > > > LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when > splitting resource requests. We propose changes to the policy so that it > receives feedback from SCs and can load balance requests across the federated > cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10201) Make AMRMProxyPolicy aware of SC load
[ https://issues.apache.org/jira/browse/YARN-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-10201: -- Attachment: YARN-10201.v0.patch > Make AMRMProxyPolicy aware of SC load > - > > Key: YARN-10201 > URL: https://issues.apache.org/jira/browse/YARN-10201 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy >Reporter: Young Chen >Assignee: Young Chen >Priority: Major > Attachments: YARN-10201.v0.patch > > > LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when > splitting resource requests. We propose changes to the policy so that it > receives feedback from SCs and can load balance requests across the federated > cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10201) Make AMRMProxyPolicy aware of SC load
[ https://issues.apache.org/jira/browse/YARN-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-10201: -- Attachment: YARN-5597.v0.patch > Make AMRMProxyPolicy aware of SC load > - > > Key: YARN-10201 > URL: https://issues.apache.org/jira/browse/YARN-10201 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy >Reporter: Young Chen >Assignee: Young Chen >Priority: Major > Attachments: YARN-5597.v0.patch, YARN-5597.v0.patch > > > LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when > splitting resource requests. We propose changes to the policy so that it > receives feedback from SCs and can load balance requests across the federated > cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9940) avoid continuous scheduling thread crashes while sorting nodes get 'Comparison method violates its general contract'
[ https://issues.apache.org/jira/browse/YARN-9940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wilfred Spiegelenburg resolved YARN-9940. - Resolution: Not A Problem This issue is fixed in later versions via YARN-8373. In the version it is logged against it does not exist. The custom code that caused the issue to show up is a mix of Hadoop 2.7 and Hadoop 2.9. > avoid continuous scheduling thread crashes while sorting nodes get > 'Comparison method violates its general contract' > > > Key: YARN-9940 > URL: https://issues.apache.org/jira/browse/YARN-9940 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: kailiu_dev >Assignee: kailiu_dev >Priority: Major > Attachments: YARN-9940-branch-2.7.2.001.patch > > > 2019-10-16 09:14:51,215 ERROR > org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread > Thread[FairSchedulerContinuousScheduling,5,main] threw an Exception. > java.lang.IllegalArgumentException: Comparison method violates its general > contract! > at java.util.TimSort.mergeHi(TimSort.java:868) > at java.util.TimSort.mergeAt(TimSort.java:485) > at java.util.TimSort.mergeForceCollapse(TimSort.java:426) > at java.util.TimSort.sort(TimSort.java:223) > at java.util.TimSort.sort(TimSort.java:173) > at java.util.Arrays.sort(Arrays.java:659) > at java.util.Collections.sort(Collections.java:217) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:1117) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:296) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10200) Add number of containers to RMAppManager summary
[
https://issues.apache.org/jira/browse/YARN-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061318#comment-17061318
]
Hadoop QA commented on YARN-10200:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 30m
4s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 463 unchanged - 0 fixed = 464 total (was 463)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m
5s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10200 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12996965/YARN-10200.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux 5de150383a7d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8d63734 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/25702/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25702/testReport/ |
| Max
[jira] [Updated] (YARN-10201) Make AMRMProxyPolicy aware of SC load
[ https://issues.apache.org/jira/browse/YARN-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-10201: -- Attachment: YARN-5597.v0.patch > Make AMRMProxyPolicy aware of SC load > - > > Key: YARN-10201 > URL: https://issues.apache.org/jira/browse/YARN-10201 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy >Reporter: Young Chen >Assignee: Young Chen >Priority: Major > Attachments: YARN-5597.v0.patch > > > LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when > splitting resource requests. We propose changes to the policy so that it > receives feedback from SCs and can load balance requests across the federated > cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10201) Make AMRMProxyPolicy aware of SC load
[ https://issues.apache.org/jira/browse/YARN-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-10201: -- Description: LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when splitting resource requests. We propose changes to the policy so that it receives feedback from SCs and can load balance requests across the federated cluster. (was: LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when splitting resource requests.) > Make AMRMProxyPolicy aware of SC load > - > > Key: YARN-10201 > URL: https://issues.apache.org/jira/browse/YARN-10201 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy >Reporter: Young Chen >Assignee: Young Chen >Priority: Major > > LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when > splitting resource requests. We propose changes to the policy so that it > receives feedback from SCs and can load balance requests across the federated > cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10201) Make AMRMProxyPolicy aware of SC load
Young Chen created YARN-10201: - Summary: Make AMRMProxyPolicy aware of SC load Key: YARN-10201 URL: https://issues.apache.org/jira/browse/YARN-10201 Project: Hadoop YARN Issue Type: Sub-task Components: amrmproxy Reporter: Young Chen Assignee: Young Chen LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when splitting resource requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10200) Add number of containers to RMAppManager summary
[ https://issues.apache.org/jira/browse/YARN-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-10200: - Attachment: YARN-10200.001.patch > Add number of containers to RMAppManager summary > > > Key: YARN-10200 > URL: https://issues.apache.org/jira/browse/YARN-10200 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-10200.001.patch > > > It would be useful to persist this so we can track containers processed by RM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2710) RM HA tests failed intermittently on trunk
[
https://issues.apache.org/jira/browse/YARN-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-2710:
--
Fix Version/s: (was: 2.10.1)
Thanks for the update [~brahmareddy] and [~ahussein].
Since the branch-3.2 patch is not ready, I have reverted this from branch-2.10
as I don't think it is good practice to have a patch skip branches. Once the
branch-3.2/3.1 patches are ready, I will commit them as well as recommit the
2.10 patch
> RM HA tests failed intermittently on trunk
> --
>
> Key: YARN-2710
> URL: https://issues.apache.org/jira/browse/YARN-2710
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
> Environment: Java 8, jenkins
>Reporter: Wangda Tan
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: TestResourceTrackerOnHA-output.2.txt,
> YARN-2710-branch-2.10.001.patch, YARN-2710-branch-2.10.002.patch,
> YARN-2710-branch-2.10.003.patch, YARN-2710.001.patch, YARN-2710.002.patch,
> YARN-2710.003.patch,
> org.apache.hadoop.yarn.client.TestResourceTrackerOnHA-output.txt
>
>
> Failure like, it can be happened in TestApplicationClientProtocolOnHA,
> TestResourceTrackerOnHA, etc.
> {code}
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
> testGetApplicationAttemptsOnHA(org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA)
> Time elapsed: 9.491 sec <<< ERROR!
> java.net.ConnectException: Call From asf905.gq1.ygridcore.net/67.195.81.149
> to asf905.gq1.ygridcore.net:28032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
> at org.apache.hadoop.ipc.Client.call(Client.java:1438)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy17.getApplicationAttempts(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationAttempts(ApplicationClientProtocolPBClientImpl.java:372)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy18.getApplicationAttempts(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationAttempts(YarnClientImpl.java:583)
> at
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.testGetApplicationAttemptsOnHA(TestApplicationClientProtocolOnHA.java:137)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10200) Add number of containers to RMAppManager summary
[ https://issues.apache.org/jira/browse/YARN-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung reassigned YARN-10200: Assignee: Jonathan Hung > Add number of containers to RMAppManager summary > > > Key: YARN-10200 > URL: https://issues.apache.org/jira/browse/YARN-10200 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > > It would be useful to persist this so we can track containers processed by RM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10003) YarnConfigurationStore#checkVersion throws exception that belongs to RMStateStore
[
https://issues.apache.org/jira/browse/YARN-10003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061074#comment-17061074
]
Hadoop QA commented on YARN-10003:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 6m
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
51s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}303m 28s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}378m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector
|
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2 |
| |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:0f25cbbb251 |
| JIRA Issue | YARN-10003 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12996541/YARN-10003.branch-3.2.POC001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c7d7a0f41e7a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / e991605 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit
[jira] [Commented] (YARN-9879) Allow multiple leaf queues with the same name in CS
[
https://issues.apache.org/jira/browse/YARN-9879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061061#comment-17061061
]
Hadoop QA commented on YARN-9879:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 32m
57s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 32 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 10s{color} | {color:orange} root: The patch generated 2 new + 2367 unchanged
- 16 fixed = 2369 total (was 2383) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 49s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
2s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}239m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-9879 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12996923/YARN-9879.014.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bfe2905efa8e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8d63734 |
| maven | version: Apache Maven 3.3.9 |
| Default J
[jira] [Commented] (YARN-10200) Add number of containers to RMAppManager summary
[ https://issues.apache.org/jira/browse/YARN-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061059#comment-17061059 ] Jonathan Hung commented on YARN-10200: -- Yeah [~maniraj...@gmail.com] I think that makes sense. > Add number of containers to RMAppManager summary > > > Key: YARN-10200 > URL: https://issues.apache.org/jira/browse/YARN-10200 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Priority: Major > > It would be useful to persist this so we can track containers processed by RM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10200) Add number of containers to RMAppManager summary
[ https://issues.apache.org/jira/browse/YARN-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061053#comment-17061053 ] Manikandan R commented on YARN-10200: - Should we sum up all attempt's totalallocatedcontainers (from RMAppAttemptMetrics) and add it to this summary? > Add number of containers to RMAppManager summary > > > Key: YARN-10200 > URL: https://issues.apache.org/jira/browse/YARN-10200 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Priority: Major > > It would be useful to persist this so we can track containers processed by RM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2710) RM HA tests failed intermittently on trunk
[
https://issues.apache.org/jira/browse/YARN-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated YARN-2710:
---
Fix Version/s: (was: 3.4.0)
3.3.0
> RM HA tests failed intermittently on trunk
> --
>
> Key: YARN-2710
> URL: https://issues.apache.org/jira/browse/YARN-2710
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
> Environment: Java 8, jenkins
>Reporter: Wangda Tan
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.0, 2.10.1
>
> Attachments: TestResourceTrackerOnHA-output.2.txt,
> YARN-2710-branch-2.10.001.patch, YARN-2710-branch-2.10.002.patch,
> YARN-2710-branch-2.10.003.patch, YARN-2710.001.patch, YARN-2710.002.patch,
> YARN-2710.003.patch,
> org.apache.hadoop.yarn.client.TestResourceTrackerOnHA-output.txt
>
>
> Failure like, it can be happened in TestApplicationClientProtocolOnHA,
> TestResourceTrackerOnHA, etc.
> {code}
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
> testGetApplicationAttemptsOnHA(org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA)
> Time elapsed: 9.491 sec <<< ERROR!
> java.net.ConnectException: Call From asf905.gq1.ygridcore.net/67.195.81.149
> to asf905.gq1.ygridcore.net:28032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
> at org.apache.hadoop.ipc.Client.call(Client.java:1438)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy17.getApplicationAttempts(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationAttempts(ApplicationClientProtocolPBClientImpl.java:372)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy18.getApplicationAttempts(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationAttempts(YarnClientImpl.java:583)
> at
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.testGetApplicationAttemptsOnHA(TestApplicationClientProtocolOnHA.java:137)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2710) RM HA tests failed intermittently on trunk
[
https://issues.apache.org/jira/browse/YARN-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061049#comment-17061049
]
Brahma Reddy Battula commented on YARN-2710:
{quote}I also don't see a branch-3.3, but 3.4 is the latest fix version
available. [~brahma], can you advise on this?
{quote}
As of now I didn't cut the branch for 3.3, will cut and announce in mailing
list.So I mark this Jira fix version as 3.3.0
> RM HA tests failed intermittently on trunk
> --
>
> Key: YARN-2710
> URL: https://issues.apache.org/jira/browse/YARN-2710
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
> Environment: Java 8, jenkins
>Reporter: Wangda Tan
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 2.10.1, 3.4.0
>
> Attachments: TestResourceTrackerOnHA-output.2.txt,
> YARN-2710-branch-2.10.001.patch, YARN-2710-branch-2.10.002.patch,
> YARN-2710-branch-2.10.003.patch, YARN-2710.001.patch, YARN-2710.002.patch,
> YARN-2710.003.patch,
> org.apache.hadoop.yarn.client.TestResourceTrackerOnHA-output.txt
>
>
> Failure like, it can be happened in TestApplicationClientProtocolOnHA,
> TestResourceTrackerOnHA, etc.
> {code}
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
> testGetApplicationAttemptsOnHA(org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA)
> Time elapsed: 9.491 sec <<< ERROR!
> java.net.ConnectException: Call From asf905.gq1.ygridcore.net/67.195.81.149
> to asf905.gq1.ygridcore.net:28032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
> at org.apache.hadoop.ipc.Client.call(Client.java:1438)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy17.getApplicationAttempts(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationAttempts(ApplicationClientProtocolPBClientImpl.java:372)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy18.getApplicationAttempts(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationAttempts(YarnClientImpl.java:583)
> at
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.testGetApplicationAttemptsOnHA(TestApplicationClientProtocolOnHA.java:137)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2710) RM HA tests failed intermittently on trunk
[
https://issues.apache.org/jira/browse/YARN-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061037#comment-17061037
]
Ahmed Hussein commented on YARN-2710:
-
Thanks [~ebadger] for committing the patch to trunk and 2.10
{quote}[~ahussein], could you put up a patch for branch-3.2?
{quote}
branch-3.2 uses a different version of JUnit which does not accept TimeUnit as
a parameter in the constructor of Timeout.
After I fixed the compilation error, the fix does not work for branch-3.2. I
will need to further investigate that branch.
> RM HA tests failed intermittently on trunk
> --
>
> Key: YARN-2710
> URL: https://issues.apache.org/jira/browse/YARN-2710
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
> Environment: Java 8, jenkins
>Reporter: Wangda Tan
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 2.10.1, 3.4.0
>
> Attachments: TestResourceTrackerOnHA-output.2.txt,
> YARN-2710-branch-2.10.001.patch, YARN-2710-branch-2.10.002.patch,
> YARN-2710-branch-2.10.003.patch, YARN-2710.001.patch, YARN-2710.002.patch,
> YARN-2710.003.patch,
> org.apache.hadoop.yarn.client.TestResourceTrackerOnHA-output.txt
>
>
> Failure like, it can be happened in TestApplicationClientProtocolOnHA,
> TestResourceTrackerOnHA, etc.
> {code}
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
> testGetApplicationAttemptsOnHA(org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA)
> Time elapsed: 9.491 sec <<< ERROR!
> java.net.ConnectException: Call From asf905.gq1.ygridcore.net/67.195.81.149
> to asf905.gq1.ygridcore.net:28032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
> at org.apache.hadoop.ipc.Client.call(Client.java:1438)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy17.getApplicationAttempts(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationAttempts(ApplicationClientProtocolPBClientImpl.java:372)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
> at com.sun.proxy.$Proxy18.getApplicationAttempts(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationAttempts(YarnClientImpl.java:583)
> at
> org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA.testGetApplicationAttemptsOnHA(TestApplicationClientProtocolOnHA.java:137)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061033#comment-17061033
]
Manikandan R commented on YARN-10198:
-
[~pbacsko] Thanks for the patch.
1. Are we planning to have this check only for regular parent queue in separate
jira?
2. Are we going to handle [managedParentQueue].%secondary_group in separate
jira?
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser
[ https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori updated YARN-10199: Attachment: YARN-10199.002.patch > Simplify UserGroupMappingPlacementRule#getPlacementForUser > -- > > Key: YARN-10199 > URL: https://issues.apache.org/jira/browse/YARN-10199 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Attachments: YARN-10199.001.patch, YARN-10199.002.patch > > > The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly > responsible for queue naming, contains deeply nested branches. In order to > provide an extendable mapping logic, the branches could be flattened and > simplified. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser
[ https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061032#comment-17061032 ] Andras Gyori commented on YARN-10199: - Thank you for the suggestion [~pbacsko]. It is indeed a very good idea and would facilitate the testing of queue mapping. I have acted accordingly and the logic, with the involved methods have been extracted to their own class. > Simplify UserGroupMappingPlacementRule#getPlacementForUser > -- > > Key: YARN-10199 > URL: https://issues.apache.org/jira/browse/YARN-10199 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Attachments: YARN-10199.001.patch > > > The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly > responsible for queue naming, contains deeply nested branches. In order to > provide an extendable mapping logic, the branches could be flattened and > simplified. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060958#comment-17060958
]
Peter Bacsko commented on YARN-10198:
-
Test failure is unrelated.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060942#comment-17060942
]
Hadoop QA commented on YARN-10198:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 0s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerOvercommit |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10198 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12996906/YARN-10198-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5b10f876dd8c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1975479 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25700/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25700/testReport/ |
| Max. process+thread count | 824 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-pr
[jira] [Comment Edited] (YARN-9879) Allow multiple leaf queues with the same name in CS
[ https://issues.apache.org/jira/browse/YARN-9879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060899#comment-17060899 ] Gergely Pollak edited comment on YARN-9879 at 3/17/20, 1:09 PM: In the latest patch, there are mostly cosmetic changes. I've removed most of the TODOs as [~wangda] and [~snemeth] suggested, will create follow up Jira based on those. Fixed additional checkstyle issues, and added a proper error message when a queue is ambiguously referenced during submission, so the user will now get feedback, that the queue exists, but actually multiple queues exist with the same name. Also dropped the POC prefix, since it's getting near completion, so the latest patch is YARN-9879.014.patch was (Author: shuzirra): In the latest patch, there are mostly cosmetic changes. I've removed most of the TODOs as [~wangda] and [~snemeth] suggested, will create follow up Jira based on those. Fixed additional checkstyle issues, and added a proper error message when a queue is ambiguously referenced during submission, so the user will now get feedback, that the queue exists, but actually multiple queues exist with the same name. > Allow multiple leaf queues with the same name in CS > --- > > Key: YARN-9879 > URL: https://issues.apache.org/jira/browse/YARN-9879 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Labels: fs2cs > Attachments: CSQueue.getQueueUsage.txt, DesignDoc_v1.pdf, > YARN-9879.014.patch, YARN-9879.POC001.patch, YARN-9879.POC002.patch, > YARN-9879.POC003.patch, YARN-9879.POC004.patch, YARN-9879.POC005.patch, > YARN-9879.POC006.patch, YARN-9879.POC007.patch, YARN-9879.POC008.patch, > YARN-9879.POC009.patch, YARN-9879.POC010.patch, YARN-9879.POC011.patch, > YARN-9879.POC012.patch, YARN-9879.POC013.patch > > > Currently the leaf queue's name must be unique regardless of its position in > the queue hierarchy. > Design doc and first proposal is being made, I'll attach it as soon as it's > done. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9879) Allow multiple leaf queues with the same name in CS
[ https://issues.apache.org/jira/browse/YARN-9879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060899#comment-17060899 ] Gergely Pollak commented on YARN-9879: -- In the latest patch, there are mostly cosmetic changes. I've removed most of the TODOs as [~wangda] and [~snemeth] suggested, will create follow up Jira based on those. Fixed additional checkstyle issues, and added a proper error message when a queue is ambiguously referenced during submission, so the user will now get feedback, that the queue exists, but actually multiple queues exist with the same name. > Allow multiple leaf queues with the same name in CS > --- > > Key: YARN-9879 > URL: https://issues.apache.org/jira/browse/YARN-9879 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Labels: fs2cs > Attachments: CSQueue.getQueueUsage.txt, DesignDoc_v1.pdf, > YARN-9879.014.patch, YARN-9879.POC001.patch, YARN-9879.POC002.patch, > YARN-9879.POC003.patch, YARN-9879.POC004.patch, YARN-9879.POC005.patch, > YARN-9879.POC006.patch, YARN-9879.POC007.patch, YARN-9879.POC008.patch, > YARN-9879.POC009.patch, YARN-9879.POC010.patch, YARN-9879.POC011.patch, > YARN-9879.POC012.patch, YARN-9879.POC013.patch > > > Currently the leaf queue's name must be unique regardless of its position in > the queue hierarchy. > Design doc and first proposal is being made, I'll attach it as soon as it's > done. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9879) Allow multiple leaf queues with the same name in CS
[ https://issues.apache.org/jira/browse/YARN-9879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-9879: - Attachment: YARN-9879.014.patch > Allow multiple leaf queues with the same name in CS > --- > > Key: YARN-9879 > URL: https://issues.apache.org/jira/browse/YARN-9879 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Labels: fs2cs > Attachments: CSQueue.getQueueUsage.txt, DesignDoc_v1.pdf, > YARN-9879.014.patch, YARN-9879.POC001.patch, YARN-9879.POC002.patch, > YARN-9879.POC003.patch, YARN-9879.POC004.patch, YARN-9879.POC005.patch, > YARN-9879.POC006.patch, YARN-9879.POC007.patch, YARN-9879.POC008.patch, > YARN-9879.POC009.patch, YARN-9879.POC010.patch, YARN-9879.POC011.patch, > YARN-9879.POC012.patch, YARN-9879.POC013.patch > > > Currently the leaf queue's name must be unique regardless of its position in > the queue hierarchy. > Design doc and first proposal is being made, I'll attach it as soon as it's > done. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser
[
https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060876#comment-17060876
]
Peter Bacsko edited comment on YARN-10199 at 3/17/20, 12:26 PM:
[~gandras] I'm wondering whether it makes sense to create a completely new
class for this like {{CurrentUserMappingFactory}}. Then the whole logic becomes
more testable and we can also expand test coverage.
was (Author: pbacsko):
[~gandras] I'm wondering whether it makes sense to create a completely new
class for this. Then the whole logic becomes more testable.
> Simplify UserGroupMappingPlacementRule#getPlacementForUser
> --
>
> Key: YARN-10199
> URL: https://issues.apache.org/jira/browse/YARN-10199
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Minor
> Attachments: YARN-10199.001.patch
>
>
> The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly
> responsible for queue naming, contains deeply nested branches. In order to
> provide an extendable mapping logic, the branches could be flattened and
> simplified.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser
[ https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060876#comment-17060876 ] Peter Bacsko commented on YARN-10199: - [~gandras] I'm wondering whether it makes sense to create a completely new class for this. Then the whole logic becomes more testable. > Simplify UserGroupMappingPlacementRule#getPlacementForUser > -- > > Key: YARN-10199 > URL: https://issues.apache.org/jira/browse/YARN-10199 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Attachments: YARN-10199.001.patch > > > The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly > responsible for queue naming, contains deeply nested branches. In order to > provide an extendable mapping logic, the branches could be flattened and > simplified. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10197) FS-CS converter: fix emitted ordering policy string and max-am-resource percent value
[
https://issues.apache.org/jira/browse/YARN-10197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060874#comment-17060874
]
Peter Bacsko commented on YARN-10197:
-
[~prabhujoseph] could you review & commit this patch?
> FS-CS converter: fix emitted ordering policy string and max-am-resource
> percent value
> -
>
> Key: YARN-10197
> URL: https://issues.apache.org/jira/browse/YARN-10197
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10197-001.patch, YARN-10197-002.patch,
> YARN-10197-003.patch
>
>
> There are three problems that have to be addressed in the converter:
> 1) For {{yarn.scheduler.capacity..ordering-policy}}, the emitted value
> should be "fifo", not "FIFO". The reason we generate "FIFO" is because
> {{FifoPolicy.NAME}} consists of uppercase letters.
> 2) {{maximum-am-resource-percent}} calculation is faulty too. For example, in
> FS, you can globally disable it with
> {{-1.0}}. However this case
> is not handled properly in
> {{FSConfigToCSConfigConverter.emitDefaultMaxAMShare()}}. -1.0 means that max
> AM check is disabled, therefore we have to generate the value "1.0" to allow
> as many AMs as possible. In {{FSQueueConverter.emitMaxAMShare()}}, we should
> also check if the current value differs from the global setting and only then
> output "1.0" for a given queue.
> 3) The multi-leaf queue check is no longer necessary and it doesn't work
> anyway. The CS instance catches this kind of error during the verification
> step.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10181) Managing Centralized Node Attribute via RMWebServices.
[
https://issues.apache.org/jira/browse/YARN-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060866#comment-17060866
]
Prabhu Joseph commented on YARN-10181:
--
Yes Okay, Thanks.
> Managing Centralized Node Attribute via RMWebServices.
> --
>
> Key: YARN-10181
> URL: https://issues.apache.org/jira/browse/YARN-10181
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodeattibute
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Priority: Major
>
> Currently Centralized NodeAttributes can be managed only through Yarn
> NodeAttribute CLI. This is to support via RMWebServices.
> {code}
> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeAttributes.html#Centralised_Node_Attributes_mapping.
> Centralised : Node to attributes mapping can be done through RM exposed CLI
> or RPC (REST is yet to be supported).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10181) Managing Centralized Node Attribute via RMWebServices.
[
https://issues.apache.org/jira/browse/YARN-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-10181:
-
Parent: YARN-8766
Issue Type: Sub-task (was: Improvement)
> Managing Centralized Node Attribute via RMWebServices.
> --
>
> Key: YARN-10181
> URL: https://issues.apache.org/jira/browse/YARN-10181
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodeattibute
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Priority: Major
>
> Currently Centralized NodeAttributes can be managed only through Yarn
> NodeAttribute CLI. This is to support via RMWebServices.
> {code}
> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeAttributes.html#Centralised_Node_Attributes_mapping.
> Centralised : Node to attributes mapping can be done through RM exposed CLI
> or RPC (REST is yet to be supported).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10199) Simplify UserGroupMappingPlacementRule#getPlacementForUser
[ https://issues.apache.org/jira/browse/YARN-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori updated YARN-10199: Attachment: YARN-10199.001.patch > Simplify UserGroupMappingPlacementRule#getPlacementForUser > -- > > Key: YARN-10199 > URL: https://issues.apache.org/jira/browse/YARN-10199 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Attachments: YARN-10199.001.patch > > > The UserGroupMappingPlacementRule#getPlacementForUser method, which is mainly > responsible for queue naming, contains deeply nested branches. In order to > provide an extendable mapping logic, the branches could be flattened and > simplified. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060857#comment-17060857
]
Peter Bacsko commented on YARN-10198:
-
Thanks [~akhilpb].
[~prabhujoseph] could you please review patch 001?
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10181) Managing Centralized Node Attribute via RMWebServices.
[
https://issues.apache.org/jira/browse/YARN-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060849#comment-17060849
]
Bibin Chundatt commented on YARN-10181:
---
Could you move this jira as part of YARN-8766 ?
> Managing Centralized Node Attribute via RMWebServices.
> --
>
> Key: YARN-10181
> URL: https://issues.apache.org/jira/browse/YARN-10181
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodeattibute
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Priority: Major
>
> Currently Centralized NodeAttributes can be managed only through Yarn
> NodeAttribute CLI. This is to support via RMWebServices.
> {code}
> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeAttributes.html#Centralised_Node_Attributes_mapping.
> Centralised : Node to attributes mapping can be done through RM exposed CLI
> or RPC (REST is yet to be supported).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10198:
Attachment: YARN-10198-001.patch
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10166) Add detail log for ApplicationAttemptNotFoundException
[
https://issues.apache.org/jira/browse/YARN-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060770#comment-17060770
]
Adam Antal commented on YARN-10166:
---
+1 (non-binding). [~sunilg] can you take a look at this?
> Add detail log for ApplicationAttemptNotFoundException
> --
>
> Key: YARN-10166
> URL: https://issues.apache.org/jira/browse/YARN-10166
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Youquan Lin
>Priority: Minor
> Labels: patch
> Attachments: YARN-10166-001.patch, YARN-10166-002.patch,
> YARN-10166-003.patch, YARN-10166-004.patch
>
>
> Suppose user A killed the app, then ApplicationMasterService will call
> unregisterAttempt() for this app. Sometimes, app's AM continues to call the
> alloate() method and reports an error as follows.
> {code:java}
> Application attempt appattempt_1582520281010_15271_01 doesn't exist in
> ApplicationMasterService cache.
> {code}
> If user B has been watching the AM log, he will be confused why the
> attempt is no longer in the ApplicationMasterService cache. So I think we can
> add detail log for ApplicationAttemptNotFoundException as follows.
> {code:java}
> Application attempt appattempt_1582630210671_14658_01 doesn't exist in
> ApplicationMasterService cache.App state: KILLED,finalStatus: KILLED
> ,diagnostics: App application_1582630210671_14658 killed by userA from
> 127.0.0.1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10179) Queue mapping based on group id passed through application tag
[ https://issues.apache.org/jira/browse/YARN-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori reassigned YARN-10179: --- Assignee: Andras Gyori (was: Szilard Nemeth) > Queue mapping based on group id passed through application tag > -- > > Key: YARN-10179 > URL: https://issues.apache.org/jira/browse/YARN-10179 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Szilard Nemeth >Assignee: Andras Gyori >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10179.001.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on the user's group. > Unfortunately in this case YARN doesn't have any information about the real > user and there are cases when the customer may want to map these applications > to the real submitting user's queue (based on the group id) instead of the > Hive queue. > For these cases, if they would pass the group id (or name) in the application > tag we may read it and use it during the queue mapping, if that user has > rights to run on the real user's queue. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10179) Queue mapping based on group id passed through application tag
[ https://issues.apache.org/jira/browse/YARN-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060730#comment-17060730 ] Andras Gyori commented on YARN-10179: - Thank you, [~snemeth] for the references. I have made an early version of the proposed changes, however there might be further questions about the requirements. As for now, the default behavior in case of an invalid tag/non-whitelisted tag is to fall back to the original source of mapping. Moreover, this approach only works if the user is a member of the group. Might be interesting for [~lkovari], [~wilfreds] > Queue mapping based on group id passed through application tag > -- > > Key: YARN-10179 > URL: https://issues.apache.org/jira/browse/YARN-10179 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10179.001.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on the user's group. > Unfortunately in this case YARN doesn't have any information about the real > user and there are cases when the customer may want to map these applications > to the real submitting user's queue (based on the group id) instead of the > Hive queue. > For these cases, if they would pass the group id (or name) in the application > tag we may read it and use it during the queue mapping, if that user has > rights to run on the real user's queue. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10179) Queue mapping based on group id passed through application tag
[ https://issues.apache.org/jira/browse/YARN-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori updated YARN-10179: Attachment: YARN-10179.001.patch > Queue mapping based on group id passed through application tag > -- > > Key: YARN-10179 > URL: https://issues.apache.org/jira/browse/YARN-10179 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10179.001.patch > > > There are situations when the real submitting user differs from the user what > arrives to YARN. For example in case of a Hive application when Hive > impersonation is turned off, the hive queries will run as Hive user and the > mapping is done based on the user's group. > Unfortunately in this case YARN doesn't have any information about the real > user and there are cases when the customer may want to map these applications > to the real submitting user's queue (based on the group id) instead of the > Hive queue. > For these cases, if they would pass the group id (or name) in the application > tag we may read it and use it during the queue mapping, if that user has > rights to run on the real user's queue. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060683#comment-17060683
]
Akhil PB edited comment on YARN-10198 at 3/17/20, 7:40 AM:
---
Expected behaviour:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
** If the user does not have *%secondary_group*, then the app should be
submitted to the "default" queue (I guess based on the below static mapping).
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group* is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
was (Author: akhilpb):
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
** If the user does not have *%secondary_group*, then the app should be
submitted to the "default" queue (I guess based on the below).
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group* is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060683#comment-17060683
]
Akhil PB edited comment on YARN-10198 at 3/17/20, 7:34 AM:
---
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
** If the user does not have *%secondary_group*, then the app should be
submitted to the "default" queue (I guess based on the below).
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group* is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
was (Author: akhilpb):
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group* is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060683#comment-17060683
]
Akhil PB edited comment on YARN-10198 at 3/17/20, 7:30 AM:
---
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group* is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
was (Author: akhilpb):
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group* is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060683#comment-17060683
]
Akhil PB edited comment on YARN-10198 at 3/17/20, 7:30 AM:
---
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group* is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
was (Author: akhilpb):
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group* is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060683#comment-17060683
]
Akhil PB edited comment on YARN-10198 at 3/17/20, 7:29 AM:
---
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group*( is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
was (Author: akhilpb):
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue name *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue name *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group*( is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060683#comment-17060683
]
Akhil PB edited comment on YARN-10198 at 3/17/20, 7:29 AM:
---
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group* is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
was (Author: akhilpb):
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue named *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue named *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group*( is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060683#comment-17060683
]
Akhil PB commented on YARN-10198:
-
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue name *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue name *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group*( is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
