[jira] [Commented] (YARN-9819) Make TestOpportunisticContainerAllocatorAMService more resilient.
[ https://issues.apache.org/jira/browse/YARN-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928148#comment-16928148 ] Hudson commented on YARN-9819: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17280 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17280/]) YARN-9819. Make TestOpportunisticContainerAllocatorAMService more (abmodi: rev 3b06f0bf9e4c3d7bc50e5e2f9b44c1eeec897680) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java > Make TestOpportunisticContainerAllocatorAMService more resilient. > - > > Key: YARN-9819 > URL: https://issues.apache.org/jira/browse/YARN-9819 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9819.001.patch, YARN-9819.002.patch, > YARN-9819.003.patch > > > Currently, TestOpportunisticContainerAllocatorAMService tries to set the > Opportunistic container status directly in RMNode but that can be updated by > NM heartbeat. Correct way would be to send it through NM heartbeat. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9819) Make TestOpportunisticContainerAllocatorAMService more resilient.
[ https://issues.apache.org/jira/browse/YARN-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928147#comment-16928147 ] Abhishek Modi commented on YARN-9819: - Thanks [~elgoiri] for review. Committed to trunk. > Make TestOpportunisticContainerAllocatorAMService more resilient. > - > > Key: YARN-9819 > URL: https://issues.apache.org/jira/browse/YARN-9819 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9819.001.patch, YARN-9819.002.patch, > YARN-9819.003.patch > > > Currently, TestOpportunisticContainerAllocatorAMService tries to set the > Opportunistic container status directly in RMNode but that can be updated by > NM heartbeat. Correct way would be to send it through NM heartbeat. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9762) Add submission context label to audit logs
[
https://issues.apache.org/jira/browse/YARN-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928115#comment-16928115
]
Hadoop QA commented on YARN-9762:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 63 unchanged - 1 fixed = 65 total (was 64) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m
3s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9762 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12980047/YARN-9762.01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a8de569f234d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 56b7571 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24789/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24789/testReport/ |
| Max. process+thread count | 831 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-9825) Changes for initializing placement rules with ResourceScheduler in branch-2
[ https://issues.apache.org/jira/browse/YARN-9825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928059#comment-16928059 ] Jonathan Hung commented on YARN-9825: - I'm not really inclined to fix the checkstyle issues since this code was taken from branch-3.x. > Changes for initializing placement rules with ResourceScheduler in branch-2 > --- > > Key: YARN-9825 > URL: https://issues.apache.org/jira/browse/YARN-9825 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Labels: release-blocker > Attachments: YARN-9825-branch-2.001.patch > > > YARN-8016 and YARN-8948 add functionality to initialize placement rules with > ResourceScheduler. We need this in branch-2, but it doesn't apply cleanly. > Hence we just port the initialization logic. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9825) Changes for initializing placement rules with ResourceScheduler in branch-2
[
https://issues.apache.org/jira/browse/YARN-9825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928045#comment-16928045
]
Hadoop QA commented on YARN-9825:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 84 unchanged - 0 fixed = 89 total (was 84) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 59m
21s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:da675796017 |
| JIRA Issue | YARN-9825 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12980123/YARN-9825-branch-2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8e787883acfe 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-9815) ReservationACLsTestBase fails with NPE
[
https://issues.apache.org/jira/browse/YARN-9815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928040#comment-16928040
]
Hudson commented on YARN-9815:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17278 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17278/])
YARN-9815 ReservationACLsTestBase fails with NPE. Contributed by Ahmed
(ebadger: rev 56b7571131b0af03b32bf1c5673c32634652df21)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ReservationsACLsManager.java
> ReservationACLsTestBase fails with NPE
> --
>
> Key: YARN-9815
> URL: https://issues.apache.org/jira/browse/YARN-9815
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9805.001.patch, YARN-9815.001.patch,
> YARN-9815.002.patch
>
>
> Running ReservationACLsTestBase throws a NPE running the FairScheduler. Old
> revisions back in 2016 also throw NPE.
> In the test case, QueueC does not have reserveACLs, so
> ReservationsACLsManager would throw NPE when it tries to access the ACL on
> line 82.
> I still could not find what was the first revision that caused this test case
> to fail. I stopped at bbfaf3c2712c9ba82b0f8423bdeb314bf505a692 which was
> working fine.
> I have OsX with java 1.8.0_201
>
> {code:java}
> [ERROR]
> testApplicationACLs[1](org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase)
> Time elapsed: 1.897 s <<< ERROR![ERROR]
> testApplicationACLs[1](org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase)
> Time elapsed: 1.897 s <<<
> ERROR!java.lang.NullPointerException:java.lang.NullPointerException at
> org.apache.hadoop.yarn.server.resourcemanager.security.ReservationsACLsManager.checkAccess(ReservationsACLsManager.java:83)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.checkReservationACLs(ClientRMService.java:1527)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1290)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:511)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:645)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:529)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1001) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:929) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2921)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitReservation(ApplicationClientProtocolPBClientImpl.java:511)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase.submitReservation(ReservationACLsTestBase.java:447)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase.verifySubmitReservationSuccess(ReservationACLsTestBase.java:247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase.testApplicationACLs(ReservationACLsTestBase.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
>
[jira] [Updated] (YARN-9815) ReservationACLsTestBase fails with NPE
[
https://issues.apache.org/jira/browse/YARN-9815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-9815:
--
Fix Version/s: 3.3.0
> ReservationACLsTestBase fails with NPE
> --
>
> Key: YARN-9815
> URL: https://issues.apache.org/jira/browse/YARN-9815
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9805.001.patch, YARN-9815.001.patch,
> YARN-9815.002.patch
>
>
> Running ReservationACLsTestBase throws a NPE running the FairScheduler. Old
> revisions back in 2016 also throw NPE.
> In the test case, QueueC does not have reserveACLs, so
> ReservationsACLsManager would throw NPE when it tries to access the ACL on
> line 82.
> I still could not find what was the first revision that caused this test case
> to fail. I stopped at bbfaf3c2712c9ba82b0f8423bdeb314bf505a692 which was
> working fine.
> I have OsX with java 1.8.0_201
>
> {code:java}
> [ERROR]
> testApplicationACLs[1](org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase)
> Time elapsed: 1.897 s <<< ERROR![ERROR]
> testApplicationACLs[1](org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase)
> Time elapsed: 1.897 s <<<
> ERROR!java.lang.NullPointerException:java.lang.NullPointerException at
> org.apache.hadoop.yarn.server.resourcemanager.security.ReservationsACLsManager.checkAccess(ReservationsACLsManager.java:83)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.checkReservationACLs(ClientRMService.java:1527)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1290)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:511)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:645)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:529)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1001) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:929) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2921)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitReservation(ApplicationClientProtocolPBClientImpl.java:511)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase.submitReservation(ReservationACLsTestBase.java:447)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase.verifySubmitReservationSuccess(ReservationACLsTestBase.java:247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase.testApplicationACLs(ReservationACLsTestBase.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
>
[jira] [Commented] (YARN-9825) Changes for initializing placement rules with ResourceScheduler in branch-2
[ https://issues.apache.org/jira/browse/YARN-9825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927970#comment-16927970 ] Jonathan Hung commented on YARN-9825: - Attached [^YARN-9825-branch-2.001.patch] which contains PlacementRule initialization changes from YARN-8016 and YARN-8948. > Changes for initializing placement rules with ResourceScheduler in branch-2 > --- > > Key: YARN-9825 > URL: https://issues.apache.org/jira/browse/YARN-9825 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Labels: release-blocker > Attachments: YARN-9825-branch-2.001.patch > > > YARN-8016 and YARN-8948 add functionality to initialize placement rules with > ResourceScheduler. We need this in branch-2, but it doesn't apply cleanly. > Hence we just port the initialization logic. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9825) Changes for initializing placement rules with ResourceScheduler in branch-2
[ https://issues.apache.org/jira/browse/YARN-9825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9825: Attachment: YARN-9825-branch-2.001.patch > Changes for initializing placement rules with ResourceScheduler in branch-2 > --- > > Key: YARN-9825 > URL: https://issues.apache.org/jira/browse/YARN-9825 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Labels: release-blocker > Attachments: YARN-9825-branch-2.001.patch > > > YARN-8016 and YARN-8948 add functionality to initialize placement rules with > ResourceScheduler. We need this in branch-2, but it doesn't apply cleanly. > Hence we just port the initialization logic. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8972) [Router] Add support to prevent DoS attack over ApplicationSubmissionContext size
[
https://issues.apache.org/jira/browse/YARN-8972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927891#comment-16927891
]
Hadoop QA commented on YARN-8972:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
48s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-8972 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947619/YARN-8972.v5.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux dca440397bf7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9221704 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24787/testReport/ |
| Max. process+thread count | 731 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (YARN-9808) Zero length files in container log output haven't got a header
[
https://issues.apache.org/jira/browse/YARN-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927861#comment-16927861
]
Hadoop QA commented on YARN-9808:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 12s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 7 new + 306 unchanged - 1 fixed = 313 total (was 307) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
43s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
11s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 26m
15s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9808 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12980103/YARN-9808.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6392d0bb1bf2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5a381f7 |
|
[jira] [Commented] (YARN-8972) [Router] Add support to prevent DoS attack over ApplicationSubmissionContext size
[ https://issues.apache.org/jira/browse/YARN-8972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927813#comment-16927813 ] Abhishek Modi commented on YARN-8972: - [~giovanni.fumarola] are you still working on it. Thanks. > [Router] Add support to prevent DoS attack over ApplicationSubmissionContext > size > - > > Key: YARN-8972 > URL: https://issues.apache.org/jira/browse/YARN-8972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: YARN-8972.v1.patch, YARN-8972.v2.patch, > YARN-8972.v3.patch, YARN-8972.v4.patch, YARN-8972.v5.patch > > > This jira tracks the effort to add a new interceptor in the Router to prevent > user to submit applications with oversized ASC. > This avoid YARN cluster to failover. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9827) Fix Http Response code in GenericExceptionHandler.
Abhishek Modi created YARN-9827: --- Summary: Fix Http Response code in GenericExceptionHandler. Key: YARN-9827 URL: https://issues.apache.org/jira/browse/YARN-9827 Project: Hadoop YARN Issue Type: Bug Reporter: Abhishek Modi Assignee: Abhishek Modi GenericExceptionHandler should respond with SERVICE_UNAVAILABLE in case of connection and service unavailable exception instead of INTERNAL_SERVICE_ERROR. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9808) Zero length files in container log output haven't got a header
[
https://issues.apache.org/jira/browse/YARN-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Antal updated YARN-9808:
-
Attachment: YARN-9808.001.patch
> Zero length files in container log output haven't got a header
> --
>
> Key: YARN-9808
> URL: https://issues.apache.org/jira/browse/YARN-9808
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: log-aggregation, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: YARN-9808.001.patch
>
>
> Using the Yarn logs CLI for containers that have zero length files produces
> output similar to this:
> {noformat}
> End of LogType:stderr
> ***
> End of LogType:prelaunch.err
> **
> Container: container_e25_1567431105510_0001_01_02 on host-1
> LogAggregationType: AGGREGATED
> ===
> LogType:container.log
> LogLastModifiedTime:Mon Sep 02 06:34:48 -0700 2019
> LogLength:5442
> LogContents:
> ...
> ...
> {noformat}
> Note that stderr and prelaunch.err are both zero length files. Though the
> output is not misleading, the header is missing.
> I suggest to add the header for zero length files as well, primarily for the
> following reasons:
> - for applications having multiple files with the same name you may want to
> distinguish them by host - if many of those are of zero length, you can not
> extract this information from here. Note that this is a common case for
> stderr and prelaunch.err.
> - you may want to see the modification time (which corresponds to the
> creation time of the zero length file)
> - would explicitly display the "LogLength:0" line, which would avoid any
> confusion from end user side.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9819) Make TestOpportunisticContainerAllocatorAMService more resilient.
[ https://issues.apache.org/jira/browse/YARN-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927701#comment-16927701 ] Íñigo Goiri commented on YARN-9819: --- +1 on [^YARN-9819.003.patch]. > Make TestOpportunisticContainerAllocatorAMService more resilient. > - > > Key: YARN-9819 > URL: https://issues.apache.org/jira/browse/YARN-9819 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9819.001.patch, YARN-9819.002.patch, > YARN-9819.003.patch > > > Currently, TestOpportunisticContainerAllocatorAMService tries to set the > Opportunistic container status directly in RMNode but that can be updated by > NM heartbeat. Correct way would be to send it through NM heartbeat. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9782) Avoid DNS resolution while running SLS.
[ https://issues.apache.org/jira/browse/YARN-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927699#comment-16927699 ] Íñigo Goiri commented on YARN-9782: --- Thanks [~abmodi] for checking; can we file a separate JIRA to make the delete resilient? Regarding the patch itself, a few minor comments: * It would be better to have the long comment as a javadoc instead of a comment in the caller. * Let's rename enableDnsCaching to enableDNSCaching(). * Let's import the assertEquals() instead of calling Assert all the time. * It may be good to have "networkaddress.cache.ttl" and "networkaddress.cache.negative.ttl" defined somewhere. No lib containing this already? > Avoid DNS resolution while running SLS. > --- > > Key: YARN-9782 > URL: https://issues.apache.org/jira/browse/YARN-9782 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-9782.001.patch, YARN-9782.002.patch, > YARN-9782.003.patch > > > In SLS, we add nodes with random names and rack. DNS resolution of these > nodes takes around 2 seconds because it will timeout after that. This makes > the result of SLS unreliable and adds spikes. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9766) YARN CapacityScheduler QueueMetrics has missing metrics for parent queues having same name
[ https://issues.apache.org/jira/browse/YARN-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927618#comment-16927618 ] Tarun Parimi commented on YARN-9766: [~sunilg], [~eepayne], [~Prabhu Joseph] Please take a look at the patch and review it when possible. The change has been tested manually and I don't see any valid issues with existing unit tests or the added unit tests. > YARN CapacityScheduler QueueMetrics has missing metrics for parent queues > having same name > -- > > Key: YARN-9766 > URL: https://issues.apache.org/jira/browse/YARN-9766 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: YARN-9766.001.patch > > > In Capacity Scheduler, we enforce Leaf Queues to have unique names. But it is > not the case for Parent Queues. For example, we can have the below queue > hierarchy, where "b" is the queue name for two different queue paths root.a.b > and root.a.d.b . Since it is not a leaf queue this configuration works and > apps run fine in the leaf queues 'c' and 'e'. > * root > ** a > *** b > c > *** d > b > * e > But the jmx metrics does not show the metrics for the parent queue > "root.a.d.b" . We can see metrics only for "root.a.b" queue. > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9805) Fine-grained SchedulerNode synchronization
[ https://issues.apache.org/jira/browse/YARN-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927609#comment-16927609 ] Jim Brennan commented on YARN-9805: --- [~ahussein] here's my initial impression of the AutoCloseableRWLock(). I am concerned that some of the methods like release() and isLocked() are adding logic to deal with the fact that we don't know if we are operating on the readlock or the writelock. I think a better approach would be to have wrappers for the readlock and writelock that implement AutoCloseable, rather than trying to do it at the AutoCloseableRWLock() level. > Fine-grained SchedulerNode synchronization > -- > > Key: YARN-9805 > URL: https://issues.apache.org/jira/browse/YARN-9805 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: YARN-9805.001.patch, YARN-9805.002.patch, > YARN-9805.003.patch > > > Yarn schedulerNode and RMNode are using synchronized methods on reading and > updating the resources. > Instead, use read-write reentrant locks to provide fine-grained locking and > to avoid blocking concurrent reads. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9805) Fine-grained SchedulerNode synchronization
[ https://issues.apache.org/jira/browse/YARN-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927594#comment-16927594 ] Jim Brennan commented on YARN-9805: --- [~ahussein], before getting into a detailed review of the changes, I think you need to provide more details about what motivated this change, and specifically why you think this approach is better than the existing code. Changing the synchronization approach in key components of the system is tricky, and I don't think the community is likely to accept this type of change without making a convincing case for why it is better. > Fine-grained SchedulerNode synchronization > -- > > Key: YARN-9805 > URL: https://issues.apache.org/jira/browse/YARN-9805 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: YARN-9805.001.patch, YARN-9805.002.patch, > YARN-9805.003.patch > > > Yarn schedulerNode and RMNode are using synchronized methods on reading and > updating the resources. > Instead, use read-write reentrant locks to provide fine-grained locking and > to avoid blocking concurrent reads. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9815) ReservationACLsTestBase fails with NPE
[
https://issues.apache.org/jira/browse/YARN-9815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927589#comment-16927589
]
Jim Brennan commented on YARN-9815:
---
[~ahussein] I am +1 (non-binding) on patch 002. [~eepayne] or [~ebadger], if
you agree, can one of you commit this?
> ReservationACLsTestBase fails with NPE
> --
>
> Key: YARN-9815
> URL: https://issues.apache.org/jira/browse/YARN-9815
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: YARN-9805.001.patch, YARN-9815.001.patch,
> YARN-9815.002.patch
>
>
> Running ReservationACLsTestBase throws a NPE running the FairScheduler. Old
> revisions back in 2016 also throw NPE.
> In the test case, QueueC does not have reserveACLs, so
> ReservationsACLsManager would throw NPE when it tries to access the ACL on
> line 82.
> I still could not find what was the first revision that caused this test case
> to fail. I stopped at bbfaf3c2712c9ba82b0f8423bdeb314bf505a692 which was
> working fine.
> I have OsX with java 1.8.0_201
>
> {code:java}
> [ERROR]
> testApplicationACLs[1](org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase)
> Time elapsed: 1.897 s <<< ERROR![ERROR]
> testApplicationACLs[1](org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase)
> Time elapsed: 1.897 s <<<
> ERROR!java.lang.NullPointerException:java.lang.NullPointerException at
> org.apache.hadoop.yarn.server.resourcemanager.security.ReservationsACLsManager.checkAccess(ReservationsACLsManager.java:83)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.checkReservationACLs(ClientRMService.java:1527)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1290)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:511)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:645)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:529)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1001) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:929) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2921)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitReservation(ApplicationClientProtocolPBClientImpl.java:511)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase.submitReservation(ReservationACLsTestBase.java:447)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase.verifySubmitReservationSuccess(ReservationACLsTestBase.java:247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ReservationACLsTestBase.testApplicationACLs(ReservationACLsTestBase.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
[jira] [Created] (YARN-9826) Blocked threads at EntityGroupFSTimelineStore#getCachedStore
Harunobu Daikoku created YARN-9826:
--
Summary: Blocked threads at
EntityGroupFSTimelineStore#getCachedStore
Key: YARN-9826
URL: https://issues.apache.org/jira/browse/YARN-9826
Project: Hadoop YARN
Issue Type: Improvement
Components: timelineserver
Affects Versions: 2.7.3
Reporter: Harunobu Daikoku
We have observed this case several times on our production cluster where 100s
of TimelineServer threads are blocked at the following synchronized block in
EntityGroupFSTimelineStore#getCachedStore when our HDFS NameNode is under high
load.
{code:java}
synchronized (this.cachedLogs) {
// Note that the content in the cache log storage may be stale.
cacheItem = this.cachedLogs.get(groupId);
if (cacheItem == null) {
LOG.debug("Set up new cache item for id {}", groupId);
cacheItem = new EntityCacheItem(groupId, getConfig());
AppLogs appLogs = getAndSetAppLogs(groupId.getApplicationId());
if (appLogs != null) {
LOG.debug("Set applogs {} for group id {}", appLogs, groupId);
cacheItem.setAppLogs(appLogs);
this.cachedLogs.put(groupId, cacheItem);
} else {
LOG.warn("AppLogs for groupId {} is set to null!", groupId);
}
}
}
{code}
One thread inside the synchronized block performs multiple fs operations
(fs.exists) inside getAndSetAppLogs, which could block other threads when, for
instance, the NameNode RPC queue is full.
One possible solution is to move getAndSetAppLogs outside the synchronized
block.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9762) Add submission context label to audit logs
[ https://issues.apache.org/jira/browse/YARN-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Kumar updated YARN-9762: -- Attachment: YARN-9762.01.patch > Add submission context label to audit logs > -- > > Key: YARN-9762 > URL: https://issues.apache.org/jira/browse/YARN-9762 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Manoj Kumar >Priority: Major > Labels: release-blocker > Attachments: YARN-9762.01.patch > > > Currently we log NODELABEL in container allocation/release audit logs, we > should also log NODELABEL of application submission context on app submission. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9762) Add submission context label to audit logs
[ https://issues.apache.org/jira/browse/YARN-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Kumar updated YARN-9762: -- Attachment: (was: YARN-9762.01.patch) > Add submission context label to audit logs > -- > > Key: YARN-9762 > URL: https://issues.apache.org/jira/browse/YARN-9762 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Manoj Kumar >Priority: Major > Labels: release-blocker > Attachments: YARN-9762.01.patch > > > Currently we log NODELABEL in container allocation/release audit logs, we > should also log NODELABEL of application submission context on app submission. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9762) Add submission context label to audit logs
[ https://issues.apache.org/jira/browse/YARN-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Kumar updated YARN-9762: -- Attachment: (was: YARN-9762.01.patch) > Add submission context label to audit logs > -- > > Key: YARN-9762 > URL: https://issues.apache.org/jira/browse/YARN-9762 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Manoj Kumar >Priority: Major > Labels: release-blocker > Attachments: YARN-9762.01.patch > > > Currently we log NODELABEL in container allocation/release audit logs, we > should also log NODELABEL of application submission context on app submission. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9674) Max AM Resource calculation is wrong
[ https://issues.apache.org/jira/browse/YARN-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927303#comment-16927303 ] Sunil Govindan commented on YARN-9674: -- Across partitions, it should be same behaviour. Looks like a bug to me. cc [~Prabhu Joseph] > Max AM Resource calculation is wrong > > > Key: YARN-9674 > URL: https://issues.apache.org/jira/browse/YARN-9674 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.1.2 >Reporter: ANANDA G B >Priority: Major > Attachments: RM_Issue.png > > > 'Max AM Resource' calculated for default partition using 'Effective Max > Capacity' and ohter partitions it using 'Effective Capacity'. > Which one is correct implemenation? -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
