[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438922#comment-15438922 ] Rohith Sharma K S commented on YARN-3940: - +1LGTM > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch, YARN-3940.0007.patch, YARN-3940.0008.patch, > YARN-3940.0009.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438377#comment-15438377 ] Naganarasimha G R commented on YARN-3940: - [~sunilg] & [~rohithsharma], if no other comments on the latest patch, i plan to go ahead and commit the patch. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch, YARN-3940.0007.patch, YARN-3940.0008.patch, > YARN-3940.0009.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436360#comment-15436360 ] Hadoop QA commented on YARN-3940: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 28s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m 8s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 55m 17s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12825411/YARN-3940.0009.patch | | JIRA Issue | YARN-3940 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 05d0b97b3a40 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ab3b727 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12888/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12888/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436235#comment-15436235 ] Sunil G commented on YARN-3940: --- That makes sense. Thanks [~bibinchundatt] > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch, YARN-3940.0007.patch, YARN-3940.0008.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436231#comment-15436231 ] Bibin A Chundatt commented on YARN-3940: [~sunilg] Will add as a comment that all labels in life time will be considered while move. Additional requests with different partition can come in any time so best option was current. Patch will be updated soon adding comment,minor testcode change and ANY scenarios handling > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch, YARN-3940.0007.patch, YARN-3940.0008.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436200#comment-15436200 ] Sunil G commented on YARN-3940: --- Sorry for pitching in late. I thought of discussing a scenario here. I know [~leftnoteasy] and [~bibinchundatt] were trying to make simpler implementation before clear usecases are available. But one scenario popped up while I was checking a recent issue from Jason YARN-5540. Assume an application's AM containers was given a specific node label and it got allocated. Now moveToQueue is performed, eventhough there are no pending resource requests on AM container any more, I think as per current impl will block move queue. This is because {{requestedPartitions}} is not cleared with its ANY requests. So I have 2 doubts: - AM container allocated on label1. Now label1 is no more used by app1. Can app1 be moved to a queue which does not *label1* as its labels. - There were some out standing requests for a label called *label2*. But app doesnt have any more outstanding there. This means that there are no containers for this app is running on *label2*. Do we need to block this app from moving to another queue which does nt have *label2* I think its better to note this case for now some where so that we can track it later. [~bibinchundatt] [~Naganarasimha Garla] thoughts? > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch, YARN-3940.0007.patch, YARN-3940.0008.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435528#comment-15435528 ] Naganarasimha G R commented on YARN-3940: - Thanks for pointing it out [~rohithsharma], just cross verified whether in any case leaf queue can return null, IIUC if nothing is configured it will inherit from the parent and by default root queue is *ANY*, so {{AccessibleNodeLabels}} for a queue can be having some set of values, empty or *ANY* but never null, so just correcting the log should be sufficient. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch, YARN-3940.0007.patch, YARN-3940.0008.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435472#comment-15435472 ] Rohith Sharma K S commented on YARN-3940: - Unexpectedly I focused on this JIRA. And got interest to review this patch too. Overall patch looks good to me as well. One small nit, but not very mandatory to fix here. # {{dest.getAccessibleNodeLabels() == null ? ""}} is not necessarily required. If it is null, then execution will fail in {{targetqueuelabels.containsAll(appLabels);}} itself. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch, YARN-3940.0007.patch, YARN-3940.0008.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435390#comment-15435390 ] Naganarasimha G R commented on YARN-3940: - Thanks for the patch [~bibinchundatt], +1 LGTM, Will wait for some time before committing! > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch, YARN-3940.0007.patch, YARN-3940.0008.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434535#comment-15434535 ] Hadoop QA commented on YARN-3940: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 44s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m 7s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 55m 13s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12825219/YARN-3940.0008.patch | | JIRA Issue | YARN-3940 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 5b7366cbedf1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 092b4d5 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12873/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12873/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433672#comment-15433672 ] Naganarasimha G R commented on YARN-3940: - Hi [~bibinchundatt], {code} boolean isValidrequest = SchedulerUtils.checkQueueLabelExpression( targetqueuelabels, StringUtils.join(appLabels, ","), null); {code} Seems like the approach is wrong here, as check in {{checkQueueLabelExpression}} is checking for {{"&&" }} but i would suggest each label expression present in app's resource request to be validated against the queue's accessible labels rather than doing in this way !. Few nits about the patch : # Add a test case for different resource request requesting for diff partitions # TestCapacitySchedulerNodeLabelUpdate l no. 445 & 447, have proper message that target queue of move operation doesn't have the required labels # Please cross check the reported Checkstyle issues which seems to be valid. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch, YARN-3940.0007.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431237#comment-15431237 ] Hadoop QA commented on YARN-3940: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 58s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 133 unchanged - 0 fixed = 141 total (was 133) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 24s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 51m 57s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart | | | hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12824886/YARN-3940.0007.patch | | JIRA Issue | YARN-3940 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux ed6cf6c2ea56 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 115ecb5 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/12850/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/12850/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit test logs | https://builds.apache.org/job/PreCommit-YARN-Build/12850/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12850/te
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120731#comment-15120731 ] Wangda Tan commented on YARN-3940: -- I think you can take a look at AppSchedulingInfo#getRequestedPartitions (will be added with YARN-3215) to check if an app can run in another queue. It could be better than simply use labels in ASC. [~Naganarasimha]. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106437#comment-15106437 ] Bibin A Chundatt commented on YARN-3940: Hi [~leftnoteasy], Could you please review patch attached. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch, > 0006-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089764#comment-15089764 ] Hadoop QA commented on YARN-3940: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 53s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 24s {color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s {color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 36s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s {color} | {color:red} Patch generated 1 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 113, now 113). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 31s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 13s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 5s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 144m 14s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_66 Failed junit tests | hadoop.yarn.server.resourcemanager.TestAMAuthorization | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler | | | hadoop.yarn.server.resourcemanager.TestClientRMTokens | | JDK v1.7.0_91 Failed junit tests | hadoop.yarn.server.resourcemanager.TestAMAuthorization | | | hadoop.yarn.server.resourcemanager.TestClientRMTokens | \\ \\ || Subsystem ||
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088779#comment-15088779 ] Bibin A Chundatt commented on YARN-3940: Hi [~Naganarasimha] Thank you for review .I have already taken care of that too {noformat} || targetqueuelabels.contains(RMNodeLabelsManager.ANY)) {noformat} > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch, 0005-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087686#comment-15087686 ] Hadoop QA commented on YARN-3940: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 29s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s {color} | {color:red} Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 113, now 114). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 19s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager introduced 1 new FindBugs issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m 11s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed with JDK v1.8.0_66. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m 28s {color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 144m 55s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | | java.util.Set is incompatible with expected argument type String in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkQueuePartition(ApplicationId, LeafQueue) At CapacityScheduler.java:argument type String in or
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086195#comment-15086195 ] Hadoop QA commented on YARN-3940: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 3s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s {color} | {color:red} Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 113, now 114). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 34s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager introduced 1 new FindBugs issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 25s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 31s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 145m 29s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | | java.util.Set is incompatible with expected argument type String in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkQueuePartition(ApplicationId, LeafQueue) At CapacityScheduler.java:argument type String in org.apache.hadoop.y
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085972#comment-15085972 ] Sunil G commented on YARN-3940: --- Thanks [~bibinchundatt] Few comments: 1. {{ LeafQueue dest}} --> {{ LeafQueue target}} seems more better. 2. destqueuelabels --> targetQueueLabels 3. {code} +Set sourcequeuelabels = +getAndCheckLeafQueue(sourceQueueName).getAccessibleNodeLabels(); .. .. if (destqueuelabels.contains(sourcequeuelabels) {code} I think its not very good to have a check like this. Its fine and valid, but we could check for those labels which are used by this app under movement. Such an api can be implemented, and i think it will be better. 4. {{public FiCaSchedulerApp getApplicationAttempt(ApplicationAttemptId applicationAttemptId) }}can be used to get the app object directly instead of 2 calls explicitly. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085963#comment-15085963 ] Hadoop QA commented on YARN-3940: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 44s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 43s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s {color} | {color:red} Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 113, now 114). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 18s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager introduced 1 new FindBugs issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 37s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 43s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 147m 32s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | | LeafQueue is incompatible with expected argument type String in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkQueuePartition(ApplicationId, LeafQueue) At CapacityScheduler.java:argument type String in org.apache.hadoop.yarn.
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085917#comment-15085917 ] Bibin A Chundatt commented on YARN-3940: Uploading patch after fixing check style issue. Please do review latest patch > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch, > 0003-YARN-3940.patch, 0004-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085828#comment-15085828 ] Hadoop QA commented on YARN-3940: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s {color} | {color:red} Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 113, now 114). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 17s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager introduced 1 new FindBugs issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 4s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 24s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 136m 53s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | | LeafQueue is incompatible with expected argument type String in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkQueuePartition(ApplicationId, LeafQueue) At CapacityScheduler.java:argument type String in org.apache.hadoop.yarn.s
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073930#comment-15073930 ] Bibin A Chundatt commented on YARN-3940: Hi [~leftnoteasy] Sorry for delayed response. But if QueueA accessible labels X,Y QueueB accesible labels are z,y If application 1 submitted to label y of Queue A should be able to move to Queue B rt?? Might not be a valid use case. Thoughts? > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681880#comment-14681880 ] Bibin A Chundatt commented on YARN-3940: [~leftnoteasy] Thanks for comment I have small doubt please clarify Consider Queue A is having access to x,y Queue A Default label=y Queue B is having access to y Queue B Default label=y # Case 1: If application is submitted as below *-Dmapreduce.job.node-label-expression=y Queue=A* and try to move from Queue A to Queue B should work ?? {quote} Which is the dest.accessible-node-label is-super-set-of origin.accessible-node-label. {quote} But if we check as per above suggestion will not allow to move rt? Check should be context label of app is subset of dest-access-label queue then should allow to move.(0002-YARN-3940.patch) # Case 2: if application is submitted to queue ,default node label expression for queue is configured and label explicitly not mentioned then {{rmApp.getApplicationSubmissionContext().getNodeLabelExpression()==null}} so in this case the *source default queue nodelabel* should be subset of *dest.accessible-node-label* of queue if these two condition are not satisfying then check {{dest.accessible-node-label is-super-set-of origin.accessible-node-label}} . I support combination of above all since {{check resource usage by partition}} is not required. Please correct me if i am wrong. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680538#comment-14680538 ] Wangda Tan commented on YARN-3940: -- [~bibinchundatt], Rethink about this, I think to make it simpler, we can check if dest queue is compatible to original queue. Which is the dest.accessible-node-label is-super-set-of origin.accessible-node-label. Benefits of doing this: - No need to check resource usage by partition. - Application can still ask for resource request with label supported by original queue. - Avoid premature complexities before we have concrete use cases. Thoughts? > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680272#comment-14680272 ] Bibin A Chundatt commented on YARN-3940: [~leftnoteasy] Any thoughts?? > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648939#comment-14648939 ] Bibin A Chundatt commented on YARN-3940: For first case mentioned above Will check number of used containers/Resource for *application has usage* validation using {code} RMApp rmApp = rmContext.getRMApps().get(appId); rmApp.getCurrentAppAttempt().getApplicationResourceUsageReport() {code} > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648686#comment-14648686 ] Bibin A Chundatt commented on YARN-3940: Hi [~leftnoteasy] Thank you for review comments. {quote} We should check usage as I mentioned at: https://issues.apache.org/jira/browse/YARN-3940?focusedCommentId=14633876&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14633876. {quote} Will check how handle this too. {quote} we may need to consider how to deal with node label update, currently, if we change labels on a node, all containers running on the node will be killed. I suggest to clear think about both of the problem before moving forward. {quote} As i understand the below cases containers shouldn't be killed # Running containers of applications submitted for default partition on partition with label incase of exclusivity(false) # when queue is having access to new label / Node Any other case ? Can we move second part to separate jira for discussion ? Thoughts? Please do correct me if i am wrong. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647955#comment-14647955 ] Wangda Tan commented on YARN-3940: -- [~bibinchundatt], I took a look at the patch, it checked app's node-label-expression AND queue's accessible-node-label, which is not enough to me. We should check usage as I mentioned at: https://issues.apache.org/jira/browse/YARN-3940?focusedCommentId=14633876&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14633876. Actually I don't have a clear idea about how to solve this problem as well. Another related problem to this is, we may need to consider how to deal with node label update, currently, if we change labels on a node, all containers running on the node will be killed. I suggest to clear think about both of the problem before moving forward. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14641914#comment-14641914 ] Hadoop QA commented on YARN-3940: - \\ \\ | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 16m 3s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 38s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 0m 47s | There were no new checkstyle issues. | | {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 22s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 25s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | yarn tests | 52m 26s | Tests passed in hadoop-yarn-server-resourcemanager. | | | | 90m 17s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12747227/0002-YARN-3940.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 156f24e | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8673/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8673/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8673/console | This message was automatically generated. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch, 0002-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14641769#comment-14641769 ] Hadoop QA commented on YARN-3940: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 15m 7s | Findbugs (version ) appears to be broken on trunk. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 38s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 0m 23s | There were no new checkstyle issues. | | {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 22s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 26s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | yarn tests | 52m 16s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 88m 51s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12746872/0001-YARN-3940.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 156f24e | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8672/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8672/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8672/console | This message was automatically generated. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-3940.patch > > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) >
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633881#comment-14633881 ] Bibin A Chundatt commented on YARN-3940: [~leftnoteasy] Thnks for comment will upload the patch soon. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633876#comment-14633876 ] Wangda Tan commented on YARN-3940: -- [~bibinchundatt], I agree with you. One approach is in {{CS#moveApplication}}, you can check resource usage (by partition) for application, and accessible-node-label of destination queue. If a node partition is an exclusive partition AND application has usage on it AND destination queue cannot access it, we should stop the moving. Does this make sense to you? > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633870#comment-14633870 ] Bibin A Chundatt commented on YARN-3940: [~leftnoteasy] IMO ,Since the destination queue is not having permission for label we should not allow to move. *Currenty its giving success message* . We can validate permission of nodeLabel specified {{X}} in {{CapacityScheduler#moveApplication}} for the destination queue. Is that ok? Please do correct me if i am wrong. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3940) Application moveToQueue should check NodeLabel permission
[ https://issues.apache.org/jira/browse/YARN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633847#comment-14633847 ] Wangda Tan commented on YARN-3940: -- I just converted this to sub-task of YARN-2492 for better tracking. IMHO, AM should handle such failures, like change resource request, etc. And currently, invalid resource request will be interpret as invalid heartbeat, this has pros/cons: - Pros: It can make sure no application is hang because of ignoring InvalidResourceRequestException. - Cons: An application will be killed because of ignoring InvalidResourceRequestException. I'm not sure if we need to change this behavior in YARN side. > Application moveToQueue should check NodeLabel permission > -- > > Key: YARN-3940 > URL: https://issues.apache.org/jira/browse/YARN-3940 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Configure capacity scheduler > Configure node label an submit application {{queue=A Label=X}} > Move application to queue {{B}} and x is not having access > {code} > 2015-07-20 19:46:19,626 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Application attempt appattempt_1437385548409_0005_01 released container > container_e08_1437385548409_0005_01_02 on node: host: > host-10-19-92-117:64318 #containers=1 available= > used= with event: KILL > 2015-07-20 19:46:20,970 WARN > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > Invalid resource ask by application appattempt_1437385548409_0005_01 > org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid > resource request, queue=b1 doesn't have permission to access all labels in > resource request. labelExpression of resource request=x. Queue labels=y > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250) > at > org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106) > at > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:515) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168) > {code} > Same exception will be thrown till *heartbeat timeout* > Then application state will be updated to *FAILED* -- This message was sent by Atlassian JIRA (v6.3.4#6332)