[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642814#comment-14642814 ] Sangjin Lee commented on YARN-3369: --- The patch applies to 2.6.0 cleanly. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.7.0 > > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372825#comment-14372825 ] Hudson commented on YARN-3369: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #139 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/139/]) YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) (wangda: rev 6bc7710ec7f2592c4c87dd940fbe5827ef81fe72) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * hadoop-yarn-project/CHANGES.txt > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372790#comment-14372790 ] Hudson commented on YARN-3369: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #130 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/130/]) YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) (wangda: rev 6bc7710ec7f2592c4c87dd940fbe5827ef81fe72) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * hadoop-yarn-project/CHANGES.txt > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372764#comment-14372764 ] Hudson commented on YARN-3369: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2071 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2071/]) YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) (wangda: rev 6bc7710ec7f2592c4c87dd940fbe5827ef81fe72) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372738#comment-14372738 ] Hudson commented on YARN-3369: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2089 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2089/]) YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) (wangda: rev 6bc7710ec7f2592c4c87dd940fbe5827ef81fe72) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372659#comment-14372659 ] Hudson commented on YARN-3369: -- FAILURE: Integrated in Hadoop-Yarn-trunk #873 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/873/]) YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) (wangda: rev 6bc7710ec7f2592c4c87dd940fbe5827ef81fe72) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * hadoop-yarn-project/CHANGES.txt > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372645#comment-14372645 ] Hudson commented on YARN-3369: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #139 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/139/]) YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) (wangda: rev 6bc7710ec7f2592c4c87dd940fbe5827ef81fe72) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * hadoop-yarn-project/CHANGES.txt > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371772#comment-14371772 ] Hudson commented on YARN-3369: -- FAILURE: Integrated in Hadoop-trunk-Commit #7384 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7384/]) YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) (wangda: rev 6bc7710ec7f2592c4c87dd940fbe5827ef81fe72) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371329#comment-14371329 ] Hadoop QA commented on YARN-3369: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705912/YARN-3369-003.patch against trunk revision 8041267. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The following test timeouts occurred in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRM Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7040//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7040//console This message is automatically generated. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > a
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371286#comment-14371286 ] Wangda Tan commented on YARN-3369: -- Thanks for [~rohithsharma] for the additional review, {{getResource}} is only used by tests, but it's better to add null check. Latest patch LGTM, will commit once Jenkins get back. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371240#comment-14371240 ] Brahma Reddy Battula commented on YARN-3369: Thanks for taking look into this issue..Yes.I missed this one..Updated the patch > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370661#comment-14370661 ] Rohith commented on YARN-3369: -- Thanks [~brahmareddy] for providing patch. I think all the caller of getResourceRequest() needs be verified for null check. Below code also should check for null else NPE can be thrown. {code} public synchronized Resource getResource(Priority priority) { ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); return request.getCapability(); } {code} > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370573#comment-14370573 ] Brahma Reddy Battula commented on YARN-3369: Thanks a lot!!! > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370480#comment-14370480 ] Hadoop QA commented on YARN-3369: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705751/YARN-3369.2.patch against trunk revision 91baca1. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7032//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7032//console This message is automatically generated. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369.2.patch, YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedu
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370271#comment-14370271 ] Hadoop QA commented on YARN-3369: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705656/YARN-3369.patch against trunk revision 61a4c7f. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:red}-1 javac{color}. The applied patch generated 1152 javac compiler warnings (more than the trunk's current 206 warnings). {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 43 warning messages. See https://builds.apache.org/job/PreCommit-YARN-Build/7026//artifact/patchprocess/diffJavadocWarnings.txt for details. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ipc.TestRPCWaitForProxy org.apache.hadoop.tracing.TestTracing org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7026//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7026//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7026//console This message is automatically generated. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.j
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369917#comment-14369917 ] Wangda Tan commented on YARN-3369: -- I think we don't need check null here, since Map will be always created when priority added. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369914#comment-14369914 ] Wangda Tan commented on YARN-3369: -- Mostly LGTM, could you make code inner {{if ... }} indent? > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369897#comment-14369897 ] Brahma Reddy Battula commented on YARN-3369: Attached patch ..[~leftnoteasy] and [~kasha] kindly review!! > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369875#comment-14369875 ] Brahma Reddy Battula commented on YARN-3369: Yes,Adding null check is sufficient..priority can null if container got allocated under any other priority,,please check the following snippet for same.. {code} Map nodeRequests = requests.get(priority); return (nodeRequests == null) ? null : nodeRequests.get(resourceName); {code} > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > Attachments: YARN-3369.patch > > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369839#comment-14369839 ] Wangda Tan commented on YARN-3369: -- As far as I can see, this should only happen when user writes their own application, using ApplicationMasterProtocol (instead of AMRMClient). AMRMClient will handle the case -- when user require a host or rack under a priority, offswitch resource request will be added automatically. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369827#comment-14369827 ] Wangda Tan commented on YARN-3369: -- Increased priority to blocker since this will cause RM down no matter which scheduler we are using. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Blocker > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369824#comment-14369824 ] Wangda Tan commented on YARN-3369: -- Took a look at code, this happens when: 1) Application request a resource request (name != *) under a priority, there's no off switch resource request under the priority. 2) Allocate any container under any other priority will trigger the issue. I think add a null check in {{checkForDeactivation}} should be sufficient. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Critical > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369813#comment-14369813 ] Giovanni Matteo Fumarola commented on YARN-3369: You can work on it. > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Critical > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369751#comment-14369751 ] Karthik Kambatla commented on YARN-3369: It would be good to avoid the NPE, however I am surprised we run into this. If there are no ANY requests, why are we allocating a container? Is this an indicator of a scheduler bug? > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula >Priority: Critical > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die
[ https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368347#comment-14368347 ] Brahma Reddy Battula commented on YARN-3369: [~giovanni.fumarola] thanks for reporting..I would like to work on this jira, If you have patch, you can reassign to yourself...thanks > Missing NullPointer check in AppSchedulingInfo causes RM to die > > > Key: YARN-3369 > URL: https://issues.apache.org/jira/browse/YARN-3369 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Brahma Reddy Battula > > In AppSchedulingInfo.java the method checkForDeactivation() has these 2 > consecutive lines: > {code} > ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); > if (request.getNumContainers() > 0) { > {code} > the first line calls getResourceRequest and it can return null. > {code} > synchronized public ResourceRequest getResourceRequest( > Priority priority, String resourceName) { > Map nodeRequests = requests.get(priority); > return (nodeRequests == null) ? {color:red} null : > nodeRequests.get(resourceName); > } > {code} > The second line dereferences the pointer directly without a check. > If the pointer is null, the RM dies. > {quote}2015-03-17 14:14:04,757 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739) > at java.lang.Thread.run(Thread.java:722) > {color:red} *2015-03-17 14:14:04,758 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, > bbye..*{color} {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)