[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-07-27 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642814#comment-14642814
 ] 

Sangjin Lee commented on YARN-3369:
---

The patch applies to 2.6.0 cleanly.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
  Labels: 2.6.1-candidate
 Fix For: 2.7.0

 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-21 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372825#comment-14372825
 ] 

Hudson commented on YARN-3369:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #139 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/139/])
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. 
(Brahma Reddy Battula via wangda) (wangda: rev 
6bc7710ec7f2592c4c87dd940fbe5827ef81fe72)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
* hadoop-yarn-project/CHANGES.txt


 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-21 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372790#comment-14372790
 ] 

Hudson commented on YARN-3369:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #130 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/130/])
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. 
(Brahma Reddy Battula via wangda) (wangda: rev 
6bc7710ec7f2592c4c87dd940fbe5827ef81fe72)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
* hadoop-yarn-project/CHANGES.txt


 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-21 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372738#comment-14372738
 ] 

Hudson commented on YARN-3369:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2089 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2089/])
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. 
(Brahma Reddy Battula via wangda) (wangda: rev 
6bc7710ec7f2592c4c87dd940fbe5827ef81fe72)
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java


 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-21 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372764#comment-14372764
 ] 

Hudson commented on YARN-3369:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2071 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2071/])
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. 
(Brahma Reddy Battula via wangda) (wangda: rev 
6bc7710ec7f2592c4c87dd940fbe5827ef81fe72)
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java


 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-21 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372645#comment-14372645
 ] 

Hudson commented on YARN-3369:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #139 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/139/])
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. 
(Brahma Reddy Battula via wangda) (wangda: rev 
6bc7710ec7f2592c4c87dd940fbe5827ef81fe72)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
* hadoop-yarn-project/CHANGES.txt


 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-21 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372659#comment-14372659
 ] 

Hudson commented on YARN-3369:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #873 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/873/])
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. 
(Brahma Reddy Battula via wangda) (wangda: rev 
6bc7710ec7f2592c4c87dd940fbe5827ef81fe72)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
* hadoop-yarn-project/CHANGES.txt


 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-20 Thread Brahma Reddy Battula (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371240#comment-14371240
 ] 

Brahma Reddy Battula commented on YARN-3369:


Thanks for taking look into this issue..Yes.I missed this one..Updated the patch

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-20 Thread Wangda Tan (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371286#comment-14371286
 ] 

Wangda Tan commented on YARN-3369:
--

Thanks for [~rohithsharma] for the additional review, {{getResource}} is only 
used by tests, but it's better to add null check.

Latest patch LGTM, will commit once Jenkins get back.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-20 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371329#comment-14371329
 ] 

Hadoop QA commented on YARN-3369:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12705912/YARN-3369-003.patch
  against trunk revision 8041267.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The following test timeouts occurred in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.yarn.server.resourcemanager.TestRM

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/7040//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7040//console

This message is automatically generated.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 

[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-20 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371772#comment-14371772
 ] 

Hudson commented on YARN-3369:
--

FAILURE: Integrated in Hadoop-trunk-Commit #7384 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7384/])
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. 
(Brahma Reddy Battula via wangda) (wangda: rev 
6bc7710ec7f2592c4c87dd940fbe5827ef81fe72)
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java


 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Fix For: 2.7.0

 Attachments: YARN-3369-003.patch, YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370480#comment-14370480
 ] 

Hadoop QA commented on YARN-3369:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12705751/YARN-3369.2.patch
  against trunk revision 91baca1.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/7032//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7032//console

This message is automatically generated.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)

[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370271#comment-14370271
 ] 

Hadoop QA commented on YARN-3369:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12705656/YARN-3369.patch
  against trunk revision 61a4c7f.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

  {color:red}-1 javac{color}.  The applied patch generated 1152 javac 
compiler warnings (more than the trunk's current 206 warnings).

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
43 warning messages.
See 
https://builds.apache.org/job/PreCommit-YARN-Build/7026//artifact/patchprocess/diffJavadocWarnings.txt
 for details.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.ipc.TestRPCWaitForProxy
  org.apache.hadoop.tracing.TestTracing
  
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
  org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/7026//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-YARN-Build/7026//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7026//console

This message is automatically generated.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 

[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Brahma Reddy Battula (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370573#comment-14370573
 ] 

Brahma Reddy Battula commented on YARN-3369:


Thanks a lot!!!

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Rohith (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370661#comment-14370661
 ] 

Rohith commented on YARN-3369:
--

Thanks [~brahmareddy] for providing patch.
I think all the caller of getResourceRequest() needs be verified for null 
check.  Below code also should check for null else NPE can be thrown. 
{code}
 public synchronized Resource getResource(Priority priority) {
ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
return request.getCapability();
  }
{code}

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369.2.patch, YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Karthik Kambatla (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369751#comment-14369751
 ] 

Karthik Kambatla commented on YARN-3369:


It would be good to avoid the NPE, however I am surprised we run into this. If 
there are no ANY requests, why are we allocating a container? Is this an 
indicator of a scheduler bug? 

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Critical

 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Giovanni Matteo Fumarola (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369813#comment-14369813
 ] 

Giovanni Matteo Fumarola commented on YARN-3369:


You can work on it.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Critical

 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Wangda Tan (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369827#comment-14369827
 ] 

Wangda Tan commented on YARN-3369:
--

Increased priority to blocker since this will cause RM down no matter which 
scheduler we are using.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker

 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Wangda Tan (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369839#comment-14369839
 ] 

Wangda Tan commented on YARN-3369:
--

As far as I can see, this should only happen when user writes their own 
application, using ApplicationMasterProtocol (instead of AMRMClient). 
AMRMClient will handle the case -- when user require a host or rack under a 
priority, offswitch resource request will be added automatically.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker

 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Wangda Tan (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369824#comment-14369824
 ] 

Wangda Tan commented on YARN-3369:
--

Took a look at code, this happens when:
1) Application request a resource request (name != *) under a priority, there's 
no off switch resource request under the priority.
2) Allocate any container under any other priority will trigger the issue.

I think add a null check in {{checkForDeactivation}} should be sufficient.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Critical

 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Brahma Reddy Battula (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369875#comment-14369875
 ] 

Brahma Reddy Battula commented on YARN-3369:


Yes,Adding null check is sufficient..priority can null if container got 
allocated under any other priority,,please check the following snippet for 
same..

{code}
MapString, ResourceRequest nodeRequests = requests.get(priority);
return (nodeRequests == null) ? null : nodeRequests.get(resourceName);
{code}

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Wangda Tan (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369914#comment-14369914
 ] 

Wangda Tan commented on YARN-3369:
--

Mostly LGTM, could you make code inner {{if ... }} indent?

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Wangda Tan (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369917#comment-14369917
 ] 

Wangda Tan commented on YARN-3369:
--

I think we don't need check null here, since MapString, RR will be always 
created when priority added.

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-19 Thread Brahma Reddy Battula (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369897#comment-14369897
 ] 

Brahma Reddy Battula commented on YARN-3369:


Attached patch ..[~leftnoteasy] and [~kasha] kindly review!!

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula
Priority: Blocker
 Attachments: YARN-3369.patch


 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3369) Missing NullPointer check in AppSchedulingInfo causes RM to die

2015-03-18 Thread Brahma Reddy Battula (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14368347#comment-14368347
 ] 

Brahma Reddy Battula commented on YARN-3369:


[~giovanni.fumarola] thanks for reporting..I would like to work on this jira, 
If you have patch, you can reassign to yourself...thanks

 Missing NullPointer check in AppSchedulingInfo causes RM to die 
 

 Key: YARN-3369
 URL: https://issues.apache.org/jira/browse/YARN-3369
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Giovanni Matteo Fumarola
Assignee: Brahma Reddy Battula

 In AppSchedulingInfo.java the method checkForDeactivation() has these 2 
 consecutive lines:
 {code}
 ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY);
 if (request.getNumContainers()  0) {
 {code}
 the first line calls getResourceRequest and it can return null.
 {code}
 synchronized public ResourceRequest getResourceRequest(
 Priority priority, String resourceName) {
 MapString, ResourceRequest nodeRequests = requests.get(priority);
 return  (nodeRequests == null) ? {color:red} null : 
 nodeRequests.get(resourceName);
 }
 {code}
 The second line dereferences the pointer directly without a check.
 If the pointer is null, the RM dies. 
 {quote}2015-03-17 14:14:04,757 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
 handling event type NODE_UPDATE to the scheduler
 java.lang.NullPointerException
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.checkForDeactivation(AppSchedulingInfo.java:383)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.decrementOutstanding(AppSchedulingInfo.java:375)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocateOffSwitch(AppSchedulingInfo.java:360)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:270)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.allocate(FiCaSchedulerApp.java:142)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1559)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1384)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1263)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:816)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:588)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:449)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1017)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1059)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:739)
 at java.lang.Thread.run(Thread.java:722)
 {color:red} *2015-03-17 14:14:04,758 INFO 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, 
 bbye..*{color} {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)