[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-11-23 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15022885#comment-15022885
 ] 

Sangjin Lee commented on YARN-3762:
---

This should be a good candidate for branch-2.6 (2.6.3). [~kasha], what do you 
think?

> FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
> ---
>
> Key: YARN-3762
> URL: https://issues.apache.org/jira/browse/YARN-3762
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.7.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch
>
>
> In our testing, we ran into the following ConcurrentModificationException:
> {noformat}
> halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
> 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
> queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
> queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
> 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
> java.util.ConcurrentModificationException: 
> java.util.ConcurrentModificationException
>   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>   at java.util.ArrayList$Itr.next(ArrayList.java:851)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-04 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573006#comment-14573006
 ] 

Hudson commented on YARN-3762:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #216 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/216/])
YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) 
(kasha: rev edb9cd0f7aa1ecaf34afaa120e3d79583e0ec689)
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java


 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Fix For: 2.8.0

 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-04 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573034#comment-14573034
 ] 

Hudson commented on YARN-3762:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2164 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2164/])
YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) 
(kasha: rev edb9cd0f7aa1ecaf34afaa120e3d79583e0ec689)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java


 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Fix For: 2.8.0

 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-04 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572889#comment-14572889
 ] 

Hudson commented on YARN-3762:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #207 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/207/])
YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) 
(kasha: rev edb9cd0f7aa1ecaf34afaa120e3d79583e0ec689)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* hadoop-yarn-project/CHANGES.txt


 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Fix For: 2.8.0

 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-04 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572861#comment-14572861
 ] 

Hudson commented on YARN-3762:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #2146 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2146/])
YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) 
(kasha: rev edb9cd0f7aa1ecaf34afaa120e3d79583e0ec689)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* hadoop-yarn-project/CHANGES.txt


 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Fix For: 2.8.0

 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-04 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572591#comment-14572591
 ] 

Hudson commented on YARN-3762:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #218 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/218/])
YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) 
(kasha: rev edb9cd0f7aa1ecaf34afaa120e3d79583e0ec689)
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java


 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Fix For: 2.8.0

 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-04 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572611#comment-14572611
 ] 

Hudson commented on YARN-3762:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #948 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/948/])
YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) 
(kasha: rev edb9cd0f7aa1ecaf34afaa120e3d79583e0ec689)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* hadoop-yarn-project/CHANGES.txt
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java


 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Fix For: 2.8.0

 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-03 Thread Karthik Kambatla (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571502#comment-14571502
 ] 

Karthik Kambatla commented on YARN-3762:


Thanks for the review, Arun. Good points.

bq. what happens if the collection is modified in between..
The two possible modifications are adding/removing a child queue. Adding a 
child queue to the end of the list doesn't affect container assignment. 
Removing a child queue affects container assignment, but that is a good thing. 
We should probably add a comment to that effect so we don't forget this in the 
future.

bq. instead of using a List and sorting it everytime, we could use a Sorted Bag 
(MultiSet) ? 
One issue with using a sorted list is the sorting happens on addition/removal. 
FSQueues already in the list also change affecting the order. May be, we 
could remove and re-insert the queue if anything changes, but that is a much 
bigger change and needs to be carefully evaluated for performance.

 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Attachments: yarn-3762-1.patch, yarn-3762-1.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-03 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571611#comment-14571611
 ] 

Hadoop QA commented on YARN-3762:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  15m 56s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:red}-1{color} | tests included |   0m  0s | The patch doesn't appear 
to include any new or modified tests.  Please justify why no new tests are 
needed for this patch. Also please list what manual steps were performed to 
verify this patch. |
| {color:green}+1{color} | javac |   7m 32s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 35s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 45s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  1s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   1m 26s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests |  50m  7s | Tests passed in 
hadoop-yarn-server-resourcemanager. |
| | |  87m 55s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12737349/yarn-3762-2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / dbc4f64 |
| hadoop-yarn-server-resourcemanager test log | 
https://builds.apache.org/job/PreCommit-YARN-Build/8182/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/8182/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/8182/console |


This message was automatically generated.

 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-03 Thread Karthik Kambatla (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571618#comment-14571618
 ] 

Karthik Kambatla commented on YARN-3762:


Thanks Arun. Checking this in. 

 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-03 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571626#comment-14571626
 ] 

Hudson commented on YARN-3762:
--

FAILURE: Integrated in Hadoop-trunk-Commit #7955 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7955/])
YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) 
(kasha: rev edb9cd0f7aa1ecaf34afaa120e3d79583e0ec689)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
* hadoop-yarn-project/CHANGES.txt


 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Fix For: 2.8.0

 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-03 Thread Arun Suresh (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571512#comment-14571512
 ] 

Arun Suresh commented on YARN-3762:
---

Makes sense
+1, LGTM

 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Attachments: yarn-3762-1.patch, yarn-3762-1.patch, yarn-3762-2.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-03 Thread Karthik Kambatla (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571272#comment-14571272
 ] 

Karthik Kambatla commented on YARN-3762:


The patch just adds read-write locks to address any races. Haven't added any 
tests since it is hard to test race conditions. The code changed is accessed 
directly or indirectly by existing tests. Running jcarder should catch any 
deadlocks introduced by this change.

 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Attachments: yarn-3762-1.patch, yarn-3762-1.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-03 Thread Arun Suresh (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571409#comment-14571409
 ] 

Arun Suresh commented on YARN-3762:
---

[~kasha], Thanks for the patch

* In the {{assignContainer}} method, you are sorting the collection under a 
write lock and then assigning the container within the read lock, what happens 
if the collection is modified in between.. shouldnt we have the write lock 
encompass both operations ? I agree, It will not lead to the Exception (which 
is the point of the JIRA) but I feel it should be done for correctness.

* One other possible improvement is maybe instead of using a List and sorting 
it everytime, we could use a Sorted Bag (MultiSet) ? which keeps the list in 
sorted order ?


 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Attachments: yarn-3762-1.patch, yarn-3762-1.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-02 Thread Karthik Kambatla (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569992#comment-14569992
 ] 

Karthik Kambatla commented on YARN-3762:


Changed it to critical and targeting 2.8.0, as it only fails the application 
and not the RM.

 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Attachments: yarn-3762-1.patch, yarn-3762-1.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3762) FairScheduler: CME on FSParentQueue#getQueueUserAclInfo

2015-06-02 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570082#comment-14570082
 ] 

Hadoop QA commented on YARN-3762:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  15m 28s | Findbugs (version ) appears to 
be broken on trunk. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:red}-1{color} | tests included |   0m  0s | The patch doesn't appear 
to include any new or modified tests.  Please justify why no new tests are 
needed for this patch. Also please list what manual steps were performed to 
verify this patch. |
| {color:green}+1{color} | javac |   7m 53s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 51s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 32s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  1s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 34s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   1m 28s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests |  50m 25s | Tests passed in 
hadoop-yarn-server-resourcemanager. |
| | |  88m 17s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12737043/yarn-3762-1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / c1d50a9 |
| hadoop-yarn-server-resourcemanager test log | 
https://builds.apache.org/job/PreCommit-YARN-Build/8170/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/8170/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/8170/console |


This message was automatically generated.

 FairScheduler: CME on FSParentQueue#getQueueUserAclInfo
 ---

 Key: YARN-3762
 URL: https://issues.apache.org/jira/browse/YARN-3762
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.7.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
 Attachments: yarn-3762-1.patch, yarn-3762-1.patch


 In our testing, we ran into the following ConcurrentModificationException:
 {noformat}
 halxg.cloudera.com:8042, nodeRackName/rackvb07, nodeNumContainers0
 15/05/22 13:02:22 INFO distributedshell.Client: Queue info, 
 queueName=root.testyarnpool3, queueCurrentCapacity=0.0, 
 queueMaxCapacity=-1.0, queueApplicationCount=0, queueChildQueueCount=0
 15/05/22 13:02:22 FATAL distributedshell.Client: Error running Client
 java.util.ConcurrentModificationException: 
 java.util.ConcurrentModificationException
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
   at java.util.ArrayList$Itr.next(ArrayList.java:851)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.getQueueUserAclInfo(FSParentQueue.java:155)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.getQueueUserAclInfo(FairScheduler.java:1395)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:880)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)