[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631401#comment-14631401
]
Hudson commented on YARN-3885:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #248 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/248/])
YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is
more than 2 level. (Ajith S via wangda) (wangda: rev
3540d5fe4b1da942ea80c9e7ca1126b1abb8a68a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Fix For: 2.8.0
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631230#comment-14631230
]
Hudson commented on YARN-3885:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #259 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/259/])
YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is
more than 2 level. (Ajith S via wangda) (wangda: rev
3540d5fe4b1da942ea80c9e7ca1126b1abb8a68a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
* hadoop-yarn-project/CHANGES.txt
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Fix For: 2.8.0
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631240#comment-14631240
]
Hudson commented on YARN-3885:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #989 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/989/])
YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is
more than 2 level. (Ajith S via wangda) (wangda: rev
3540d5fe4b1da942ea80c9e7ca1126b1abb8a68a)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Fix For: 2.8.0
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631390#comment-14631390
]
Hudson commented on YARN-3885:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2186 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2186/])
YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is
more than 2 level. (Ajith S via wangda) (wangda: rev
3540d5fe4b1da942ea80c9e7ca1126b1abb8a68a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
* hadoop-yarn-project/CHANGES.txt
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Fix For: 2.8.0
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631304#comment-14631304
]
Hudson commented on YARN-3885:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #256 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/256/])
YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is
more than 2 level. (Ajith S via wangda) (wangda: rev
3540d5fe4b1da942ea80c9e7ca1126b1abb8a68a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
* hadoop-yarn-project/CHANGES.txt
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Fix For: 2.8.0
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631456#comment-14631456
]
Hudson commented on YARN-3885:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2205 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2205/])
YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is
more than 2 level. (Ajith S via wangda) (wangda: rev
3540d5fe4b1da942ea80c9e7ca1126b1abb8a68a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Fix For: 2.8.0
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629555#comment-14629555
]
Ajith S commented on YARN-3885:
---
not because of the patch
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629446#comment-14629446
]
Hadoop QA commented on YARN-3885:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 12s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 46s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 50s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 18s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 23s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 61m 19s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 99m 23s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12745584/YARN-3885.08.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 90bda9c |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8555/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8555/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8555/console |
This message was automatically generated.
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630022#comment-14630022
]
Wangda Tan commented on YARN-3885:
--
Patch LGTM, +1, will commit soon. Thanks [~ajithshetty].
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630669#comment-14630669
]
Ajith S commented on YARN-3885:
---
Thanks [~leftnoteasy] , [~xinxianyin] and [~sunilg] :)
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Fix For: 2.8.0
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630509#comment-14630509
]
Hudson commented on YARN-3885:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8177 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8177/])
YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is
more than 2 level. (Ajith S via wangda) (wangda: rev
3540d5fe4b1da942ea80c9e7ca1126b1abb8a68a)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
* hadoop-yarn-project/CHANGES.txt
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Fix For: 2.8.0
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.08.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627719#comment-14627719
]
Hadoop QA commented on YARN-3885:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 8s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 13s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 20s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 39s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 9m 35s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 49m 49s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
|
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
|
| |
hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
|
| | hadoop.yarn.server.resourcemanager.security.TestAMRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAppManager |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestChildQueueOrder
|
| |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityReservationSystem |
| | hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels |
| | hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
|
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication |
| |
hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower
|
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| |
hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicyForNodePartitions
|
| |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates |
| | hadoop.yarn.server.resourcemanager.scheduler.TestQueueMetrics |
| | hadoop.yarn.server.resourcemanager.TestRMNodeTransitions |
| | hadoop.yarn.server.resourcemanager.TestResourceManager |
| |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId |
| | hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservationQueue |
| | hadoop.yarn.server.resourcemanager.TestRM |
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry |
| | hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.TestApplicationACLs |
| | hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
| | hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf |
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerEventLog |
| | hadoop.yarn.server.resourcemanager.webapp.TestAppPage |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue |
| | hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
| | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing |
| | hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
| |
hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower |
| |
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627838#comment-14627838
]
Hadoop QA commented on YARN-3885:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 31s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 48s | The applied patch generated 1
new checkstyle issues (total was 63, now 64). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 45m 7s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 84m 54s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.security.TestAMRMTokens |
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| | org.apache.hadoop.yarn.server.resourcemanager.TestRM |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12745407/YARN-3885.07.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / edcaae4 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8542/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8542/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8542/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8542/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8542/console |
This message was automatically generated.
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627698#comment-14627698
]
Ajith S commented on YARN-3885:
---
Submitted new patch. this might give one white space warning, but its better to
maintain the formatting so that the test case is better explained.
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627735#comment-14627735
]
Ajith S commented on YARN-3885:
---
these failures aren't because of patch i feel :)
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.06.patch,
YARN-3885.07.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627599#comment-14627599
]
Ajith S commented on YARN-3885:
---
Hi [~leftnoteasy]
Yeah, i will change the patch for using @VisibleForTesting. But the test case
failure seems strange. In my box, the queueA has CUR=209 but in the test case
log, it looks like it has CUR=210. Will check this
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626186#comment-14626186
]
Ajith S commented on YARN-3885:
---
Hi [~leftnoteasy], thanks for the input
The testcase i have added, has the same scenario, as you can see form the log
snippet, the QueueA has 9 extra(209-200) which is less than
sum(queue.children.preemptable) is 10. But the check
{code}
if (Resources.greaterThanOrEqual(
rc, partitionResource, childrensPreemptable, extra)) {
ret.untouchableExtra = Resource.newInstance(0, 0);
{code}
makes the queueA untouchable=0, therefore resulting in correct preemption
Please correct me if i am wrong
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627426#comment-14627426
]
Ajith S commented on YARN-3885:
---
Hi [~xinxianyin] and [~leftnoteasy]
Yes, Xin's scenario makes sense, so preemptable has to be in limits of the
extra. Will attach new patch. thanks for the inputs
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627433#comment-14627433
]
Ajith S commented on YARN-3885:
---
version 05 patch
NAME: queueA CUR: memory:209, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:200, vCores:0 NORM: 1.0 IDEAL_ASSIGNED: memory:201, vCores:1
IDEAL_PREEMPT: memory:8, vCores:-1 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:9, vCores:0*
NAME: queueB CUR: memory:60, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:60, vCores:0 NORM: NaN IDEAL_ASSIGNED: memory:60, vCores:0
IDEAL_PREEMPT: memory:0, vCores:0 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:0, vCores:0*
NAME: queueC CUR: memory:150, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:139, vCores:0 NORM: 1.0 IDEAL_ASSIGNED: memory:141, vCores:1
IDEAL_PREEMPT: memory:9, vCores:-1 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:1, vCores:0 PREEMPTABLE: memory:10, vCores:0*
NAME: queueD CUR: memory:100, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:100, vCores:0 NORM: NaN IDEAL_ASSIGNED: memory:100, vCores:0
IDEAL_PREEMPT: memory:0, vCores:0 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:0, vCores:0*
NAME: queueE CUR: memory:50, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:40, vCores:0 NORM: 1.0 IDEAL_ASSIGNED: memory:41, vCores:1
IDEAL_PREEMPT: memory:9, vCores:-1 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:10, vCores:0*
Also, i took the luxury of using reflection in my testcese (as i couldn't find
better way to assert non-leaf queue resources) :)
let me know if the patch is fine
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627480#comment-14627480
]
Hadoop QA commented on YARN-3885:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 9s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 7m 38s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 24s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 5 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 51m 3s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 87m 41s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12745374/YARN-3885.05.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0a16ee6 |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/8538/artifact/patchprocess/diffJavacWarnings.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8538/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8538/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8538/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8538/console |
This message was automatically generated.
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627404#comment-14627404
]
Xianyin Xin commented on YARN-3885:
---
Agree [~leftnoteasy]. suppose we have a case
parent_of_A
/ \
AA's brother B
and A has 9 extra and 10 preemptable, but B has 9 extra and 0 preemptable (we
can set the queue unpreemptable), then parent_of_A has 18 extra, but how many
preemptable it should offer? It should be 9. so we'd better limit preemptable
to be less than extra for queues at any level.
Please correct me if i'm wrong.
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627527#comment-14627527
]
Wangda Tan commented on YARN-3885:
--
[~ajithshetty],
Thanks for updating, instead of reflection, I suggest to make
getQueueByPartition in Proportional...Policy to be @VisibleForTesting. And
could you take a look at javac warning and test failure?
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.05.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626702#comment-14626702
]
Wangda Tan commented on YARN-3885:
--
[~ajithshetty],
You can see the log:
{code}
NAME: queueA CUR: memory:209, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:200, vCores:0 NORM: 1.0 IDEAL_ASSIGNED: memory:201, vCores:1
IDEAL_PREEMPT: memory:8, vCores:-1 ACTUAL_PREEMPT: memory:0, vCores:0
UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:10, vCores:0
{code}
queuA's extra is 9, but preemptable is 10, this could lead mis computing
preemptable for A's parent queues. (preemptableExtra is used to compute
untouchableExtra, and untouchableExtra is used to compute ideal_allocation). In
your case is fine, but imagining there're some hierarchy above queueA, policy
can preempt containers from a queue more than needed.
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625302#comment-14625302
]
Wangda Tan commented on YARN-3885:
--
Increase to blocker since this is a bad bug.
Thanks for working on this [~ajithshetty].
I think patch looks generally good except one thing also mentioned by
[~xinxianyin]:
bq. for non-leaf queues, min(sum of children's PREEMOTABLEs, extra) -(this
is what CHILDRENPREEMPTABLE does in the patch)
Specifying {{ret.preemptableExtra = childrensPreemptable;}} seems not enough.
Could you add a test to verify: when sum(queue.children.preemptable) extra,
we can still get correct preemption result?
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Blocker
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621823#comment-14621823
]
Ajith S commented on YARN-3885:
---
Whitespace warning is for test case, suggest keep it as its better to explain
the queue arrangement
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Critical
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620329#comment-14620329
]
Hadoop QA commented on YARN-3885:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 41s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 48s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 21s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 51m 7s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 88m 56s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12744423/YARN-3885.04.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 63d0365 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8475/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8475/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8475/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8475/console |
This message was automatically generated.
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Assignee: Ajith S
Priority: Critical
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619989#comment-14619989
]
Ajith S commented on YARN-3885:
---
/root
A
/\
C B
/\
D E
+*Before fix:*+
NAME: queueA CUR: memory:209, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:200, vCores:0 NORM: NaN IDEAL_ASSIGNED: memory:209, vCores:0
IDEAL_PREEMPT: memory:0, vCores:0 ACTUAL_PREEMPT: memory:0, vCores:0
*{color:red}UNTOUCHABLE: memory:9, vCores:0 PREEMPTABLE: memory:0,
vCores:0{color}*
NAME: queueB CUR: memory:60, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:60, vCores:0 NORM: NaN IDEAL_ASSIGNED: memory:60, vCores:0
IDEAL_PREEMPT: memory:0, vCores:0 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:0, vCores:0*
NAME: queueC CUR: memory:150, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:139, vCores:0 NORM: 1.0 IDEAL_ASSIGNED: memory:149, vCores:1
IDEAL_PREEMPT: memory:1, vCores:-1 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:1, vCores:0 PREEMPTABLE: memory:0, vCores:0*
NAME: queueD CUR: memory:100, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:100, vCores:0 NORM: NaN IDEAL_ASSIGNED: memory:100, vCores:0
IDEAL_PREEMPT: memory:0, vCores:0 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:0, vCores:0*
NAME: queueE CUR: memory:50, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:40, vCores:0 NORM: 1.0 IDEAL_ASSIGNED: memory:49, vCores:1
IDEAL_PREEMPT: memory:1, vCores:-1 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:10, vCores:0*
+*After:*+
NAME: queueA CUR: memory:209, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:200, vCores:0 NORM: 1.0 IDEAL_ASSIGNED: memory:201, vCores:1
IDEAL_PREEMPT: memory:8, vCores:-1 ACTUAL_PREEMPT: memory:0, vCores:0
*{color:green}UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:10,
vCores:0{color}*
NAME: queueB CUR: memory:60, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:60, vCores:0 NORM: NaN IDEAL_ASSIGNED: memory:60, vCores:0
IDEAL_PREEMPT: memory:0, vCores:0 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:0, vCores:0*
NAME: queueC CUR: memory:150, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:139, vCores:0 NORM: 1.0 IDEAL_ASSIGNED: memory:141, vCores:1
IDEAL_PREEMPT: memory:9, vCores:-1 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:1, vCores:0 PREEMPTABLE: memory:10, vCores:0*
NAME: queueD CUR: memory:100, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:100, vCores:0 NORM: NaN IDEAL_ASSIGNED: memory:100, vCores:0
IDEAL_PREEMPT: memory:0, vCores:0 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:0, vCores:0*
NAME: queueE CUR: memory:50, vCores:0 PEN: memory:0, vCores:0 GAR:
memory:40, vCores:0 NORM: 1.0 IDEAL_ASSIGNED: memory:41, vCores:1
IDEAL_PREEMPT: memory:9, vCores:-1 ACTUAL_PREEMPT: memory:0, vCores:0
*UNTOUCHABLE: memory:0, vCores:0 PREEMPTABLE: memory:10, vCores:0*
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Priority: Critical
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619971#comment-14619971
]
Ajith S commented on YARN-3885:
---
Hi [~sunilg]
Sorry for the delay, i have added the testcase
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Priority: Critical
Attachments: YARN-3885.02.patch, YARN-3885.03.patch,
YARN-3885.04.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615180#comment-14615180
]
Sunil G commented on YARN-3885:
---
HI [~ajithshetty]
Thanks for reporting this.
As you mentioned, this issue will be coming from levels more than 2. Your patch
looks good.
Could you please add a test case for this scenario.
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Priority: Critical
Attachments: YARN-3885.02.patch, YARN-3885.03.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614182#comment-14614182
]
Ajith S commented on YARN-3885:
---
Hi all,
I already have a patch for it, please assign this to me
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Priority: Critical
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} is wrong as it doesnt
consider al the children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614194#comment-14614194
]
Xianyin Xin commented on YARN-3885:
---
Hi Ajith, CHILDRENPREEMPTABLE may be over counted. Suppose parent queue root
has extra 50 containers, and child queue root.A has 20 preemptable, and root.B
40 preemptable, than root has 60 preemptable by calculation. This should be
limited to 50.
An alternative solution is we use PREEMTABLE of non-leaf queue record the sum
of sub-queue's PREEMPTABLEs(i.e. PREEMPTABL=min(CHILDRENPREEMPTABLE, extra))
because it is non zero only for leaf queue, thus we can omit
CHILDRENPREEMPTABLE in the TempQueuePerPartition. It's strange that all the
non-leaf queues has 0 PREEMPTABLE.
Def PREEMPTABLE:
for leaf queues, the preemptable resources
for non-leaf queues, min(sum of children's PREEMOTABLEs, extra) -(this
is what CHILDRENPREEMPTABLE does in the patch)
Just a suggestion.
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Priority: Critical
Attachments: YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} is wrong as it doesnt
consider al the children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3885) ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level
[
https://issues.apache.org/jira/browse/YARN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614505#comment-14614505
]
Ajith S commented on YARN-3885:
---
Hello xin,
Thanks for the input, yes you are right, i will refactor the patch
ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2
level
--
Key: YARN-3885
URL: https://issues.apache.org/jira/browse/YARN-3885
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Affects Versions: 2.8.0
Reporter: Ajith S
Priority: Critical
Attachments: YARN-3885.02.patch, YARN-3885.patch
when preemption policy is {{ProportionalCapacityPreemptionPolicy.cloneQueues}}
this piece of code, to calculate {{untoucable}} doesnt consider al the
children, it considers only immediate childern
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
