[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[ https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351519#comment-14351519 ] Hudson commented on YARN-3275: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #125 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/125/]) YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne (jlowe: rev 27e8ea820fab8dce59f4db9814e73bd60c1d4ef1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/CHANGES.txt CapacityScheduler: Preemption happening on non-preemptable queues - Key: YARN-3275 URL: https://issues.apache.org/jira/browse/YARN-3275 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.0 Reporter: Eric Payne Assignee: Eric Payne Labels: capacity-scheduler Fix For: 2.7.0 Attachments: YARN-3275.v1.txt, YARN-3275.v2.txt YARN-2056 introduced the ability to turn preemption on and off at the queue level. In cases where a queue goes over its absolute max capacity (YARN-3243, for example), containers can be preempted from that queue, even though the queue is marked as non-preemptable. We are using this feature in large, busy clusters and seeing this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[ https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351639#comment-14351639 ] Hudson commented on YARN-3275: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2075 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2075/]) YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne (jlowe: rev 27e8ea820fab8dce59f4db9814e73bd60c1d4ef1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java CapacityScheduler: Preemption happening on non-preemptable queues - Key: YARN-3275 URL: https://issues.apache.org/jira/browse/YARN-3275 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.0 Reporter: Eric Payne Assignee: Eric Payne Labels: capacity-scheduler Fix For: 2.7.0 Attachments: YARN-3275.v1.txt, YARN-3275.v2.txt YARN-2056 introduced the ability to turn preemption on and off at the queue level. In cases where a queue goes over its absolute max capacity (YARN-3243, for example), containers can be preempted from that queue, even though the queue is marked as non-preemptable. We are using this feature in large, busy clusters and seeing this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[ https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351608#comment-14351608 ] Hudson commented on YARN-3275: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #116 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/116/]) YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne (jlowe: rev 27e8ea820fab8dce59f4db9814e73bd60c1d4ef1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java CapacityScheduler: Preemption happening on non-preemptable queues - Key: YARN-3275 URL: https://issues.apache.org/jira/browse/YARN-3275 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.0 Reporter: Eric Payne Assignee: Eric Payne Labels: capacity-scheduler Fix For: 2.7.0 Attachments: YARN-3275.v1.txt, YARN-3275.v2.txt YARN-2056 introduced the ability to turn preemption on and off at the queue level. In cases where a queue goes over its absolute max capacity (YARN-3243, for example), containers can be preempted from that queue, even though the queue is marked as non-preemptable. We are using this feature in large, busy clusters and seeing this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[ https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351597#comment-14351597 ] Hudson commented on YARN-3275: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2057 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2057/]) YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne (jlowe: rev 27e8ea820fab8dce59f4db9814e73bd60c1d4ef1) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java CapacityScheduler: Preemption happening on non-preemptable queues - Key: YARN-3275 URL: https://issues.apache.org/jira/browse/YARN-3275 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.0 Reporter: Eric Payne Assignee: Eric Payne Labels: capacity-scheduler Fix For: 2.7.0 Attachments: YARN-3275.v1.txt, YARN-3275.v2.txt YARN-2056 introduced the ability to turn preemption on and off at the queue level. In cases where a queue goes over its absolute max capacity (YARN-3243, for example), containers can be preempted from that queue, even though the queue is marked as non-preemptable. We are using this feature in large, busy clusters and seeing this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[ https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351621#comment-14351621 ] Hudson commented on YARN-3275: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #125 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/125/]) YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne (jlowe: rev 27e8ea820fab8dce59f4db9814e73bd60c1d4ef1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java CapacityScheduler: Preemption happening on non-preemptable queues - Key: YARN-3275 URL: https://issues.apache.org/jira/browse/YARN-3275 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.0 Reporter: Eric Payne Assignee: Eric Payne Labels: capacity-scheduler Fix For: 2.7.0 Attachments: YARN-3275.v1.txt, YARN-3275.v2.txt YARN-2056 introduced the ability to turn preemption on and off at the queue level. In cases where a queue goes over its absolute max capacity (YARN-3243, for example), containers can be preempted from that queue, even though the queue is marked as non-preemptable. We are using this feature in large, busy clusters and seeing this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[ https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351527#comment-14351527 ] Hudson commented on YARN-3275: -- FAILURE: Integrated in Hadoop-Yarn-trunk #859 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/859/]) YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne (jlowe: rev 27e8ea820fab8dce59f4db9814e73bd60c1d4ef1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * hadoop-yarn-project/CHANGES.txt CapacityScheduler: Preemption happening on non-preemptable queues - Key: YARN-3275 URL: https://issues.apache.org/jira/browse/YARN-3275 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.0 Reporter: Eric Payne Assignee: Eric Payne Labels: capacity-scheduler Fix For: 2.7.0 Attachments: YARN-3275.v1.txt, YARN-3275.v2.txt YARN-2056 introduced the ability to turn preemption on and off at the queue level. In cases where a queue goes over its absolute max capacity (YARN-3243, for example), containers can be preempted from that queue, even though the queue is marked as non-preemptable. We are using this feature in large, busy clusters and seeing this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[ https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351083#comment-14351083 ] Hudson commented on YARN-3275: -- FAILURE: Integrated in Hadoop-trunk-Commit #7276 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7276/]) YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne (jlowe: rev 27e8ea820fab8dce59f4db9814e73bd60c1d4ef1) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java CapacityScheduler: Preemption happening on non-preemptable queues - Key: YARN-3275 URL: https://issues.apache.org/jira/browse/YARN-3275 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.0 Reporter: Eric Payne Assignee: Eric Payne Labels: capacity-scheduler Fix For: 2.7.0 Attachments: YARN-3275.v1.txt, YARN-3275.v2.txt YARN-2056 introduced the ability to turn preemption on and off at the queue level. In cases where a queue goes over its absolute max capacity (YARN-3243, for example), containers can be preempted from that queue, even though the queue is marked as non-preemptable. We are using this feature in large, busy clusters and seeing this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[ https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349540#comment-14349540 ] Jason Lowe commented on YARN-3275: -- +1 lgtm. Findbug warnings appear to be unrelated. Will commit this tomorrow if there are no further comments or objections. CapacityScheduler: Preemption happening on non-preemptable queues - Key: YARN-3275 URL: https://issues.apache.org/jira/browse/YARN-3275 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.0 Reporter: Eric Payne Assignee: Eric Payne Labels: capacity-scheduler Attachments: YARN-3275.v1.txt, YARN-3275.v2.txt YARN-2056 introduced the ability to turn preemption on and off at the queue level. In cases where a queue goes over its absolute max capacity (YARN-3243, for example), containers can be preempted from that queue, even though the queue is marked as non-preemptable. We are using this feature in large, busy clusters and seeing this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[
https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347803#comment-14347803
]
Hadoop QA commented on YARN-3275:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702623/YARN-3275.v2.txt
against trunk revision ed70fa1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6850//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6850//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6850//console
This message is automatically generated.
CapacityScheduler: Preemption happening on non-preemptable queues
-
Key: YARN-3275
URL: https://issues.apache.org/jira/browse/YARN-3275
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Eric Payne
Assignee: Eric Payne
Labels: capacity-scheduler
Attachments: YARN-3275.v1.txt, YARN-3275.v2.txt
YARN-2056 introduced the ability to turn preemption on and off at the queue
level. In cases where a queue goes over its absolute max capacity (YARN-3243,
for example), containers can be preempted from that queue, even though the
queue is marked as non-preemptable.
We are using this feature in large, busy clusters and seeing this behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3275) CapacityScheduler: Preemption happening on non-preemptable queues
[
https://issues.apache.org/jira/browse/YARN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345290#comment-14345290
]
Jason Lowe commented on YARN-3275:
--
Thanks for the patch, Eric!
bq. the expectation of our users is that if they are running a job on a
non-preemptable queue, their containers should never be preempted.
I completely agree with this. IMHO the whole point of the preemption disable
feature is to guarantee a queue marked as such will never be preempted. It's
as if the entire preemption feature was turned off from that queue's
perspective.
Looking at the patch, I'm a bit worried about this part:
{code}
+ Resource absMaxCapIdealAssignedDelta = Resource.newInstance(0, 0);
+ if (Resources.greaterThanOrEqual(
+rc, clusterResource, maxCapacity, idealAssigned)) {
+absMaxCapIdealAssignedDelta = Resources.subtract(maxCapacity,
idealAssigned);
+ }
{code}
If the intent of this calculation is to guarantee none of the components of
absMaxCapIdealAssignedDelta are negative then I don't believe this accomplishes
that goal. I believe it's possible for Resources.greaterThanOrEqual to return
true yet subtracting the values will result in one of the components to be
negative. For example, what if both resources are memory dominant, maxCapacity
has more memory than idealAssigned, but the opposite is true for vcores?
Subtracting idealAssigned from maxCapacity will result in a positive memory
component but a negative vcore component. If we need to make sure neither
component goes negative then I think we need to do a component-wise max with
the zero resource rather than a comparision.
Also one style nit: we normally don't do one-liner conditionals without braces,
so I'd like to see the continue explicitly put in a block. It might be useful
to put a debug log statement with the continue to note that we wanted to
preempt this queue for some reason (and by how much) but it was marked with
preemption disabled.
CapacityScheduler: Preemption happening on non-preemptable queues
-
Key: YARN-3275
URL: https://issues.apache.org/jira/browse/YARN-3275
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Eric Payne
Assignee: Eric Payne
Labels: capacity-scheduler
Attachments: YARN-3275.v1.txt
YARN-2056 introduced the ability to turn preemption on and off at the queue
level. In cases where a queue goes over its absolute max capacity (YARN-3243,
for example), containers can be preempted from that queue, even though the
queue is marked as non-preemptable.
We are using this feature in large, busy clusters and seeing this behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
