[jira] [Commented] (YARN-1957) ProportionalCapacitPreemptionPolicy handling of corner cases...
[ https://issues.apache.org/jira/browse/YARN-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999005#comment-13999005 ] Hudson commented on YARN-1957: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5605 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5605/]) YARN-1957. Consider the max capacity of the queue when computing the ideal capacity for preemption. Contributed by Carlo Curino (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594414) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java > ProportionalCapacitPreemptionPolicy handling of corner cases... > --- > > Key: YARN-1957 > URL: https://issues.apache.org/jira/browse/YARN-1957 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Labels: capacity-scheduler, preemption > Fix For: 3.0.0, 2.5.0, 2.4.1 > > Attachments: YARN-1957.patch, YARN-1957.patch, YARN-1957_test.patch > > > The current version of ProportionalCapacityPreemptionPolicy should be > improved to deal with the following two scenarios: > 1) when rebalancing over-capacity allocations, it potentially preempts > without considering the maxCapacity constraints of a queue (i.e., preempting > possibly more than strictly necessary) > 2) a zero capacity queue is preempted even if there is no demand (coherent > with old use of zero-capacity to disabled queues) > The proposed patch fixes both issues, and introduce few new test cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1957) ProportionalCapacitPreemptionPolicy handling of corner cases...
[ https://issues.apache.org/jira/browse/YARN-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998174#comment-13998174 ] Hudson commented on YARN-1957: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1779 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1779/]) YARN-1957. Consider the max capacity of the queue when computing the ideal capacity for preemption. Contributed by Carlo Curino (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594414) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java > ProportionalCapacitPreemptionPolicy handling of corner cases... > --- > > Key: YARN-1957 > URL: https://issues.apache.org/jira/browse/YARN-1957 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Labels: capacity-scheduler, preemption > Fix For: 3.0.0, 2.5.0, 2.4.1 > > Attachments: YARN-1957.patch, YARN-1957.patch, YARN-1957_test.patch > > > The current version of ProportionalCapacityPreemptionPolicy should be > improved to deal with the following two scenarios: > 1) when rebalancing over-capacity allocations, it potentially preempts > without considering the maxCapacity constraints of a queue (i.e., preempting > possibly more than strictly necessary) > 2) a zero capacity queue is preempted even if there is no demand (coherent > with old use of zero-capacity to disabled queues) > The proposed patch fixes both issues, and introduce few new test cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1957) ProportionalCapacitPreemptionPolicy handling of corner cases...
[ https://issues.apache.org/jira/browse/YARN-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998212#comment-13998212 ] Hudson commented on YARN-1957: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1753 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1753/]) YARN-1957. Consider the max capacity of the queue when computing the ideal capacity for preemption. Contributed by Carlo Curino (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594414) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java > ProportionalCapacitPreemptionPolicy handling of corner cases... > --- > > Key: YARN-1957 > URL: https://issues.apache.org/jira/browse/YARN-1957 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Labels: capacity-scheduler, preemption > Fix For: 3.0.0, 2.5.0, 2.4.1 > > Attachments: YARN-1957.patch, YARN-1957.patch, YARN-1957_test.patch > > > The current version of ProportionalCapacityPreemptionPolicy should be > improved to deal with the following two scenarios: > 1) when rebalancing over-capacity allocations, it potentially preempts > without considering the maxCapacity constraints of a queue (i.e., preempting > possibly more than strictly necessary) > 2) a zero capacity queue is preempted even if there is no demand (coherent > with old use of zero-capacity to disabled queues) > The proposed patch fixes both issues, and introduce few new test cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1957) ProportionalCapacitPreemptionPolicy handling of corner cases...
[ https://issues.apache.org/jira/browse/YARN-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983978#comment-13983978 ] Chris Douglas commented on YARN-1957: - +1 Enforcing {{maxCapacity}} in the calculation of the ideal capacity is a good fix, and distributing capacity over queues with zero capacity (with the config knob to restore the existing 0 == disabled with aggressive preemption) makes sense. The code appears to effect this, also. There's a slight optimization that can separate the zero-capacity queues during cloning, but the overhead is negligible. > ProportionalCapacitPreemptionPolicy handling of corner cases... > --- > > Key: YARN-1957 > URL: https://issues.apache.org/jira/browse/YARN-1957 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Labels: capacity-scheduler, preemption > Attachments: YARN-1957.patch, YARN-1957.patch, YARN-1957_test.patch > > > The current version of ProportionalCapacityPreemptionPolicy should be > improved to deal with the following two scenarios: > 1) when rebalancing over-capacity allocations, it potentially preempts > without considering the maxCapacity constraints of a queue (i.e., preempting > possibly more than strictly necessary) > 2) a zero capacity queue is preempted even if there is no demand (coherent > with old use of zero-capacity to disabled queues) > The proposed patch fixes both issues, and introduce few new test cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1957) ProportionalCapacitPreemptionPolicy handling of corner cases...
[ https://issues.apache.org/jira/browse/YARN-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983943#comment-13983943 ] Hadoop QA commented on YARN-1957: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642329/YARN-1957.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3650//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3650//console This message is automatically generated. > ProportionalCapacitPreemptionPolicy handling of corner cases... > --- > > Key: YARN-1957 > URL: https://issues.apache.org/jira/browse/YARN-1957 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Labels: capacity-scheduler, preemption > Attachments: YARN-1957.patch, YARN-1957.patch, YARN-1957_test.patch > > > The current version of ProportionalCapacityPreemptionPolicy should be > improved to deal with the following two scenarios: > 1) when rebalancing over-capacity allocations, it potentially preempts > without considering the maxCapacity constraints of a queue (i.e., preempting > possibly more than strictly necessary) > 2) a zero capacity queue is preempted even if there is no demand (coherent > with old use of zero-capacity to disabled queues) > The proposed patch fixes both issues, and introduce few new test cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1957) ProportionalCapacitPreemptionPolicy handling of corner cases...
[ https://issues.apache.org/jira/browse/YARN-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983664#comment-13983664 ] Hadoop QA commented on YARN-1957: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642329/YARN-1957.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3642//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3642//console This message is automatically generated. > ProportionalCapacitPreemptionPolicy handling of corner cases... > --- > > Key: YARN-1957 > URL: https://issues.apache.org/jira/browse/YARN-1957 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Labels: capacity-scheduler, preemption > Attachments: YARN-1957.patch, YARN-1957.patch, YARN-1957_test.patch > > > The current version of ProportionalCapacityPreemptionPolicy should be > improved to deal with the following two scenarios: > 1) when rebalancing over-capacity allocations, it potentially preempts > without considering the maxCapacity constraints of a queue (i.e., preempting > possibly more than strictly necessary) > 2) a zero capacity queue is preempted even if there is no demand (coherent > with old use of zero-capacity to disabled queues) > The proposed patch fixes both issues, and introduce few new test cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1957) ProportionalCapacitPreemptionPolicy handling of corner cases...
[ https://issues.apache.org/jira/browse/YARN-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979295#comment-13979295 ] Carlo Curino commented on YARN-1957: Attached a patch fixing the issues above. Conceptually we compute the fix-point allocation for non-zero queues, and then spread the remaining capacity uniformly among zero-size queues. Also each "offer" to a queue now handles maxCapacity. The old behavior for zero-size queues (i.e., preemption of all containers regardless to external pressure) should be possible to obtain by setting maxCapacity = capacity = 0; Test have been updated to verify maxCapacity behavior as well. We also tested on a medium size cluster running gridmix with lots of preemptions and it seems to behave as expected, however a careful code review from someone else is more than welcome. > ProportionalCapacitPreemptionPolicy handling of corner cases... > --- > > Key: YARN-1957 > URL: https://issues.apache.org/jira/browse/YARN-1957 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-1957.patch, YARN-1957_test.patch > > > The current version of ProportionalCapacityPreemptionPolicy should be > improved to deal with the following two scenarios: > 1) when rebalancing over-capacity allocations, it potentially preempts > without considering the maxCapacity constraints of a queue (i.e., preempting > possibly more than strictly necessary) > 2) a zero capacity queue is preempted even if there is no demand (coherent > with old use of zero-capacity to disabled queues) > The proposed patch fixes both issues, and introduce few new test cases. -- This message was sent by Atlassian JIRA (v6.2#6252)