[jira] [Commented] (YARN-2214) preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness
[ https://issues.apache.org/jira/browse/YARN-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074509#comment-14074509 ] Karthik Kambatla commented on YARN-2214: +1. Checking this in. > preemptContainerPreCheck() in FSParentQueue delays convergence towards > fairness > --- > > Key: YARN-2214 > URL: https://issues.apache.org/jira/browse/YARN-2214 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar >Assignee: Ashwin Shankar > Attachments: YARN-2214-v1.txt, YARN-2214-v2.txt > > > preemptContainerPreCheck() in FSParentQueue rejects preemption requests if > the parent queue is below fair share. This can cause a delay in converging > towards fairness when the starved leaf queue and the queue above fairshare > belong under a non-root parent queue(ie their least common ancestor is a > parent queue which is not root). > Here is an example : > root.parent has fair share = 80% and usage = 80% > root.parent.child1 has fair share =40% usage = 80% > root.parent.child2 has fair share=40% usage=0% > Now a job is submitted to child2 and the demand is 40%. > Preemption will kick in and try to reclaim all the 40% from child1. > When it preempts the first container from child1,the usage of root.parent > will become <80%, which is less than root.parent's fair share,causing > preemption to stop.So only one container gets preempted in this round > although the need is a lot more. child2 would eventually get to half its fair > share but only after multiple rounds of preemption. > Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it > only in FSLeafQueue(which is already there). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2214) preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness
[
https://issues.apache.org/jira/browse/YARN-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074073#comment-14074073
]
Hadoop QA commented on YARN-2214:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12657768/YARN-2214-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4426//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4426//console
This message is automatically generated.
> preemptContainerPreCheck() in FSParentQueue delays convergence towards
> fairness
> ---
>
> Key: YARN-2214
> URL: https://issues.apache.org/jira/browse/YARN-2214
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.5.0
>Reporter: Ashwin Shankar
>Assignee: Ashwin Shankar
> Attachments: YARN-2214-v1.txt, YARN-2214-v2.txt
>
>
> preemptContainerPreCheck() in FSParentQueue rejects preemption requests if
> the parent queue is below fair share. This can cause a delay in converging
> towards fairness when the starved leaf queue and the queue above fairshare
> belong under a non-root parent queue(ie their least common ancestor is a
> parent queue which is not root).
> Here is an example :
> root.parent has fair share = 80% and usage = 80%
> root.parent.child1 has fair share =40% usage = 80%
> root.parent.child2 has fair share=40% usage=0%
> Now a job is submitted to child2 and the demand is 40%.
> Preemption will kick in and try to reclaim all the 40% from child1.
> When it preempts the first container from child1,the usage of root.parent
> will become <80%, which is less than root.parent's fair share,causing
> preemption to stop.So only one container gets preempted in this round
> although the need is a lot more. child2 would eventually get to half its fair
> share but only after multiple rounds of preemption.
> Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it
> only in FSLeafQueue(which is already there).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2214) preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness
[ https://issues.apache.org/jira/browse/YARN-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074039#comment-14074039 ] Ashwin Shankar commented on YARN-2214: -- Thanks [~kasha] ! Patch refreshed. > preemptContainerPreCheck() in FSParentQueue delays convergence towards > fairness > --- > > Key: YARN-2214 > URL: https://issues.apache.org/jira/browse/YARN-2214 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar >Assignee: Ashwin Shankar > Attachments: YARN-2214-v1.txt, YARN-2214-v2.txt > > > preemptContainerPreCheck() in FSParentQueue rejects preemption requests if > the parent queue is below fair share. This can cause a delay in converging > towards fairness when the starved leaf queue and the queue above fairshare > belong under a non-root parent queue(ie their least common ancestor is a > parent queue which is not root). > Here is an example : > root.parent has fair share = 80% and usage = 80% > root.parent.child1 has fair share =40% usage = 80% > root.parent.child2 has fair share=40% usage=0% > Now a job is submitted to child2 and the demand is 40%. > Preemption will kick in and try to reclaim all the 40% from child1. > When it preempts the first container from child1,the usage of root.parent > will become <80%, which is less than root.parent's fair share,causing > preemption to stop.So only one container gets preempted in this round > although the need is a lot more. child2 would eventually get to half its fair > share but only after multiple rounds of preemption. > Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it > only in FSLeafQueue(which is already there). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2214) preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness
[ https://issues.apache.org/jira/browse/YARN-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073921#comment-14073921 ] Karthik Kambatla commented on YARN-2214: Patch looks mostly good. One nit: can we move the check in FSLeafQueue#preemptContainer before the debug logging. > preemptContainerPreCheck() in FSParentQueue delays convergence towards > fairness > --- > > Key: YARN-2214 > URL: https://issues.apache.org/jira/browse/YARN-2214 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar >Assignee: Ashwin Shankar > Attachments: YARN-2214-v1.txt > > > preemptContainerPreCheck() in FSParentQueue rejects preemption requests if > the parent queue is below fair share. This can cause a delay in converging > towards fairness when the starved leaf queue and the queue above fairshare > belong under a non-root parent queue(ie their least common ancestor is a > parent queue which is not root). > Here is an example : > root.parent has fair share = 80% and usage = 80% > root.parent.child1 has fair share =40% usage = 80% > root.parent.child2 has fair share=40% usage=0% > Now a job is submitted to child2 and the demand is 40%. > Preemption will kick in and try to reclaim all the 40% from child1. > When it preempts the first container from child1,the usage of root.parent > will become <80%, which is less than root.parent's fair share,causing > preemption to stop.So only one container gets preempted in this round > although the need is a lot more. child2 would eventually get to half its fair > share but only after multiple rounds of preemption. > Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it > only in FSLeafQueue(which is already there). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2214) preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness
[ https://issues.apache.org/jira/browse/YARN-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072409#comment-14072409 ] Ashwin Shankar commented on YARN-2214: -- [~kasha], [~sandyr] Can one of you please look at this one ? Thanks in advance ! > preemptContainerPreCheck() in FSParentQueue delays convergence towards > fairness > --- > > Key: YARN-2214 > URL: https://issues.apache.org/jira/browse/YARN-2214 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar >Assignee: Ashwin Shankar > Attachments: YARN-2214-v1.txt > > > preemptContainerPreCheck() in FSParentQueue rejects preemption requests if > the parent queue is below fair share. This can cause a delay in converging > towards fairness when the starved leaf queue and the queue above fairshare > belong under a non-root parent queue(ie their least common ancestor is a > parent queue which is not root). > Here is an example : > root.parent has fair share = 80% and usage = 80% > root.parent.child1 has fair share =40% usage = 80% > root.parent.child2 has fair share=40% usage=0% > Now a job is submitted to child2 and the demand is 40%. > Preemption will kick in and try to reclaim all the 40% from child1. > When it preempts the first container from child1,the usage of root.parent > will become <80%, which is less than root.parent's fair share,causing > preemption to stop.So only one container gets preempted in this round > although the need is a lot more. child2 would eventually get to half its fair > share but only after multiple rounds of preemption. > Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it > only in FSLeafQueue(which is already there). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2214) preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness
[ https://issues.apache.org/jira/browse/YARN-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044964#comment-14044964 ] Ashwin Shankar commented on YARN-2214: -- test failure not related to this patch. There is a jira for it YARN-2216 > preemptContainerPreCheck() in FSParentQueue delays convergence towards > fairness > --- > > Key: YARN-2214 > URL: https://issues.apache.org/jira/browse/YARN-2214 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar >Assignee: Ashwin Shankar > Attachments: YARN-2214-v1.txt > > > preemptContainerPreCheck() in FSParentQueue rejects preemption requests if > the parent queue is below fair share. This can cause a delay in converging > towards fairness when the starved leaf queue and the queue above fairshare > belong under a non-root parent queue(ie their least common ancestor is a > parent queue which is not root). > Here is an example : > root.parent has fair share = 80% and usage = 80% > root.parent.child1 has fair share =40% usage = 80% > root.parent.child2 has fair share=40% usage=0% > Now a job is submitted to child2 and the demand is 40%. > Preemption will kick in and try to reclaim all the 40% from child1. > When it preempts the first container from child1,the usage of root.parent > will become <80%, which is less than root.parent's fair share,causing > preemption to stop.So only one container gets preempted in this round > although the need is a lot more. child2 would eventually get to half its fair > share but only after multiple rounds of preemption. > Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it > only in FSLeafQueue(which is already there). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2214) preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness
[
https://issues.apache.org/jira/browse/YARN-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044952#comment-14044952
]
Hadoop QA commented on YARN-2214:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652647/YARN-2214-v1.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4095//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4095//console
This message is automatically generated.
> preemptContainerPreCheck() in FSParentQueue delays convergence towards
> fairness
> ---
>
> Key: YARN-2214
> URL: https://issues.apache.org/jira/browse/YARN-2214
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 2.5.0
>Reporter: Ashwin Shankar
>Assignee: Ashwin Shankar
> Attachments: YARN-2214-v1.txt
>
>
> preemptContainerPreCheck() in FSParentQueue rejects preemption requests if
> the parent queue is below fair share. This can cause a delay in converging
> towards fairness when the starved leaf queue and the queue above fairshare
> belong under a non-root parent queue(ie their least common ancestor is a
> parent queue which is not root).
> Here is an example :
> root.parent has fair share = 80% and usage = 80%
> root.parent.child1 has fair share =40% usage = 80%
> root.parent.child2 has fair share=40% usage=0%
> Now a job is submitted to child2 and the demand is 40%.
> Preemption will kick in and try to reclaim all the 40% from child1.
> When it preempts the first container from child1,the usage of root.parent
> will become <80%, which is less than root.parent's fair share,causing
> preemption to stop.So only one container gets preempted in this round
> although the need is a lot more. child2 would eventually get to half its fair
> share but only after multiple rounds of preemption.
> Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it
> only in FSLeafQueue(which is already there).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2214) preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness
[ https://issues.apache.org/jira/browse/YARN-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044066#comment-14044066 ] Sandy Ryza commented on YARN-2214: -- Makes sense > preemptContainerPreCheck() in FSParentQueue delays convergence towards > fairness > --- > > Key: YARN-2214 > URL: https://issues.apache.org/jira/browse/YARN-2214 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar > > preemptContainerPreCheck() in FSParentQueue rejects preemption requests if > the parent queue is below fair share. This can cause a delay in converging > towards fairness when the starved leaf queue and the queue above fairshare > belong under a non-root parent queue(ie their least common ancestor is a > parent queue which is not root). > Here is an example : > root.parent has fair share = 80% and usage = 80% > root.parent.child1 has fair share =40% usage = 80% > root.parent.child2 has fair share=40% usage=0% > Now a job is submitted to child2 and the demand is 40%. > Preemption will kick in and try to reclaim all the 40% from child1. > When it preempts the first container from child1,the usage of root.parent > will become <80%, which is less than root.parent's fair share,causing > preemption to stop.So only one container gets preempted in this round > although the need is a lot more. child2 would eventually get to half its fair > share but only after multiple rounds of preemption. > Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it > only in FSLeafQueue(which is already there). -- This message was sent by Atlassian JIRA (v6.2#6252)
