[jira] [Commented] (YARN-2073) Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free
[
https://issues.apache.org/jira/browse/YARN-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14027644#comment-14027644
]
Andrey Stepachev commented on YARN-2073:
I read code and don't cleanly understand why inequality uses Available
resources instead of Allocated?
With current code preemption performed only when cluster utilised under
20percent.
Suppose we have occupied 8GB of 10GB. And default preemption threshold == 0.8f.
From description I thought that preemption should occur, but it doesn't.
Code below gives us:
{code}
0.8 availableMb / maxMem
0.8 (10-8) / 10
0.8 0.2
false
{code}
{code}
return (preemptionUtilizationThreshold Math.max(
(float) rootMetrics.getAvailableMB() / clusterResource.getMemory(),
(float) rootMetrics.getAvailableVirtualCores() /
clusterResource.getVirtualCores()));
}
{code}
Is that was an intention for this parameter (and it should be documented for
non native speakers to be
more clear, say: preemption will not performed if utilisation is above this
parameter) or here was a mistake?
Fair Scheduler: Add a utilization threshold to prevent preempting resources
when cluster is free
Key: YARN-2073
URL: https://issues.apache.org/jira/browse/YARN-2073
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.4.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Critical
Fix For: 2.5.0
Attachments: yarn-2073-0.patch, yarn-2073-1.patch, yarn-2073-2.patch,
yarn-2073-3.patch, yarn-2073-4.patch
Preemption should kick in only when the currently available slots don't match
the request.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2073) Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free
[ https://issues.apache.org/jira/browse/YARN-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028256#comment-14028256 ] Wei Yan commented on YARN-2073: --- should be a mistake here. need to use allocated resource. Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free Key: YARN-2073 URL: https://issues.apache.org/jira/browse/YARN-2073 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Fix For: 2.5.0 Attachments: yarn-2073-0.patch, yarn-2073-1.patch, yarn-2073-2.patch, yarn-2073-3.patch, yarn-2073-4.patch Preemption should kick in only when the currently available slots don't match the request. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2073) Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free
[ https://issues.apache.org/jira/browse/YARN-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008757#comment-14008757 ] Hudson commented on YARN-2073: -- FAILURE: Integrated in Hadoop-Yarn-trunk #564 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/564/]) YARN-2073. Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free (Karthik Kambatla via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1597209) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free Key: YARN-2073 URL: https://issues.apache.org/jira/browse/YARN-2073 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Fix For: 2.5.0 Attachments: yarn-2073-0.patch, yarn-2073-1.patch, yarn-2073-2.patch, yarn-2073-3.patch, yarn-2073-4.patch Preemption should kick in only when the currently available slots don't match the request. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2073) Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free
[ https://issues.apache.org/jira/browse/YARN-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008870#comment-14008870 ] Hudson commented on YARN-2073: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1756 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1756/]) YARN-2073. Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free (Karthik Kambatla via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1597209) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free Key: YARN-2073 URL: https://issues.apache.org/jira/browse/YARN-2073 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Fix For: 2.5.0 Attachments: yarn-2073-0.patch, yarn-2073-1.patch, yarn-2073-2.patch, yarn-2073-3.patch, yarn-2073-4.patch Preemption should kick in only when the currently available slots don't match the request. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2073) Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free
[ https://issues.apache.org/jira/browse/YARN-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008914#comment-14008914 ] Hudson commented on YARN-2073: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1782 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1782/]) YARN-2073. Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free (Karthik Kambatla via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1597209) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free Key: YARN-2073 URL: https://issues.apache.org/jira/browse/YARN-2073 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.4.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical Fix For: 2.5.0 Attachments: yarn-2073-0.patch, yarn-2073-1.patch, yarn-2073-2.patch, yarn-2073-3.patch, yarn-2073-4.patch Preemption should kick in only when the currently available slots don't match the request. -- This message was sent by Atlassian JIRA (v6.2#6252)
