[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011540#comment-14011540 ] Sandy Ryza commented on YARN-596: - +1 In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011542#comment-14011542 ] Sandy Ryza commented on YARN-596: - (pending Jenkins) In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011807#comment-14011807
]
Ashwin Shankar commented on YARN-596:
-
[~ywskycn],minor comment : can you please update javadoc comment for
{code:title=FairScheduler.java}
protected void preemptResources(Resource toPreempt)
{code}
it still talks about the previous preemption algorithm.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011814#comment-14011814 ] Wei Yan commented on YARN-596: -- Thanks, [~ashwinshankar77]. I'll update a patch quickly. In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011906#comment-14011906
]
Hadoop QA commented on YARN-596:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647256/YARN-596.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3852//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3852//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012030#comment-14012030 ] Sandy Ryza commented on YARN-596: - +1 In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010643#comment-14010643
]
Hadoop QA commented on YARN-596:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647026/YARN-596.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3841//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3841//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010791#comment-14010791
]
Sandy Ryza commented on YARN-596:
-
Thanks Wei. Getting close - a few more comments.
{code}
+ private static final ResourceCalculator RESOURCE_CALCULATOR =
+ new DefaultResourceCalculator();
{code}
This is no longer needed in FSQueue, right?
FIFOPolicy should throw an unsupported operation exception if its
checkIfUsageOverFairShare is called.
fairshare should be fairShare
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007592#comment-14007592
]
Hadoop QA commented on YARN-596:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12646537/YARN-596.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3798//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007689#comment-14007689
]
Hadoop QA commented on YARN-596:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12646537/YARN-596.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3806//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007927#comment-14007927
]
Hadoop QA commented on YARN-596:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12646617/YARN-596.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3818//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006248#comment-14006248 ] Wei Yan commented on YARN-596: -- Hey, [~sandyr], sorry for the late reply. Still confuse here. So as you said, a queue is safe and doesn't allow preemption only it satisfies the condition (usage.memory = fairshare.memory) (usage.vcores = fairshare.vcores). This condition works fine for DRF. But for FairSharePolicy, as the fairshare.vcores is always 0 (except for root), so this condition cannot be satisfied and all queues are always allowed to preempt. In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006609#comment-14006609 ] Sandy Ryza commented on YARN-596: - Ah, I see what you're saying. Good point. In that case we'll probably need to push that check into the SchedulingPolicy and call it inside the loop in preemptContainer(). In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006615#comment-14006615 ] Wei Yan commented on YARN-596: -- yes, we can check the queue's policy in the preCheck function. If DRF, we use Resources.fitsIn(); if Fair, we use DEFAULT_CALCULATOR. Sounds good? In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006641#comment-14006641 ] Sandy Ryza commented on YARN-596: - Sounds good In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006676#comment-14006676
]
Sandy Ryza commented on YARN-596:
-
The current patch uses the queue's policy to preemptContainerPreCheck. We
should the parent's policy. (Consider the case of a leaf queue with FIFO under
a parent queue with DRF - we should use DRF to decide whether we should skip
the leaf queue).
Also, we should add a new method to SchedulingPolicy instead of checking with
instanceof.
{code}
+ if (Resources.fitsIn(getResourceUsage(), getFairShare())) {
+return false;
+ } else {
+return true;
+ }
{code}
Can just use return Resources.fitsIn(getResourceUsage(), getFairShare()).
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963271#comment-13963271 ] Sandy Ryza commented on YARN-596: - bq. Sandy Ryza, so here the ''safe'' means usage.memory fairshare.memory and usage.vcores fairshare.vcores? Right, but with those s as =s. bq. But the fairshare.vcores for FSQueue (except root) is always 0. And fairscheduler only considers memory when do scheduling, so do we still need to consider vcores here? These are only the case when the FairSharePolicy is used. When the DominantResourceFairnessPolicy is used, the vcores fair share is greater than 0 and the Fair Scheduler considers both memory and vcores for scheduling. In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962488#comment-13962488
]
Wei Yan commented on YARN-596:
--
{quote}For preemptContainerPreCheck, it would be good to take multiple
resources into account (using the DefaultResourceCalculator will only apply to
memory). Resources.fitsIn(getResourceUsage(), getFairShare) can be used to
determine whether a Schedulable is safe from preemption.{quote}
[~sandyr], so here the ''safe'' means usage.memory fairshare.memory and
usage.vcores fairshare.vcores? But the fairshare.vcores for FSQueue (except
root) is always 0. And fairscheduler only considers memory when do scheduling,
so do we still need to consider vcores here?
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955960#comment-13955960
]
Sandy Ryza commented on YARN-596:
-
Thanks. The patch is looking almost done. I like that you replaced the O(n
log n) sort call in preemptContainer with an O(n) iteration. Just a few more
nits:
{code}
+ LOG.debug(Queue + getName() + is going to preempt a container +
+ from its childQueues.);
{code}
This doesn't make sense in FSLeafQueue, which can't have child queues.
{code}
+// Let the selected queue to preempt
+if (candidateQueue != null) {
+ toBePreempted = candidateQueue.preemptContainer();
+}
{code}
Did you mean Let the selected queue choose which of its containers to preempt?
For preemptContainerPreCheck, it would be good to take multiple resources into
account (using the DefaultResourceCalculator will only apply to memory).
Resources.fitsIn(getResourceUsage(), getFairShare) can be used to determine
whether a Schedulable is safe from preemption.
Lastly, can you add a test that makes sure that containers from apps that are
higher over their fair share get preempted first, even when containers from
other apps that are over their fair share have lower priorities?
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951040#comment-13951040 ] Sandy Ryza commented on YARN-596: - Cloning the full queue/app tree makes preemption O(apps) vs. O(log(apps)), which seems expensive to do every time we want to preempt. Could we possibly add a field to Schedulable called something like preemptedResources that gets subtracted when calculating getResourceUsage, and cleared out at the end of preemptResources? Also, minor nit: better to use sched than sche for consistency with other places in the code. In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951126#comment-13951126 ] Wei Yan commented on YARN-596: -- Thought that approach before. But I forget why give up. I'll update a patch using that approach. In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch, YARN-596.patch In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951456#comment-13951456
]
Hadoop QA commented on YARN-596:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637524/YARN-596.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3486//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3486//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949522#comment-13949522
]
Hadoop QA commented on YARN-596:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12637163/YARN-596.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3474//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3474//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch,
YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13948574#comment-13948574
]
Sandy Ryza commented on YARN-596:
-
Thanks for working on this, Wei.
{code}
+// Collect running containers from over-scheduled queues
+MapRMContainer, FSSchedulerApp apps =
+new HashMapRMContainer, FSSchedulerApp();
+MapRMContainer, FSLeafQueue queues =
+new HashMapRMContainer, FSLeafQueue();
+for (FSLeafQueue sched : scheduler.getQueueManager().getLeafQueues()) {
+ if (Resources.greaterThan(RESOURCE_CALCULATOR, clusterCapacity,
+ sched.getResourceUsage(), sched.getFairShare())) {
+for (AppSchedulable as : sched.getRunnableAppSchedulables()) {
+ for (RMContainer c : as.getApp().getLiveContainers()) {
+apps.put(c, as.getApp());
+queues.put(c, sched);
+ }
+}
+ }
+}
{code}
Do we still need this anymore? If we're not collecting the running containers
here, it seems a little weird to build up the container-app/queue maps here.
We should be able to look up the application attempt and corresponding queue
with data that the RMContainer has, using Fair Scheduler data structures. Or
return them with preemptContainer.
Can we rename FSPreemption to FairSchedulerPreemptor or just Preemptor?
What's the reasoning for cloneQueueApps and not just traversing the existing
queues?
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945609#comment-13945609
]
Hadoop QA commented on YARN-596:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636408/YARN-596.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3443//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945695#comment-13945695
]
Hadoop QA commented on YARN-596:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636418/YARN-596.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3444//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/3444//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3444//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[
https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945792#comment-13945792
]
Hadoop QA commented on YARN-596:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636436/YARN-596.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3446//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3446//console
This message is automatically generated.
In fair scheduler, intra-application container priorities affect
inter-application preemption decisions
---
Key: YARN-596
URL: https://issues.apache.org/jira/browse/YARN-596
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-596.patch, YARN-596.patch, YARN-596.patch
In the fair scheduler, containers are chosen for preemption in the following
way:
All containers for all apps that are in queues that are over their fair share
are put in a list.
The list is sorted in order of the priority that the container was requested
in.
This means that an application can shield itself from preemption by
requesting it's containers at higher priorities, which doesn't really make
sense.
Also, an application that is not over its fair share, but that is in a queue
that is over it's fair share is just as likely to have containers preempted
as an application that is over its fair share.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638209#comment-13638209 ] Karthik Kambatla commented on YARN-596: --- Looks like we need preemptContainer() that complements assignContainer(): recurses through the queue hierarchy and preempts a container. In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-596) In fair scheduler, intra-application container priorities affect inter-application preemption decisions
[ https://issues.apache.org/jira/browse/YARN-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638229#comment-13638229 ] Sandy Ryza commented on YARN-596: - Agreed In fair scheduler, intra-application container priorities affect inter-application preemption decisions --- Key: YARN-596 URL: https://issues.apache.org/jira/browse/YARN-596 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza In the fair scheduler, containers are chosen for preemption in the following way: All containers for all apps that are in queues that are over their fair share are put in a list. The list is sorted in order of the priority that the container was requested in. This means that an application can shield itself from preemption by requesting it's containers at higher priorities, which doesn't really make sense. Also, an application that is not over its fair share, but that is in a queue that is over it's fair share is just as likely to have containers preempted as an application that is over its fair share. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
