[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613418#comment-15613418
]
Karthik Kambatla commented on YARN-2083:
Firstly, thanks for working on this long-standing issue.
Correct me if I am wrong. Ideally, the check should be {{currentUsage +
newContainerCapacity <= maxShare}}. Changing it from {{currentUsage <=
maxShare}} to {{currentUsage < maxShare}} could address some cases, but not all
the cases. Can we look at fixing it for all cases, moving the place
assignContainerPreCheck is called if need be.
> In fair scheduler, Queue should not been assigned more containers when its
> usedResource had reach the maxResource limit
> ---
>
> Key: YARN-2083
> URL: https://issues.apache.org/jira/browse/YARN-2083
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.3.0
>Reporter: Yi Tian
> Labels: BB2015-05-TBR, oct16-medium
> Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch,
> YARN-2083-4.patch, YARN-2083.patch
>
>
> In fair scheduler, FSParentQueue and FSLeafQueue do an
> assignContainerPreCheck to guaranty this queue is not over its limit.
> But the fitsIn function in Resource.java did not return false when the
> usedResource equals the maxResource.
> I think we should create a new Function "fitsInWithoutEqual" instead of
> "fitsIn" in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061773#comment-15061773 ] yangcongqi commented on YARN-2083: -- There is a case: queue A maxResource is 1000mb, current usedResource is 900mb(less than maxResource), then a new request with 200mb came, fair scheduler would assign 200mb memory and usedResource=1100mb(over maxResource). > In fair scheduler, Queue should not been assigned more containers when its > usedResource had reach the maxResource limit > --- > > Key: YARN-2083 > URL: https://issues.apache.org/jira/browse/YARN-2083 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.3.0 >Reporter: Yi Tian > Labels: BB2015-05-TBR > Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch, > YARN-2083-4.patch, YARN-2083.patch > > > In fair scheduler, FSParentQueue and FSLeafQueue do an > assignContainerPreCheck to guaranty this queue is not over its limit. > But the fitsIn function in Resource.java did not return false when the > usedResource equals the maxResource. > I think we should create a new Function "fitsInWithoutEqual" instead of > "fitsIn" in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061981#comment-15061981
]
Hadoop QA commented on YARN-2083:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 3s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 42s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 12s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 157m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| |
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533766#comment-14533766 ] Yi Tian commented on YARN-2083: --- could anyone review this patch? In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit --- Key: YARN-2083 URL: https://issues.apache.org/jira/browse/YARN-2083 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 2.3.0 Reporter: Yi Tian Labels: BB2015-05-TBR Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch, YARN-2083-4.patch, YARN-2083.patch In fair scheduler, FSParentQueue and FSLeafQueue do an assignContainerPreCheck to guaranty this queue is not over its limit. But the fitsIn function in Resource.java did not return false when the usedResource equals the maxResource. I think we should create a new Function fitsInWithoutEqual instead of fitsIn in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530417#comment-14530417
]
Hadoop QA commented on YARN-2083:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 26s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 38s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 52m 15s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 92m 32s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSQueue |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730783/YARN-2083-4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a583a40 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7731/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7731/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7731/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7731/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7731/console |
This message was automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: BB2015-05-TBR
Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch,
YARN-2083-4.patch, YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531936#comment-14531936
]
Hadoop QA commented on YARN-2083:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 44s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 33s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 32s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 37s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 53m 24s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 94m 0s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731048/YARN-2083-4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4c7b9b6 |
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7747/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7747/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7747/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7747/console |
This message was automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: BB2015-05-TBR
Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch,
YARN-2083-4.patch, YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531956#comment-14531956
]
Hadoop QA commented on YARN-2083:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 43s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 29s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 46s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | yarn tests | 2m 3s | Tests passed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 54m 1s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 95m 8s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSQueue |
| | hadoop.yarn.server.resourcemanager.TestRM |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue |
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731048/YARN-2083-4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4c7b9b6 |
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7748/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7748/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7748/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7748/console |
This message was automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: BB2015-05-TBR
Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch,
YARN-2083-4.patch, YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329194#comment-14329194
]
Hadoop QA commented on YARN-2083:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650950/YARN-2083-3.patch
against trunk revision a64dd3d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6682//console
This message is automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: fairscheduler
Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch,
YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14035879#comment-14035879 ] Wei Yan commented on YARN-2083: --- Hey, [~tianyi], the patch looks good to me. In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit --- Key: YARN-2083 URL: https://issues.apache.org/jira/browse/YARN-2083 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Yi Tian Labels: assignContainer, fair, scheduler Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch, YARN-2083.patch In fair scheduler, FSParentQueue and FSLeafQueue do an assignContainerPreCheck to guaranty this queue is not over its limit. But the fitsIn function in Resource.java did not return false when the usedResource equals the maxResource. I think we should create a new Function fitsInWithoutEqual instead of fitsIn in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034154#comment-14034154
]
Hadoop QA commented on YARN-2083:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650834/YARN-2083-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSQueue
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4015//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4015//console
This message is automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: assignContainer, fair, scheduler
Fix For: 2.4.1
Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034777#comment-14034777
]
Hadoop QA commented on YARN-2083:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650950/YARN-2083-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4019//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4019//console
This message is automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: assignContainer, fair, scheduler
Fix For: 2.4.1
Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch,
YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034812#comment-14034812 ] Yi Tian commented on YARN-2083: --- [~ywskycn], thanks for your advice, YARN-2083-3.patch works fine in thunk ,YARN-2083-2.patch works fine in branch-2.4.1. is it possible to apply this patch into yarn-project? In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit --- Key: YARN-2083 URL: https://issues.apache.org/jira/browse/YARN-2083 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Yi Tian Labels: assignContainer, fair, scheduler Attachments: YARN-2083-1.patch, YARN-2083-2.patch, YARN-2083-3.patch, YARN-2083.patch In fair scheduler, FSParentQueue and FSLeafQueue do an assignContainerPreCheck to guaranty this queue is not over its limit. But the fitsIn function in Resource.java did not return false when the usedResource equals the maxResource. I think we should create a new Function fitsInWithoutEqual instead of fitsIn in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14029273#comment-14029273
]
Wei Yan commented on YARN-2083:
---
Thanks, [~tianyi]. Here are more comments.
Could we move the test code to a new file TestFSQueue.java, as the evaluated
function is located in FSQueue.
{code}
boolean couldAssignMoreContainer = schedulable.assignContainerPreCheck(
new FSSchedulerNode(fakeNode, true));
{code}
We don't need to create a new FSSchedulerNode each time. Just create one.
some nitty comments: (1) for comment style, I may much prefer // Test the...
instead of //test the You can check the other comments in the code.
(2) Not need to create couldAssignMoreContainer, just directly put the
assignContainerPreCheck function inside the assertTrue/False.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: assignContainer, fair, scheduler
Fix For: 2.4.1
Attachments: YARN-2083-1.patch, YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028741#comment-14028741 ] Yi Tian commented on YARN-2083: --- Thanks [~ywskycn] [~kasha], Here is the final patch file. In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit --- Key: YARN-2083 URL: https://issues.apache.org/jira/browse/YARN-2083 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Yi Tian Labels: assignContainer, fair, scheduler Fix For: 2.4.1 Attachments: YARN-2083-1.patch, YARN-2083.patch In fair scheduler, FSParentQueue and FSLeafQueue do an assignContainerPreCheck to guaranty this queue is not over its limit. But the fitsIn function in Resource.java did not return false when the usedResource equals the maxResource. I think we should create a new Function fitsInWithoutEqual instead of fitsIn in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028772#comment-14028772
]
Hadoop QA commented on YARN-2083:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649982/YARN-2083-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3969//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3969//console
This message is automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: assignContainer, fair, scheduler
Fix For: 2.4.1
Attachments: YARN-2083-1.patch, YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026917#comment-14026917 ] Karthik Kambatla commented on YARN-2083: Fix version is set at the time of commit by the committer depending on what branches they committed it to. In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit --- Key: YARN-2083 URL: https://issues.apache.org/jira/browse/YARN-2083 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Yi Tian Labels: assignContainer, fair, scheduler Attachments: YARN-2083.patch In fair scheduler, FSParentQueue and FSLeafQueue do an assignContainerPreCheck to guaranty this queue is not over its limit. But the fitsIn function in Resource.java did not return false when the usedResource equals the maxResource. I think we should create a new Function fitsInWithoutEqual instead of fitsIn in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026967#comment-14026967
]
Wei Yan commented on YARN-2083:
---
[~tianyi], here are some comments.
1. The tab size should be 2 characters .
{code}
public static boolean fitsInWithoutEqual(Resource smaller, Resource bigger) {
return smaller.getMemory() bigger.getMemory()
smaller.getVirtualCores() bigger.getVirtualCores();
}
{code}
2. Some lines in testAssignContainerPreCheck function are over 80 characters.
We like to keep lines to max 80 characters.
3. IMO, the objective of testAssignContainerPreCheck function is used to verify
whether the preCheck can work well when resourceUsage is smaller/euqal/larger
than maxResource. We may need to add some testcases in this function.
4. Another nitty comment: there is one unnecessary line at the end of the
TestFSLeafQueue.java.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: assignContainer, fair, scheduler
Attachments: YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026130#comment-14026130 ] Yi Tian commented on YARN-2083: --- could anyone review this patch? In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit --- Key: YARN-2083 URL: https://issues.apache.org/jira/browse/YARN-2083 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Yi Tian Labels: assignContainer, fair, scheduler Fix For: 2.3.0 Attachments: YARN-2083.patch In fair scheduler, FSParentQueue and FSLeafQueue do an assignContainerPreCheck to guaranty this queue is not over its limit. But the fitsIn function in Resource.java did not return false when the usedResource equals the maxResource. I think we should create a new Function fitsInWithoutEqual instead of fitsIn in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012223#comment-14012223
]
Hadoop QA commented on YARN-2083:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647328/YARN-2083.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3856//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3856//console
This message is automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: assignContainer, fair, scheduler
Fix For: 2.3.0
Attachments: YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008671#comment-14008671 ] Yi Tian commented on YARN-2083: --- [~sandyr] could you help me to find out how is these case failed? I ran the test without my patch, it failed too. In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit --- Key: YARN-2083 URL: https://issues.apache.org/jira/browse/YARN-2083 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Yi Tian Labels: assignContainer, fair, scheduler Fix For: 2.3.0 Attachments: YARN-2083.patch In fair scheduler, FSParentQueue and FSLeafQueue do an assignContainerPreCheck to guaranty this queue is not over its limit. But the fitsIn function in Resource.java did not return false when the usedResource equals the maxResource. I think we should create a new Function fitsInWithoutEqual instead of fitsIn in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008571#comment-14008571 ] Yi Tian commented on YARN-2083: --- I think precheck is better than aftercheck. And I think we could not do more check because we don't know the schedule application' information in assignContainerPreCheck function In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit --- Key: YARN-2083 URL: https://issues.apache.org/jira/browse/YARN-2083 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Yi Tian Labels: assignContainer, fair, scheduler Fix For: 2.3.0 Attachments: YARN-2083.patch In fair scheduler, FSParentQueue and FSLeafQueue do an assignContainerPreCheck to guaranty this queue is not over its limit. But the fitsIn function in Resource.java did not return false when the usedResource equals the maxResource. I think we should create a new Function fitsInWithoutEqual instead of fitsIn in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008587#comment-14008587
]
Hadoop QA commented on YARN-2083:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12646736/YARN-2083.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3830//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3830//console
This message is automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: assignContainer, fair, scheduler
Fix For: 2.3.0
Attachments: YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007669#comment-14007669
]
Hadoop QA commented on YARN-2083:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12646479/YARN-2083.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3800//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3800//console
This message is automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: assignContainer, fair, scheduler
Fix For: 2.3.0
Attachments: YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007757#comment-14007757
]
Hadoop QA commented on YARN-2083:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12646479/YARN-2083.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3807//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3807//console
This message is automatically generated.
In fair scheduler, Queue should not been assigned more containers when its
usedResource had reach the maxResource limit
---
Key: YARN-2083
URL: https://issues.apache.org/jira/browse/YARN-2083
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
Labels: assignContainer, fair, scheduler
Fix For: 2.3.0
Attachments: YARN-2083.patch
In fair scheduler, FSParentQueue and FSLeafQueue do an
assignContainerPreCheck to guaranty this queue is not over its limit.
But the fitsIn function in Resource.java did not return false when the
usedResource equals the maxResource.
I think we should create a new Function fitsInWithoutEqual instead of
fitsIn in this case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit
[ https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007786#comment-14007786 ] Sandy Ryza commented on YARN-2083: -- I think it would be better to solve this by making sure the queue cannot actually exceed the maxResources limit. I.e. to also check the constraint *after* picking a container that would be assigned. In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit --- Key: YARN-2083 URL: https://issues.apache.org/jira/browse/YARN-2083 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Yi Tian Labels: assignContainer, fair, scheduler Fix For: 2.3.0 Attachments: YARN-2083.patch In fair scheduler, FSParentQueue and FSLeafQueue do an assignContainerPreCheck to guaranty this queue is not over its limit. But the fitsIn function in Resource.java did not return false when the usedResource equals the maxResource. I think we should create a new Function fitsInWithoutEqual instead of fitsIn in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
