[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307602#comment-14307602 ] Sandy Ryza commented on YARN-3101: -- +1 > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch, > YARN-3101.003.patch, YARN-3101.004.patch, YARN-3101.004.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306037#comment-14306037
]
Hadoop QA commented on YARN-3101:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696541/YARN-3101.004.patch
against trunk revision 40a4157.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6511//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6511//console
This message is automatically generated.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch,
> YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch,
> YARN-3101.003.patch, YARN-3101.004.patch, YARN-3101.004.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304321#comment-14304321
]
Hadoop QA commented on YARN-3101:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696300/YARN-3101.004.patch
against trunk revision c559df2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6497//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6497//console
This message is automatically generated.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch,
> YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch,
> YARN-3101.003.patch, YARN-3101.004.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301971#comment-14301971 ] Anubhav Dhoot commented on YARN-3101: - YARN-1778 (TestFSRMStateStore) and YARN-3113 (audit failure) track these 2 failures > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch, > YARN-3101.003.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301574#comment-14301574 ] Siqi Li commented on YARN-3101: --- The patch looks good to me > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch, > YARN-3101.003.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299911#comment-14299911
]
Hadoop QA commented on YARN-3101:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695755/YARN-3101.003.patch
against trunk revision 26c2de3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6478//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6478//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6478//console
This message is automatically generated.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch,
> YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch,
> YARN-3101.003.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299689#comment-14299689 ] Anubhav Dhoot commented on YARN-3101: - The failure does not repro after pulling the latest trunk and the release audit warning is in a unrelated file > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299592#comment-14299592
]
Hadoop QA commented on YARN-3101:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695697/YARN-3101.003.patch
against trunk revision 09ad9a8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6472//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6472//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6472//console
This message is automatically generated.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch,
> YARN-3101.001.patch, YARN-3101.002.patch, YARN-3101.003.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299428#comment-14299428 ] Sandy Ryza commented on YARN-3101: -- In that case it sounds like the behavior is that we can go one container over the max resources. While this might be worth changing in a separate JIRA, we should maintain that behavior with the reservations. > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299423#comment-14299423
]
Hadoop QA commented on YARN-3101:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695632/YARN-3101-Siqi.v2.patch
against trunk revision 951b360.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6471//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6471//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6471//console
This message is automatically generated.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch,
> YARN-3101.001.patch, YARN-3101.002.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299403#comment-14299403 ] Siqi Li commented on YARN-3101: --- feel free to still use <=. This doesn't change the overall behavior > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299396#comment-14299396 ] Anubhav Dhoot commented on YARN-3101: - The test case was modified to add a negative test that show reservation that should be maintained are not being anymore. So we need the test case changes from my patch. > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299369#comment-14299369 ] Siqi Li commented on YARN-3101: --- [~adhoot] This reason of using < instead of <= is basically for keeping the test case intact. > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299348#comment-14299348 ] Anubhav Dhoot commented on YARN-3101: - Hi [~l201514] why do we need to make the comparator < instead <=. What case does this address? [~sandyr] I did not see a check when placing a reservation. We check a queue usage once in FSLeafQueue#assignContainerPreCheck but we do not know the container size until actual reserve happens in FSAppAttempt reserve. > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch, > YARN-3101.001.patch, YARN-3101.002.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299306#comment-14299306
]
Hadoop QA commented on YARN-3101:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695624/YARN-3101-Siqi.v2.patch
against trunk revision 951b360.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6470//console
This message is automatically generated.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101-Siqi.v1.patch, YARN-3101-Siqi.v2.patch,
> YARN-3101.001.patch, YARN-3101.002.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298887#comment-14298887 ] Sandy Ryza commented on YARN-3101: -- [~adhoot] is this the same condition that's evaluated when reserving a resource in the first place? I.e. might we ever make a reservation and then immediately end up canceling it? Also, I believe [~l201514] is correct that reservedAppSchedulable.getResource(reservedPriority))) will not return the right quantity and node.getReservedContainer().getReservedResource() is correct. Last of all, while we're at it, can we rename "fitInMaxShare" to "fitsInMaxShare"? > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101-Siqi.v1.patch, YARN-3101.001.patch, > YARN-3101.002.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296297#comment-14296297
]
Hadoop QA commented on YARN-3101:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695156/YARN-3101-Siqi.v1.patch
against trunk revision 5a0051f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6449//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6449//console
This message is automatically generated.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101-Siqi.v1.patch, YARN-3101.001.patch,
> YARN-3101.002.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296174#comment-14296174
]
Hadoop QA commented on YARN-3101:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695141/YARN-3101-Siqi.v1.patch
against trunk revision 5a0051f.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6447//console
This message is automatically generated.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101-Siqi.v1.patch, YARN-3101.001.patch,
> YARN-3101.002.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296046#comment-14296046 ] Siqi Li commented on YARN-3101: --- But I am not quite sure about that if reservedAppSchedulable.getResource(reservedPriority) can give us the correct reserved resource. Maybe considering node.getReservedContainer().getReservedResource() > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101.001.patch, YARN-3101.002.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295924#comment-14295924 ] Siqi Li commented on YARN-3101: --- It looks like you are right. The if condition somehow got reversed, so that reservations will be always rejected > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101.001.patch, YARN-3101.002.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295760#comment-14295760
]
Hadoop QA commented on YARN-3101:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695062/YARN-3101.002.patch
against trunk revision 9850e15.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1193 javac
compiler warnings (more than the trunk's current 1187 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6439//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6439//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6439//console
This message is automatically generated.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101.001.patch, YARN-3101.002.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[
https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295664#comment-14295664
]
Anubhav Dhoot commented on YARN-3101:
-
YARN-2811 added the following
{noformat}
private boolean fitInMaxShare(FSQueue queue) {
if (Resources.fitsIn(queue.getResourceUsage(), queue.getMaxShare())) {
return false;
}
{noformat}
This should have been
{noformat}
if (!Resources.fitsIn(queue.getResourceUsage(), queue.getMaxShare()))
{noformat}
Because of this we would always reject reservations and hence the test was
passing
Adding a new patch that shows this via a negative test case (reservation should
not be dropped when not exceeding). This fails without the new fix.
> FairScheduler#fitInMaxShare was added to validate reservations but it does
> not consider it
> ---
>
> Key: YARN-3101
> URL: https://issues.apache.org/jira/browse/YARN-3101
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
> Attachments: YARN-3101.001.patch, YARN-3101.002.patch
>
>
> YARN-2811 added fitInMaxShare to validate reservations on a queue, but did
> not count it during its calculations. It also had the condition reversed so
> the test was still passing because both cancelled each other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293900#comment-14293900 ] Siqi Li commented on YARN-3101: --- The reason that the reservation did not count in YARN-2811 is due to the fact that queue usage can go slightly beyond queue max. The difference could be just one container. In general, the patch looks good to me, but I don't quite understand what you mean by "it had the condition reversed". > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101.001.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3101) FairScheduler#fitInMaxShare was added to validate reservations but it does not consider it
[ https://issues.apache.org/jira/browse/YARN-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293010#comment-14293010 ] Anubhav Dhoot commented on YARN-3101: - [~sandyr] [~l201514] appreciate your review > FairScheduler#fitInMaxShare was added to validate reservations but it does > not consider it > --- > > Key: YARN-3101 > URL: https://issues.apache.org/jira/browse/YARN-3101 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Anubhav Dhoot > Attachments: YARN-3101.001.patch > > > YARN-2811 added fitInMaxShare to validate reservations on a queue, but did > not count it during its calculations. It also had the condition reversed so > the test was still passing because both cancelled each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
