[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349924#comment-16349924 ] Konstantinos Karanasos commented on YARN-7778: -- Thanks [~cheersyang], will commit it tomorrow morning. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch, > YARN-7778.003.patch, YARN-7778.004.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349610#comment-16349610 ] Weiwei Yang commented on YARN-7778: --- Sure thing, thanks for taking care of it [~kkaranasos] during commit. Appreciate. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch, > YARN-7778.003.patch, YARN-7778.004.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349178#comment-16349178 ] Konstantinos Karanasos commented on YARN-7778: -- +1 on latest patch, thanks [~cheersyang]. I will do a minor fix to say that the "constraint" is coming from the SchedulingRequest when I commit the patch, if you don't mind. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch, > YARN-7778.003.patch, YARN-7778.004.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348233#comment-16348233 ] Weiwei Yang commented on YARN-7778: --- Hi [~kkaranasos] please take a look when you get a chance. Thanks! > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch, > YARN-7778.003.patch, YARN-7778.004.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[
https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348210#comment-16348210
]
genericqa commented on YARN-7778:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 17s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7778 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12908734/YARN-7778.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6a87da61974c 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / af015c0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19560/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19560/testReport/ |
| Max. process+thread count | 821 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[
https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347990#comment-16347990
]
genericqa commented on YARN-7778:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 53s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 58s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7778 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12908704/YARN-7778.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 025fb8e91705 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0bee384 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19557/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[
https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347916#comment-16347916
]
Weiwei Yang commented on YARN-7778:
---
Hi [~kkaranasos]
Thanks for your comments, I agree with both #1 and #2 and get them done in next
patch. I also took a look #3, that code fragment removes null or dup entries in
a list of {{PlacementConstraint}} and then return us a list of
{{AbstractConstraint}}, it was not straightforward to me how other place is
going to re-use this. But let me know if you have a strong preference to do so.
Thanks.
> Merging of constraints defined at different levels
> --
>
> Key: YARN-7778
> URL: https://issues.apache.org/jira/browse/YARN-7778
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Merge Constraints Solution.pdf,
> YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch
>
>
> When we have multiple constraints defined for a given set of allocation tags
> at different levels (i.e., at the cluster, the application or the scheduling
> request level), we need to merge those constraints.
> Defining constraint levels as cluster > application > scheduling request,
> constraints defined at lower levels should only be more restrictive than
> those of higher levels. Otherwise the allocation should fail.
> For example, if there is an application level constraint that allows no more
> than 5 HBase containers per rack, a scheduling request can further restrict
> that to 3 containers per rack but not to 7 containers per rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347514#comment-16347514 ] Konstantinos Karanasos commented on YARN-7778: -- Thanks for the patch, [~cheersyang]. A couple of comments: * Let's rename the getResourceConstraint() to something like getMultilevelConstraint? Also the initial part of the comment "Retrieve the placement constraint of a given scheduling request from the PCM point of view" could be made more like "Consider all levels of constraints (resource request, app, cluster) and return a merged constraint". This is because we can also return a merged constraint of app and cluster level, even if there are no resource request constraints. * I would also change the signature of the method to add directly a Constraint coming from the resource request and a set of tags, rather than pass the SchedulingRequest object. This way we make clear that the set of tags is separate from the specific constraint. * Do we need the "Remove all null or duplicate constraints"? I was thinking that this should belong to a separate step that will do constraint simplification/minimization. If we keep it, I would suggest to put it in a separate method in the PCM, so that we can use it elsewhere too or extend it in the future easily. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[
https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346367#comment-16346367
]
Weiwei Yang commented on YARN-7778:
---
Test {{TestPlacementProcessor#testRePlacementAfterSchedulerRejection}} was not
related, I saw this several times in past jenkins reports, looks like a flaky
one. The other one is tracked via YARN-7860.
> Merging of constraints defined at different levels
> --
>
> Key: YARN-7778
> URL: https://issues.apache.org/jira/browse/YARN-7778
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Merge Constraints Solution.pdf,
> YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch
>
>
> When we have multiple constraints defined for a given set of allocation tags
> at different levels (i.e., at the cluster, the application or the scheduling
> request level), we need to merge those constraints.
> Defining constraint levels as cluster > application > scheduling request,
> constraints defined at lower levels should only be more restrictive than
> those of higher levels. Otherwise the allocation should fail.
> For example, if there is an application level constraint that allows no more
> than 5 HBase containers per rack, a scheduling request can further restrict
> that to 3 containers per rack but not to 7 containers per rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[
https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346365#comment-16346365
]
genericqa commented on YARN-7778:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} YARN-7812 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
32s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} YARN-7812 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 48s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 42s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
| |
hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementProcessor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7778 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12908499/YARN-7778-YARN-7812.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d889e313a343 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-7812 / e6d2d26 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19540/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19540/testReport/ |
| Max. process+thread count | 865 (vs. ulimit of 5000) |
| modules | C:
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346362#comment-16346362 ] Weiwei Yang commented on YARN-7778: --- [~kkaranasos] could you please help to review the patch? Thanks > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf, > YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[
https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346276#comment-16346276
]
Weiwei Yang commented on YARN-7778:
---
Fix the UT failure in {{TestSingleConstraintAppPlacementAllocator}}, the other
one is caused by {{YARN-7817}}, will open a JIRA to track.
> Merging of constraints defined at different levels
> --
>
> Key: YARN-7778
> URL: https://issues.apache.org/jira/browse/YARN-7778
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Merge Constraints Solution.pdf,
> YARN-7778-YARN-7812.001.patch, YARN-7778-YARN-7812.002.patch
>
>
> When we have multiple constraints defined for a given set of allocation tags
> at different levels (i.e., at the cluster, the application or the scheduling
> request level), we need to merge those constraints.
> Defining constraint levels as cluster > application > scheduling request,
> constraints defined at lower levels should only be more restrictive than
> those of higher levels. Otherwise the allocation should fail.
> For example, if there is an application level constraint that allows no more
> than 5 HBase containers per rack, a scheduling request can further restrict
> that to 3 containers per rack but not to 7 containers per rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[
https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346225#comment-16346225
]
genericqa commented on YARN-7778:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-7812 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
4s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} YARN-7812 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} YARN-7812 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 33s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
| |
hadoop.yarn.server.resourcemanager.scheduler.placement.TestSingleConstraintAppPlacementAllocator
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7778 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12908472/YARN-7778-YARN-7812.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9079a777e1b7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-7812 / e6d2d26 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19537/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19537/testReport/ |
| Max. process+thread count | 894 (vs. ulimit of 5000) |
|
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344379#comment-16344379 ] Weiwei Yang commented on YARN-7778: --- Thanks for the feedback [~kkaranasos], I will work on a patch once YARN-7822 is done. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344154#comment-16344154 ] Konstantinos Karanasos commented on YARN-7778: -- Hi [~cheersyang], just checked your proposal. I like the idea of taking advantage of the AND constraints, instead of actually merging the constraints. So +1 to your approach. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339242#comment-16339242 ] Weiwei Yang commented on YARN-7778: --- Hi [~kkaranasos], I thought about this today and come up with a doc [^Merge Constraints Solution.pdf], please take a look and let me know your thoughts over this. Thanks a lot. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Weiwei Yang >Priority: Major > Attachments: Merge Constraints Solution.pdf > > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333136#comment-16333136 ] Konstantinos Karanasos commented on YARN-7778: -- Sure [~cheersyang], go ahead. Let me know if you want to discuss about any details. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Priority: Major > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7778) Merging of constraints defined at different levels
[ https://issues.apache.org/jira/browse/YARN-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333091#comment-16333091 ] Weiwei Yang commented on YARN-7778: --- I volunteer to help on this task, [~kkaranasos] can I take over? I would like to explore this some day next week. > Merging of constraints defined at different levels > -- > > Key: YARN-7778 > URL: https://issues.apache.org/jira/browse/YARN-7778 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Priority: Major > > When we have multiple constraints defined for a given set of allocation tags > at different levels (i.e., at the cluster, the application or the scheduling > request level), we need to merge those constraints. > Defining constraint levels as cluster > application > scheduling request, > constraints defined at lower levels should only be more restrictive than > those of higher levels. Otherwise the allocation should fail. > For example, if there is an application level constraint that allows no more > than 5 HBase containers per rack, a scheduling request can further restrict > that to 3 containers per rack but not to 7 containers per rack. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
