[jira] [Commented] (YARN-7793) Lots of parsing error reported on nodemanager after activating yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled
[
https://issues.apache.org/jira/browse/YARN-7793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365198#comment-16365198
]
Nicolas Fraison commented on YARN-7793:
---
[~vinodkv] and [~rajesh.balamohan] could you please review.
Thks
> Lots of parsing error reported on nodemanager after activating
> yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled
> -
>
> Key: YARN-7793
> URL: https://issues.apache.org/jira/browse/YARN-7793
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Environment: cloudera release cdh-5.11.0
> openjdk version "1.8.0_91"
> linux centos7 servers
>Reporter: Nicolas Fraison
>Priority: Minor
> Attachments: YARN-7793.001.patch, YARN-7793.002.patch, YARN-7793.patch
>
>
> After activating
> yarn.nodemanager.container-monitor.procfs-tree.smaps-based-rss.enabled
> nodemanagers are reporting lots of parsing error like this one
> {code:java}
> Error in parsing : INVALID : valuerd wr mr mw me ac{code}
> Looking at smaps file I can see that there are in fact one line with no
> Number:
> {code:java}
> VmFlags: rd ex mr mw me de{code}
> This line is retreived by the MEM_INFO_PATTERN in ProcfsBasedProcessTree
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[
https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365174#comment-16365174
]
genericqa commented on YARN-7920:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 3s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 15 new + 408 unchanged - 0 fixed = 423 total (was 408) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
19s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m
40s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 8s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-yarn-site in the patch
[jira] [Commented] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
[
https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365144#comment-16365144
]
genericqa commented on YARN-7813:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 10s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 9 new + 919 unchanged - 1 fixed = 928 total (was 920) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m
32s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 4s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
|
[jira] [Commented] (YARN-7936) Add default service AM Xmx
[
https://issues.apache.org/jira/browse/YARN-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365121#comment-16365121
]
genericqa commented on YARN-7936:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 42s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 7s{color}
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7936 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910675/YARN-7936.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux db40c690ff3b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8f66aff |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19698/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19698/testReport/ |
| Max. process+thread count | 603 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
[jira] [Commented] (YARN-7707) [GPG] Policy generator framework
[
https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365110#comment-16365110
]
genericqa commented on YARN-7707:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} YARN-7402 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
23s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
21s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
20s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
YARN-7402 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} YARN-7402 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 6 new + 226 unchanged - 0 fixed = 232 total (was 226) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
12s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
14s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-yarn-server-globalpolicygenerator in the
patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
35s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7707 |
| JIRA Patch URL |
[jira] [Commented] (YARN-7292) Revisit Resource Profile Behavior
[
https://issues.apache.org/jira/browse/YARN-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365099#comment-16365099
]
genericqa commented on YARN-7292:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} YARN-7292 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7292 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910164/YARN-7292.006.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19699/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Revisit Resource Profile Behavior
> -
>
> Key: YARN-7292
> URL: https://issues.apache.org/jira/browse/YARN-7292
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-7292.002.patch, YARN-7292.003.patch,
> YARN-7292.004.patch, YARN-7292.005.patch, YARN-7292.006.patch,
> YARN-7292.wip.001.patch
>
>
> Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a
> couple of resource profile related behaviors might need to be updated:
> 1) Configure resource profile in server side or client side:
> Currently resource profile can be only configured centrally:
> - Advantages:
> A given resource profile has a the same meaning in the cluster. It won’t
> change when we run different apps in different configurations. A job can run
> under Amazon’s G2.8X can also run on YARN with G2.8X profile. A side benefit
> is YARN scheduler can potentially do better bin packing.
> - Disadvantages:
> Hard for applications to add their own resource profiles.
> 2) Do we really need mandatory resource profiles such as
> minimum/maximum/default?
> 3) Should we send resource profile name inside ResourceRequest, or should
> client/AM translate it to resource and set it to the existing resource
> fields?
> 4) Related to above, should we allow resource overrides or client/AM should
> send final resource to RM?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365097#comment-16365097 ] Wangda Tan commented on YARN-7920: -- Updated (Ver.5) patch, also converted to standard markdown, only a few lines of change. Removed .vm suffix. > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch, YARN-7920.005.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7920: - Attachment: YARN-7920.005.patch > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch, YARN-7920.005.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365092#comment-16365092 ] Wangda Tan commented on YARN-7920: -- Thanks [~kkaranasos], the .md.vm is not supported by Intellij and standard markdown editor. If everybody agrees, I can convert it to the standard .md DisabledConstraintProcessor is fine. Will update the patch shortly. > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7403) [GQ] Compute global and local "IdealAllocation"
[
https://issues.apache.org/jira/browse/YARN-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365084#comment-16365084
]
genericqa commented on YARN-7403:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
3m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
54s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 54s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 34 new + 212 unchanged - 0 fixed = 246 total (was 212) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
59s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 3 new + 4 unchanged - 0 fixed = 7 total (was 4) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
17s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7403 |
| JIRA Patch URL |
[jira] [Updated] (YARN-7936) Add default service AM Xmx
[ https://issues.apache.org/jira/browse/YARN-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7936: -- Attachment: YARN-7936.1.patch > Add default service AM Xmx > -- > > Key: YARN-7936 > URL: https://issues.apache.org/jira/browse/YARN-7936 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He >Priority: Major > Attachments: YARN-7936.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7936) Add default service AM Xmx
Jian He created YARN-7936: - Summary: Add default service AM Xmx Key: YARN-7936 URL: https://issues.apache.org/jira/browse/YARN-7936 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7403) [GQ] Compute global and local "IdealAllocation"
[
https://issues.apache.org/jira/browse/YARN-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365063#comment-16365063
]
Carlo Curino commented on YARN-7403:
The v1 patch contains a much reworked version of the code. This depends on
YARN-7934 so will not compile. The key idea here is to provide algorithms that
will run every few seconds in the GPG and observe the overall state of a
federated cluster. The algorithm will leverage some of the
{{PreemptableResourceCalculator}} logic (with several additions) to compute
what is the ideal allocation for each queues. The extra effort is put into
considering "locality" among sub-clusters, which require some careful
consideration. Various heuristics can be chosen from (we implement two as
reference), and once YARN-7885/YARN-7886 will be ready to use we can experiment
on which is most closely approximating the behavior of a single
{{CapacityScheduler}} overlooking the entire federation.
> [GQ] Compute global and local "IdealAllocation"
> ---
>
> Key: YARN-7403
> URL: https://issues.apache.org/jira/browse/YARN-7403
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Reporter: Carlo Curino
>Assignee: Carlo Curino
>Priority: Major
> Attachments: YARN-7403.draft.patch, YARN-7403.draft2.patch,
> YARN-7403.draft3.patch, YARN-7403.v1.patch, global-queues-preemption.PNG
>
>
> This JIRA tracks algorithmic effort to combine the local queue views of
> capacity guarantee/use/demand and compute the global amount of preemption,
> and based on that, "where" (in which sub-cluster) preemption will be enacted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7403) [GQ] Compute global and local "IdealAllocation"
[ https://issues.apache.org/jira/browse/YARN-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-7403: --- Attachment: YARN-7403.v1.patch > [GQ] Compute global and local "IdealAllocation" > --- > > Key: YARN-7403 > URL: https://issues.apache.org/jira/browse/YARN-7403 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Major > Attachments: YARN-7403.draft.patch, YARN-7403.draft2.patch, > YARN-7403.draft3.patch, YARN-7403.v1.patch, global-queues-preemption.PNG > > > This JIRA tracks algorithmic effort to combine the local queue views of > capacity guarantee/use/demand and compute the global amount of preemption, > and based on that, "where" (in which sub-cluster) preemption will be enacted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7707) [GPG] Policy generator framework
[ https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-7707: - Attachment: YARN-7707-YARN-7402.02.patch > [GPG] Policy generator framework > > > Key: YARN-7707 > URL: https://issues.apache.org/jira/browse/YARN-7707 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Labels: federation, gpg > Attachments: YARN-7707-YARN-7402.01.patch, > YARN-7707-YARN-7402.02.patch > > > This JIRA tracks the development of a generic framework for querying > sub-clusters for metrics, running policies, and updating them in the > FederationStateStore. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365047#comment-16365047
]
Yufei Gu commented on YARN-5028:
In case of {{createAndPopulateNewRMApp}}, it uses the ApplicationACLs in
submission context, if you replace the original one with a new object. There is
a risk to load wrong ACL.
Nits in the unit test:
# The first set of assert seems unnecessary to me.
{code}
assertEquals(appId, ctx.getApplicationId());
assertEquals("a_queue", ctx.getQueue());
assertEquals(Collections.singletonList("a_command"),
ctx.getAMContainerSpec().getCommands());
assertEquals(17L, ctx.getResource().getMemorySize());
assertEquals(schedulingPropertiesMap,
ctx.getApplicationSchedulingPropertiesMap());
{code}
If you think it is necessary, please add some comments.
# For the second set, I think one assert should be enough. Something like this:
{code}
assertEquals("No prune if application state isn't FINISHED",
schedulingPropertiesMap,
ctx.getApplicationSchedulingPropertiesMap());
{code}
Otherwise, looks good to me.
> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch,
> YARN-5028.002.patch
>
>
> RMStateStore stores enough information to recover applications in case of a
> restart. The store also retains this information for completed applications
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all
> the information we store today to serve application status; for instance, we
> don't need the {{ApplicationSubmissionContext}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
[
https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365044#comment-16365044
]
Eric Payne commented on YARN-7813:
--
{{YARN-7813.004.patch}} should address the failures.
> Capacity Scheduler Intra-queue Preemption should be configurable for each
> queue
> ---
>
> Key: YARN-7813
> URL: https://issues.apache.org/jira/browse/YARN-7813
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.9.0, 2.8.3, 3.0.0
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Attachments: YARN-7813.001.patch, YARN-7813.002.branch-3.0.patch,
> YARN-7813.002.patch, YARN-7813.003.branch-2.patch,
> YARN-7813.003.branch-3.0.patch, YARN-7813.004.patch
>
>
> Just as inter-queue (a.k.a. cross-queue) preemption is configurable per
> queue, intra-queue (a.k.a. in-queue) preemption should be configurable per
> queue. If a queue does not have a setting for intra-queue preemption, it
> should inherit its parents value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
[ https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-7813: - Attachment: YARN-7813.004.patch > Capacity Scheduler Intra-queue Preemption should be configurable for each > queue > --- > > Key: YARN-7813 > URL: https://issues.apache.org/jira/browse/YARN-7813 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, scheduler preemption >Affects Versions: 2.9.0, 2.8.3, 3.0.0 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: YARN-7813.001.patch, YARN-7813.002.branch-3.0.patch, > YARN-7813.002.patch, YARN-7813.003.branch-2.patch, > YARN-7813.003.branch-3.0.patch, YARN-7813.004.patch > > > Just as inter-queue (a.k.a. cross-queue) preemption is configurable per > queue, intra-queue (a.k.a. in-queue) preemption should be configurable per > queue. If a queue does not have a setting for intra-queue preemption, it > should inherit its parents value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7934) [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos
[
https://issues.apache.org/jira/browse/YARN-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365038#comment-16365038
]
genericqa commented on YARN-7934:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 35 unchanged - 1 fixed = 37 total (was 36) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m
37s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7934 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910654/YARN-7934.v1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b49d595540e0 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8f66aff |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19694/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| javadoc |
[jira] [Created] (YARN-7935) Expose container's hostname to applications running within the docker container
Suma Shivaprasad created YARN-7935: -- Summary: Expose container's hostname to applications running within the docker container Key: YARN-7935 URL: https://issues.apache.org/jira/browse/YARN-7935 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Suma Shivaprasad Assignee: Suma Shivaprasad Some applications have a need to bind to the container's hostname (like Spark) which is different from the NodeManager's hostname(NM_HOST which is available as an env during container launch) when launched through Docker runtime. The container's hostname can be exposed to applications via an env CONTAINER_HOSTNAME. Another potential candidate is the container's IP but this can be addressed in a separate jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7934) [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos
[ https://issues.apache.org/jira/browse/YARN-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364966#comment-16364966 ] Carlo Curino commented on YARN-7934: [~leftnoteasy] the intention is to commit this directly to trunk, so we avoid churn, as the rest of the development will continue in YARN-7402 branch. Please check it out if you can, if none complaints and YETUS is happy, this will go straight to trunk. > [GQ] Refactor preemption calculators to allow overriding for Federation > Global Algos > > > Key: YARN-7934 > URL: https://issues.apache.org/jira/browse/YARN-7934 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Major > Attachments: YARN-7934.v1.patch > > > This Jira tracks minimal changes in the capacity scheduler preemption > mechanics that allow for sub-classing and overriding of certain behaviors, > which we use to implement federation global algorithms, e.g., in YARN-7403. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7934) [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos
[ https://issues.apache.org/jira/browse/YARN-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-7934: --- Attachment: (was: More.url) > [GQ] Refactor preemption calculators to allow overriding for Federation > Global Algos > > > Key: YARN-7934 > URL: https://issues.apache.org/jira/browse/YARN-7934 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Major > Attachments: YARN-7934.v1.patch > > > This Jira tracks minimal changes in the capacity scheduler preemption > mechanics that allow for sub-classing and overriding of certain behaviors, > which we use to implement federation global algorithms, e.g., in YARN-7403. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7934) [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos
[ https://issues.apache.org/jira/browse/YARN-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-7934: --- Attachment: YARN-7934.v1.patch > [GQ] Refactor preemption calculators to allow overriding for Federation > Global Algos > > > Key: YARN-7934 > URL: https://issues.apache.org/jira/browse/YARN-7934 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Major > Attachments: More.url, YARN-7934.v1.patch > > > This Jira tracks minimal changes in the capacity scheduler preemption > mechanics that allow for sub-classing and overriding of certain behaviors, > which we use to implement federation global algorithms, e.g., in YARN-7403. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7934) [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos
[ https://issues.apache.org/jira/browse/YARN-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-7934: --- Attachment: More.url > [GQ] Refactor preemption calculators to allow overriding for Federation > Global Algos > > > Key: YARN-7934 > URL: https://issues.apache.org/jira/browse/YARN-7934 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Major > Attachments: More.url > > > This Jira tracks minimal changes in the capacity scheduler preemption > mechanics that allow for sub-classing and overriding of certain behaviors, > which we use to implement federation global algorithms, e.g., in YARN-7403. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7677) Docker image cannot set HADOOP_CONF_DIR
[
https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364945#comment-16364945
]
Jim Brennan commented on YARN-7677:
---
{quote}I noticed that sanitizeWindowsEnv no longer adds CLASSPATH to the set of
NM vars as it did in the previous patch. Intentional?
{quote}
Yes. That was intentional. I don't think it should be adding it.
sanitizeWindowsEnv only adds to an existing CLASSPATH to make it work in the
Windows Env. It seems to me like the CLASSPATH belongs with the user
variables, after the NM variables.
I think this becomes more important when we address YARN-5714 - if we preserve
the order of the ContainerLaunchContext (user) variables, I don't think we'd
want to pull the CLASSPATH out and define it with the NM variables, potentially
reordering it with respect to other user variables. That said, there is
currently no guarantee about the relative ordering of user-defined variables.
> Docker image cannot set HADOOP_CONF_DIR
> ---
>
> Key: YARN-7677
> URL: https://issues.apache.org/jira/browse/YARN-7677
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Eric Badger
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-7677.001.patch, YARN-7677.002.patch,
> YARN-7677.003.patch, YARN-7677.004.patch, YARN-7677.005.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether
> it's set by the user or not. It completely bypasses the whitelist and so
> there is no way for a task to not have {{HADOOP_CONF_DIR}} set. This causes
> problems in the Docker use case where Docker containers will set up their own
> environment and have their own {{HADOOP_CONF_DIR}} preset in the image
> itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7934) [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos
[ https://issues.apache.org/jira/browse/YARN-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino reassigned YARN-7934: -- Assignee: Carlo Curino > [GQ] Refactor preemption calculators to allow overriding for Federation > Global Algos > > > Key: YARN-7934 > URL: https://issues.apache.org/jira/browse/YARN-7934 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Major > > This Jira tracks minimal changes in the capacity scheduler preemption > mechanics that allow for sub-classing and overriding of certain behaviors, > which we use to implement federation global algorithms, e.g., in YARN-7403. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7934) [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos
Carlo Curino created YARN-7934: -- Summary: [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos Key: YARN-7934 URL: https://issues.apache.org/jira/browse/YARN-7934 Project: Hadoop YARN Issue Type: Sub-task Reporter: Carlo Curino This Jira tracks minimal changes in the capacity scheduler preemption mechanics that allow for sub-classing and overriding of certain behaviors, which we use to implement federation global algorithms, e.g., in YARN-7403. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) [PERF/TEST] Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6528: --- Summary: [PERF/TEST] Add JMX metrics for Plan Follower and Agent Placement and Plan Operations (was: Add JMX metrics for Plan Follower and Agent Placement and Plan Operations) > [PERF/TEST] Add JMX metrics for Plan Follower and Agent Placement and Plan > Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sean Po >Assignee: Xiaohua (Victor) Liang >Priority: Major > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch, > YARN-6528.v006.patch, YARN-6528.v007.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7677) Docker image cannot set HADOOP_CONF_DIR
[
https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364890#comment-16364890
]
Jason Lowe commented on YARN-7677:
--
Thanks for updating the patch! Overall looks good, but I noticed that
sanitizeWindowsEnv no longer adds CLASSPATH to the set of NM vars as it did in
the previous patch. Intentional?
> Docker image cannot set HADOOP_CONF_DIR
> ---
>
> Key: YARN-7677
> URL: https://issues.apache.org/jira/browse/YARN-7677
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Eric Badger
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-7677.001.patch, YARN-7677.002.patch,
> YARN-7677.003.patch, YARN-7677.004.patch, YARN-7677.005.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether
> it's set by the user or not. It completely bypasses the whitelist and so
> there is no way for a task to not have {{HADOOP_CONF_DIR}} set. This causes
> problems in the Docker use case where Docker containers will set up their own
> environment and have their own {{HADOOP_CONF_DIR}} preset in the image
> itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7933) [atsv2 read acls] Add TimelineWriter#writeDomain
Vrushali C created YARN-7933: Summary: [atsv2 read acls] Add TimelineWriter#writeDomain Key: YARN-7933 URL: https://issues.apache.org/jira/browse/YARN-7933 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vrushali C Add an API TimelineWriter#writeDomain for writing the domain info -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364848#comment-16364848 ] Xiaohua (Victor) Liang commented on YARN-6528: -- Sure. I will post a v008 patch first and see what checkstyle violations there are (I had to make some code change to merge the v007 patch with the current trunk, so it is possible that I have introduced a few more checkstyle issues). After that, I would clean everything up and do another patch. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sean Po >Assignee: Xiaohua (Victor) Liang >Priority: Major > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch, > YARN-6528.v006.patch, YARN-6528.v007.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7932) [atsv2 read acls] Add TimelineDomain
Vrushali C created YARN-7932: Summary: [atsv2 read acls] Add TimelineDomain Key: YARN-7932 URL: https://issues.apache.org/jira/browse/YARN-7932 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vrushali C Assignee: Rohith Sharma K S Need to have a TimelineDomain class in atsv2 that can contain the acls info for entities. TimelineEntity will contain TimelineDomain as a member -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7932) [atsv2 read acls] Add TimelineDomain to entity
[ https://issues.apache.org/jira/browse/YARN-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-7932: - Summary: [atsv2 read acls] Add TimelineDomain to entity (was: [atsv2 read acls] Add TimelineDomain) > [atsv2 read acls] Add TimelineDomain to entity > -- > > Key: YARN-7932 > URL: https://issues.apache.org/jira/browse/YARN-7932 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Rohith Sharma K S >Priority: Major > > > Need to have a TimelineDomain class in atsv2 that can contain the acls info > for entities. TimelineEntity will contain TimelineDomain as a member -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7931) [atsv2 read acls] Include domain table creation as part of schema creator
Vrushali C created YARN-7931: Summary: [atsv2 read acls] Include domain table creation as part of schema creator Key: YARN-7931 URL: https://issues.apache.org/jira/browse/YARN-7931 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vrushali C Assignee: Vrushali C Update the schema creator to create a domain table to store timeline entity domain info. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364783#comment-16364783 ] Konstantinos Karanasos commented on YARN-7920: -- Re: configuration, I think it is simpler at the moment to leave it as is (i.e., placement-processor/scheduler). I chatted offline with Arun and he is fine with is as well. Re: the .md.vm format, I remember [~cheersyang] used this format for the opportunistic containers doc that I had written, so I thought it is preferable. Weiwei, any advantages in this format? Re: the naming, I think all is fine now apart from the NonePlacementProcessor that sounds weird. Looking at it better, indeed Default is not okay (initially I thought it is a no-op, but in fact it rejects scheduling requests). Any of the following is fine with me: DisabledConstraintProcessor, UnsupportedConstraintProcessor, RejectConstraintProcessor? >From a quick look, documentation looks fine. I can make a pass over it before >I commit the patch. > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7677) Docker image cannot set HADOOP_CONF_DIR
[
https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364778#comment-16364778
]
Jim Brennan commented on YARN-7677:
---
The remaining check-style issue is for adding an 8th parameter to
writeLaunchEnv().
[~jlowe], please review.
> Docker image cannot set HADOOP_CONF_DIR
> ---
>
> Key: YARN-7677
> URL: https://issues.apache.org/jira/browse/YARN-7677
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Eric Badger
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-7677.001.patch, YARN-7677.002.patch,
> YARN-7677.003.patch, YARN-7677.004.patch, YARN-7677.005.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether
> it's set by the user or not. It completely bypasses the whitelist and so
> there is no way for a task to not have {{HADOOP_CONF_DIR}} set. This causes
> problems in the Docker use case where Docker containers will set up their own
> environment and have their own {{HADOOP_CONF_DIR}} preset in the image
> itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7677) Docker image cannot set HADOOP_CONF_DIR
[
https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364771#comment-16364771
]
genericqa commented on YARN-7677:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 162 unchanged - 3 fixed = 164 total (was 165) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
17s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
20s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7677 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910619/YARN-7677.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a41262b6c071 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1f20f43 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
[
https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364766#comment-16364766
]
Eric Payne commented on YARN-7813:
--
I reverted the checkins for trunk, branch-3.1, and branch-3.0 because this
patch is affecting the behavior of Auto Queue creation in the
{{TestCapacitySchedulerAutoCreatedQueuePreemption}}
> Capacity Scheduler Intra-queue Preemption should be configurable for each
> queue
> ---
>
> Key: YARN-7813
> URL: https://issues.apache.org/jira/browse/YARN-7813
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.9.0, 2.8.3, 3.0.0
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Attachments: YARN-7813.001.patch, YARN-7813.002.branch-3.0.patch,
> YARN-7813.002.patch, YARN-7813.003.branch-2.patch,
> YARN-7813.003.branch-3.0.patch
>
>
> Just as inter-queue (a.k.a. cross-queue) preemption is configurable per
> queue, intra-queue (a.k.a. in-queue) preemption should be configurable per
> queue. If a queue does not have a setting for intra-queue preemption, it
> should inherit its parents value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
[ https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364762#comment-16364762 ] Hudson commented on YARN-7813: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13658 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13658/]) Revert "YARN-7813: Capacity Scheduler Intra-queue Preemption should be (epayne: rev bddfe42e2ccda2e6cc9d149461640ba36eca5922) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestConfigurationMutationACLPolicies.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/QueueCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/QueueInfoPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/IntraQueueCandidatesSelector.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java > Capacity Scheduler Intra-queue Preemption should be configurable for each > queue > --- > > Key: YARN-7813 > URL: https://issues.apache.org/jira/browse/YARN-7813 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, scheduler preemption >Affects Versions: 2.9.0, 2.8.3, 3.0.0 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: YARN-7813.001.patch, YARN-7813.002.branch-3.0.patch, > YARN-7813.002.patch, YARN-7813.003.branch-2.patch, > YARN-7813.003.branch-3.0.patch > > > Just as inter-queue (a.k.a. cross-queue) preemption is configurable per > queue, intra-queue (a.k.a. in-queue) preemption should be configurable per > queue. If a queue does not have a setting for intra-queue preemption, it > should inherit its parents value. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[
https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364757#comment-16364757
]
Carlo Curino commented on YARN-6528:
Thanks [~seanpo03], I am ok with the left-over checkstyle. I am assigning this
to [~lxhfirenking] who volunteered to rebase and extend this, I will mark you
both as contributor when we get to commit this.
[~lxhfirenking] please see if you can shush the checkstyle using
{{@SuppressWarnings("checkstyle:XYZ")}} , with XYZ being the right checkstyle
rule (or similar tricks).
> Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
> -
>
> Key: YARN-6528
> URL: https://issues.apache.org/jira/browse/YARN-6528
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sean Po
>Assignee: Xiaohua (Victor) Liang
>Priority: Major
> Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch,
> YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch,
> YARN-6528.v006.patch, YARN-6528.v007.patch
>
>
> YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle
> time explicitly, i.e. users can now "reserve" capacity ahead of time which is
> predictably allocated to them. In order to understand in finer detail the
> performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan
> Follower, Agent Placement and Plan Operations components of Rayon.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino reassigned YARN-6528: -- Assignee: Xiaohua (Victor) Liang (was: Carlo Curino) > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sean Po >Assignee: Xiaohua (Victor) Liang >Priority: Major > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch, > YARN-6528.v006.patch, YARN-6528.v007.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6528: --- Issue Type: Sub-task (was: Task) Parent: YARN-7402 > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sean Po >Assignee: Sean Po >Priority: Major > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch, > YARN-6528.v006.patch, YARN-6528.v007.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7677) Docker image cannot set HADOOP_CONF_DIR
[
https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364652#comment-16364652
]
Jim Brennan commented on YARN-7677:
---
Uploaded another patch with updates based on [~jlowe]'s review.
> Docker image cannot set HADOOP_CONF_DIR
> ---
>
> Key: YARN-7677
> URL: https://issues.apache.org/jira/browse/YARN-7677
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Eric Badger
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-7677.001.patch, YARN-7677.002.patch,
> YARN-7677.003.patch, YARN-7677.004.patch, YARN-7677.005.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether
> it's set by the user or not. It completely bypasses the whitelist and so
> there is no way for a task to not have {{HADOOP_CONF_DIR}} set. This causes
> problems in the Docker use case where Docker containers will set up their own
> environment and have their own {{HADOOP_CONF_DIR}} preset in the image
> itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7677) Docker image cannot set HADOOP_CONF_DIR
[
https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Brennan updated YARN-7677:
--
Attachment: YARN-7677.005.patch
> Docker image cannot set HADOOP_CONF_DIR
> ---
>
> Key: YARN-7677
> URL: https://issues.apache.org/jira/browse/YARN-7677
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Eric Badger
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-7677.001.patch, YARN-7677.002.patch,
> YARN-7677.003.patch, YARN-7677.004.patch, YARN-7677.005.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether
> it's set by the user or not. It completely bypasses the whitelist and so
> there is no way for a task to not have {{HADOOP_CONF_DIR}} set. This causes
> problems in the Docker use case where Docker containers will set up their own
> environment and have their own {{HADOOP_CONF_DIR}} preset in the image
> itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7927) TestRMWebServicesSchedulerActivities is failing
[
https://issues.apache.org/jira/browse/YARN-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364556#comment-16364556
]
Eric Payne commented on YARN-7927:
--
{{TestCapacitySchedulerAutoCreatedQueuePreemption}} may also be related
> TestRMWebServicesSchedulerActivities is failing
> ---
>
> Key: YARN-7927
> URL: https://issues.apache.org/jira/browse/YARN-7927
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.1.0, 3.2.0
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Attachments: YARN-7927.001.patch
>
>
> YARN-7813 broke TestRMWebServicesSchedulerActivities. The test needs to be
> updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7927) TestRMWebServicesSchedulerActivities is failing
[
https://issues.apache.org/jira/browse/YARN-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364514#comment-16364514
]
genericqa commented on YARN-7927:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 18s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoCreatedQueuePreemption
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7927 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910599/YARN-7927.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9ba3d9d82378 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 042ef2f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19691/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19691/testReport/ |
| Max. process+thread count | 873 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (YARN-7919) Split timelineservice-hbase module to make YARN-7346 easier
[ https://issues.apache.org/jira/browse/YARN-7919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364466#comment-16364466 ] Rohith Sharma K S commented on YARN-7919: - bq. Did you see some source files in the src/ directory? I didn't see any java file, but I see folders and subfolders i.e src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader folder > Split timelineservice-hbase module to make YARN-7346 easier > --- > > Key: YARN-7919 > URL: https://issues.apache.org/jira/browse/YARN-7919 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineservice >Affects Versions: 3.0.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7919.00.patch, YARN-7919.01.patch, > YARN-7919.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7919) Split timelineservice-hbase module to make YARN-7346 easier
[ https://issues.apache.org/jira/browse/YARN-7919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364460#comment-16364460 ] Haibo Chen commented on YARN-7919: -- Did you see some source files in the src/ directory? I applied the patch to trunk and but did not see src/ locally. I will address your coprocessor naming comment together with reviews that Vrushali may have. > Split timelineservice-hbase module to make YARN-7346 easier > --- > > Key: YARN-7919 > URL: https://issues.apache.org/jira/browse/YARN-7919 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineservice >Affects Versions: 3.0.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-7919.00.patch, YARN-7919.01.patch, > YARN-7919.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7927) TestRMWebServicesSchedulerActivities is failing
[ https://issues.apache.org/jira/browse/YARN-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-7927: - Attachment: YARN-7927.001.patch > TestRMWebServicesSchedulerActivities is failing > --- > > Key: YARN-7927 > URL: https://issues.apache.org/jira/browse/YARN-7927 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: YARN-7927.001.patch > > > YARN-7813 broke TestRMWebServicesSchedulerActivities. The test needs to be > updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7927) TestRMWebServicesSchedulerActivities is failing
[ https://issues.apache.org/jira/browse/YARN-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned YARN-7927: Assignee: Eric Payne Description: YARN-7813 broke TestRMWebServicesSchedulerActivities. The test needs to be updated. Component/s: test Summary: TestRMWebServicesSchedulerActivities is failing (was: YARN-7813 caused test failure in TestRMWebServicesSchedulerActivities ) > TestRMWebServicesSchedulerActivities is failing > --- > > Key: YARN-7927 > URL: https://issues.apache.org/jira/browse/YARN-7927 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > > YARN-7813 broke TestRMWebServicesSchedulerActivities. The test needs to be > updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7789) Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364293#comment-16364293 ] Hudson commented on YARN-7789: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13654 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13654/]) YARN-7789. Should fail RM if 3rd resource type is configured but RM uses (wangda: rev 042ef2fa7bcc22e3ca4eb8205c34d83e594bc7de) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerWithMultiResourceTypes.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java > Should fail RM if 3rd resource type is configured but RM uses > DefaultResourceCalculator > --- > > Key: YARN-7789 > URL: https://issues.apache.org/jira/browse/YARN-7789 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sumana Sathish >Assignee: Zian Chen >Priority: Critical > Fix For: 3.1.0 > > Attachments: YARN-7789.001.patch, YARN-7789.002.patch > > > We may need to revisit this behavior: Currently, RM doesn't fail if 3rd > resource type is configured, allocated containers will be automatically > assigned minimum allocation for all resource types except memory, this makes > really hard for troubleshooting. I prefer to fail RM if 3rd or more resource > type is configured inside resource-types.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7789) Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7789: - Fix Version/s: 3.1.0 > Should fail RM if 3rd resource type is configured but RM uses > DefaultResourceCalculator > --- > > Key: YARN-7789 > URL: https://issues.apache.org/jira/browse/YARN-7789 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sumana Sathish >Assignee: Zian Chen >Priority: Critical > Fix For: 3.1.0 > > Attachments: YARN-7789.001.patch, YARN-7789.002.patch > > > We may need to revisit this behavior: Currently, RM doesn't fail if 3rd > resource type is configured, allocated containers will be automatically > assigned minimum allocation for all resource types except memory, this makes > really hard for troubleshooting. I prefer to fail RM if 3rd or more resource > type is configured inside resource-types.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications
[
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364266#comment-16364266
]
Gergo Repas commented on YARN-5028:
---
[~yufeigu] - Right, the null check can be abandoned (I'll upload a new patch
when the rest of the questions are cleared).
Regarding your second question: I'm using a new instance to trim down the state
as much as possible (also, I had to provide a non-null reference, otherwise an
exception is thrown at the line you quoted from createAndPopulateNewRMApp()).
> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch,
> YARN-5028.002.patch
>
>
> RMStateStore stores enough information to recover applications in case of a
> restart. The store also retains this information for completed applications
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all
> the information we store today to serve application status; for instance, we
> don't need the {{ApplicationSubmissionContext}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[
https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364248#comment-16364248
]
Sunil G commented on YARN-7920:
---
I think a name like {{placement-processor}} seems more simple for now. We can
add more detailed description for user configuration to explain the meaning a
bit more.
> Cleanup configuration of PlacementConstraints
> -
>
> Key: YARN-7920
> URL: https://issues.apache.org/jira/browse/YARN-7920
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-7920.001.patch, YARN-7920.002.patch,
> YARN-7920.003.patch, YARN-7920.004.patch
>
>
> Currently it is very confusing to have the two configs in two different files
> (yarn-site.xml and capacity-scheduler.xml).
>
> Maybe a better approach is: we can delete the scheduling-request.allowed in
> CS, and update placement-constraints configs in yarn-site.xml a bit:
>
> - Remove placement-constraints.enabled, and add a new
> placement-constraints.handler, by default is none, and other acceptable
> values are a. external-processor (since algorithm is too generic to me), b.
> scheduler.
> - And add a new PlacementProcessor just to pass SchedulingRequest to
> scheduler without any modifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7813) Capacity Scheduler Intra-queue Preemption should be configurable for each queue
[
https://issues.apache.org/jira/browse/YARN-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364217#comment-16364217
]
Eric Payne commented on YARN-7813:
--
{code:java}
Removing intermediate container 0bc484afac12
Step 14/33 : RUN mkdir -p /opt/findbugs && curl -L -s -S
https://sourceforge.net/projects/findbugs/files/findbugs/3.0.1/findbugs-noUpdateChecks-3.0.1.tar.gz/download
-o /opt/findbugs.tar.gz && tar xzf /opt/findbugs.tar.gz
--strip-components 1 -C /opt/findbugs
---> Running in 1f76e2c688d5
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c mkdir -p /opt/findbugs && curl -L -s -S
https://sourceforge.net/projects/findbugs/files/findbugs/3.0.1/findbugs-noUpdateChecks-3.0.1.tar.gz/download
-o /opt/findbugs.tar.gz && tar xzf /opt/findbugs.tar.gz
--strip-components 1 -C /opt/findbugs' returned a non-zero code: 2
{code}
The pre-commit build failed for {{YARN-7813.003.branch-2.patch}} with {{gzip:
stdin: not in gzip format}}. I don't think that is related to the patch itself,
so I'm restarting the build.
> Capacity Scheduler Intra-queue Preemption should be configurable for each
> queue
> ---
>
> Key: YARN-7813
> URL: https://issues.apache.org/jira/browse/YARN-7813
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.9.0, 2.8.3, 3.0.0
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Attachments: YARN-7813.001.patch, YARN-7813.002.branch-3.0.patch,
> YARN-7813.002.patch, YARN-7813.003.branch-2.patch,
> YARN-7813.003.branch-3.0.patch
>
>
> Just as inter-queue (a.k.a. cross-queue) preemption is configurable per
> queue, intra-queue (a.k.a. in-queue) preemption should be configurable per
> queue. If a queue does not have a setting for intra-queue preemption, it
> should inherit its parents value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7574) Add support for Node Labels on Auto Created Leaf Queue Template
[
https://issues.apache.org/jira/browse/YARN-7574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364225#comment-16364225
]
Sunil G commented on YARN-7574:
---
Thanks [~suma.shivaprasad] for the patch.
Few comments
1. {{//map of queueName-> partition->{leaf queue's state}} seems not clean to
me. (GuaranteedOrZeroCapacityOverTimePolicy). We should ideally derive queues
from partition.
2. {{//Map of LeafQueue->QueueCapacities - keep adding the computed}} We
already have QueueCapacities per queue. So do we need such a mapping ?
> Add support for Node Labels on Auto Created Leaf Queue Template
> ---
>
> Key: YARN-7574
> URL: https://issues.apache.org/jira/browse/YARN-7574
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
> Attachments: YARN-7574.1.patch, YARN-7574.2.patch, YARN-7574.3.patch
>
>
> YARN-7473 adds support for auto created leaf queues to inherit node labels
> capacities from parent queues. Howebver there is no support for leaf queue
> template to allow different configured capacities for different node labels.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364214#comment-16364214 ] Wangda Tan commented on YARN-7920: -- + [~cheersyang] > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7838) Support AND/OR constraints in Distributed Shell
[
https://issues.apache.org/jira/browse/YARN-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364094#comment-16364094
]
Arun Suresh commented on YARN-7838:
---
Ah - thought is wasn't included. Cool, i've updated the fix version to reflect
the same.
> Support AND/OR constraints in Distributed Shell
> ---
>
> Key: YARN-7838
> URL: https://issues.apache.org/jira/browse/YARN-7838
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Labels: constraints
> Fix For: 3.1.0
>
> Attachments: YARN-7838.001.patch, YARN-7838.002.patch,
> YARN-7838.003.patch, YARN-7838.prelim.patch
>
>
> Extending DS placement spec syntax to support AND/OR constraints, something
> like
> {code}
> // simple
> -placement_spec foo=4,AND(NOTIN,NODE,foo:NOTIN,NODE,bar)
> // nested
> -placement_spec foo=4,AND(NOTIN,NODE,foo:OR(IN,NODE,moo:IN,NODE,bar))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7838) Support AND/OR constraints in Distributed Shell
[
https://issues.apache.org/jira/browse/YARN-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-7838:
--
Fix Version/s: (was: 3.2.0)
3.1.0
> Support AND/OR constraints in Distributed Shell
> ---
>
> Key: YARN-7838
> URL: https://issues.apache.org/jira/browse/YARN-7838
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Labels: constraints
> Fix For: 3.1.0
>
> Attachments: YARN-7838.001.patch, YARN-7838.002.patch,
> YARN-7838.003.patch, YARN-7838.prelim.patch
>
>
> Extending DS placement spec syntax to support AND/OR constraints, something
> like
> {code}
> // simple
> -placement_spec foo=4,AND(NOTIN,NODE,foo:NOTIN,NODE,bar)
> // nested
> -placement_spec foo=4,AND(NOTIN,NODE,foo:OR(IN,NODE,moo:IN,NODE,bar))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7838) Support AND/OR constraints in Distributed Shell
[
https://issues.apache.org/jira/browse/YARN-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364087#comment-16364087
]
Weiwei Yang commented on YARN-7838:
---
Hi [~asuresh], [~leftnoteasy], this was already committed to branch-3.1. So it
should be included in 3.1.0 already right?
> Support AND/OR constraints in Distributed Shell
> ---
>
> Key: YARN-7838
> URL: https://issues.apache.org/jira/browse/YARN-7838
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Labels: constraints
> Fix For: 3.2.0
>
> Attachments: YARN-7838.001.patch, YARN-7838.002.patch,
> YARN-7838.003.patch, YARN-7838.prelim.patch
>
>
> Extending DS placement spec syntax to support AND/OR constraints, something
> like
> {code}
> // simple
> -placement_spec foo=4,AND(NOTIN,NODE,foo:NOTIN,NODE,bar)
> // nested
> -placement_spec foo=4,AND(NOTIN,NODE,foo:OR(IN,NODE,moo:IN,NODE,bar))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7838) Support AND/OR constraints in Distributed Shell
[
https://issues.apache.org/jira/browse/YARN-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364085#comment-16364085
]
Wangda Tan commented on YARN-7838:
--
Since this is a self-contained code, and YARN-7822 is part of 3.1.0, I'm fine
with backporting put this to branch-3.1.
> Support AND/OR constraints in Distributed Shell
> ---
>
> Key: YARN-7838
> URL: https://issues.apache.org/jira/browse/YARN-7838
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Labels: constraints
> Fix For: 3.2.0
>
> Attachments: YARN-7838.001.patch, YARN-7838.002.patch,
> YARN-7838.003.patch, YARN-7838.prelim.patch
>
>
> Extending DS placement spec syntax to support AND/OR constraints, something
> like
> {code}
> // simple
> -placement_spec foo=4,AND(NOTIN,NODE,foo:NOTIN,NODE,bar)
> // nested
> -placement_spec foo=4,AND(NOTIN,NODE,foo:OR(IN,NODE,moo:IN,NODE,bar))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[
https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364083#comment-16364083
]
Wangda Tan commented on YARN-7920:
--
{quote}If you don't like priority/placement-optimized, how about calling the
key "placement strategy" and the values "reject-after-retry" and
"queue-till-satisfied" where reject implies the processor
{quote}
This is even worse than {{priority-optimized}} / {{placement-optimized}}, why
it is important to user to understand a scheduling request will be rejected
after retry, and this is subject to change as well. I would still prefer the
"scheduler", "placement-processor" value. Just like the AMS processor config.
[~sunilg] / [~kkaranasos], could you share your thoughts here?
> Cleanup configuration of PlacementConstraints
> -
>
> Key: YARN-7920
> URL: https://issues.apache.org/jira/browse/YARN-7920
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-7920.001.patch, YARN-7920.002.patch,
> YARN-7920.003.patch, YARN-7920.004.patch
>
>
> Currently it is very confusing to have the two configs in two different files
> (yarn-site.xml and capacity-scheduler.xml).
>
> Maybe a better approach is: we can delete the scheduling-request.allowed in
> CS, and update placement-constraints configs in yarn-site.xml a bit:
>
> - Remove placement-constraints.enabled, and add a new
> placement-constraints.handler, by default is none, and other acceptable
> values are a. external-processor (since algorithm is too generic to me), b.
> scheduler.
> - And add a new PlacementProcessor just to pass SchedulingRequest to
> scheduler without any modifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7838) Support AND/OR constraints in Distributed Shell
[
https://issues.apache.org/jira/browse/YARN-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364039#comment-16364039
]
Arun Suresh commented on YARN-7838:
---
[~leftnoteasy], can we consider this for 3.1.0 ? Don't think this is really a
disruptive change, and it is a useful inclusion, especially for demo purposes.
Also note the constraint checker changes, implemented in YARN-7822, will anyway
be available in 3.1.0 and will not be testable/demo-able until 3.1.x
> Support AND/OR constraints in Distributed Shell
> ---
>
> Key: YARN-7838
> URL: https://issues.apache.org/jira/browse/YARN-7838
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: distributed-shell
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Labels: constraints
> Fix For: 3.2.0
>
> Attachments: YARN-7838.001.patch, YARN-7838.002.patch,
> YARN-7838.003.patch, YARN-7838.prelim.patch
>
>
> Extending DS placement spec syntax to support AND/OR constraints, something
> like
> {code}
> // simple
> -placement_spec foo=4,AND(NOTIN,NODE,foo:NOTIN,NODE,bar)
> // nested
> -placement_spec foo=4,AND(NOTIN,NODE,foo:OR(IN,NODE,moo:IN,NODE,bar))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[
https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363849#comment-16363849
]
genericqa commented on YARN-7920:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
22s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
9s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 6s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 14 new + 409 unchanged - 0 fixed = 423 total (was 409) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
18s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 57s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-yarn-site in the patch passed.
[jira] [Commented] (YARN-7930) Add configuration to initialize RM with configured labels.
[
https://issues.apache.org/jira/browse/YARN-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363740#comment-16363740
]
genericqa commented on YARN-7930:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
29s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 5s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 267 unchanged - 0 fixed = 268 total (was 267) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 42s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 8s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| | hadoop.yarn.nodelabels.TestCommonNodeLabelsManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7930 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910535/YARN-7930.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b8ef9fe5abb7 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[
https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363733#comment-16363733
]
genericqa commented on YARN-7920:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 57s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 409 unchanged - 0 fixed = 414 total (was 409) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 53s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
20s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 11s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 32s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-yarn-site in the patch passed.
[jira] [Commented] (YARN-5973) TestCapacitySchedulerSurgicalPreemption sometimes fails
[
https://issues.apache.org/jira/browse/YARN-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363676#comment-16363676
]
Dibyendu Karmakar commented on YARN-5973:
-
I have created a patch for this issue (attached). I have run the test case in
a loop for 100 times. Working fine.
> TestCapacitySchedulerSurgicalPreemption sometimes fails
> ---
>
> Key: YARN-5973
> URL: https://issues.apache.org/jira/browse/YARN-5973
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0
>Reporter: Eric Payne
>Priority: Minor
> Attachments: YARN-5973-branch-2.8.0.001.patch
>
>
> The tests in {{TestCapacitySchedulerSurgicalPreemption}} appear to be racy.
> They often pass, but the following errors sometimes occur:
> {noformat}
> testSimpleSurgicalPreemption(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption)
> Time elapsed: 14.671 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.fail(Assert.java:95)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerPreemptionTestBase.waitNumberOfLiveContainersFromApp(CapacitySchedulerPreemptionTestBase.java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption.testSimpleSurgicalPreemption(TestCapacitySchedulerSurgicalPreemption.java:143)
> {noformat}
> {noformat}
> testSurgicalPreemptionWithAvailableResource(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption)
> Time elapsed: 9.503 sec <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption.testSurgicalPreemptionWithAvailableResource(TestCapacitySchedulerSurgicalPreemption.java:220)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5973) TestCapacitySchedulerSurgicalPreemption sometimes fails
[
https://issues.apache.org/jira/browse/YARN-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dibyendu Karmakar updated YARN-5973:
Attachment: YARN-5973-branch-2.8.0.001.patch
> TestCapacitySchedulerSurgicalPreemption sometimes fails
> ---
>
> Key: YARN-5973
> URL: https://issues.apache.org/jira/browse/YARN-5973
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0
>Reporter: Eric Payne
>Priority: Minor
> Attachments: YARN-5973-branch-2.8.0.001.patch
>
>
> The tests in {{TestCapacitySchedulerSurgicalPreemption}} appear to be racy.
> They often pass, but the following errors sometimes occur:
> {noformat}
> testSimpleSurgicalPreemption(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption)
> Time elapsed: 14.671 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.fail(Assert.java:95)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerPreemptionTestBase.waitNumberOfLiveContainersFromApp(CapacitySchedulerPreemptionTestBase.java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption.testSimpleSurgicalPreemption(TestCapacitySchedulerSurgicalPreemption.java:143)
> {noformat}
> {noformat}
> testSurgicalPreemptionWithAvailableResource(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption)
> Time elapsed: 9.503 sec <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption.testSurgicalPreemptionWithAvailableResource(TestCapacitySchedulerSurgicalPreemption.java:220)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363667#comment-16363667 ] Arun Suresh commented on YARN-7920: --- bq. More likely, the two handler will be merged in the future into the same one and user doesn't need to choose. Agreed that both approaches might merge in the future. But at the moment, I strongly believe we should give it a more user focused value (since end users don't care HOW it is implemented - only what each option offers). If you don't like priority/placement-optimized, how about calling the key "placement strategy" and the values "reject-after-retry" and "queue-till-satisfied" where reject implies the processor - since the requests are rejected if not satisfied in a configurable number of AM heartbeats, while the scheduler approach keeps them in queue. Everything else looks fine. > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7920) Cleanup configuration of PlacementConstraints
[
https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363645#comment-16363645
]
Wangda Tan commented on YARN-7920:
--
Thanks [~asuresh],
Addressed:
{quote} * the \{{ amsProcessingChain.init(rmContext, null);}} call should be in
the {{initializeProcessingChain}} method.
* With regard to the {{SchedulerPlacementProcessor}}, we are assuming that if
enabled, then placement constraints CANNOT be specified via the registerAM
call.. Technically, you can still specify constraints in the register call -
the schedulingRequest just overrides it.{quote}
For:
{quote}maybe we should call it "priority-optimized" and "placement-optimized" ?
Thoughts ?
{quote}
I would prefer not, the processor handler could improve priority support and
the scheduler handler could improve placement support. More likely, the two
handler will be merged in the future into the same one and user doesn't need to
choose. Let's keep the name open so we don't have to find a different name once
the functionalities of each handler get improved.
Attached ver.4, please review.
> Cleanup configuration of PlacementConstraints
> -
>
> Key: YARN-7920
> URL: https://issues.apache.org/jira/browse/YARN-7920
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-7920.001.patch, YARN-7920.002.patch,
> YARN-7920.003.patch, YARN-7920.004.patch
>
>
> Currently it is very confusing to have the two configs in two different files
> (yarn-site.xml and capacity-scheduler.xml).
>
> Maybe a better approach is: we can delete the scheduling-request.allowed in
> CS, and update placement-constraints configs in yarn-site.xml a bit:
>
> - Remove placement-constraints.enabled, and add a new
> placement-constraints.handler, by default is none, and other acceptable
> values are a. external-processor (since algorithm is too generic to me), b.
> scheduler.
> - And add a new PlacementProcessor just to pass SchedulingRequest to
> scheduler without any modifications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7920) Cleanup configuration of PlacementConstraints
[ https://issues.apache.org/jira/browse/YARN-7920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7920: - Attachment: YARN-7920.004.patch > Cleanup configuration of PlacementConstraints > - > > Key: YARN-7920 > URL: https://issues.apache.org/jira/browse/YARN-7920 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7920.001.patch, YARN-7920.002.patch, > YARN-7920.003.patch, YARN-7920.004.patch > > > Currently it is very confusing to have the two configs in two different files > (yarn-site.xml and capacity-scheduler.xml). > > Maybe a better approach is: we can delete the scheduling-request.allowed in > CS, and update placement-constraints configs in yarn-site.xml a bit: > > - Remove placement-constraints.enabled, and add a new > placement-constraints.handler, by default is none, and other acceptable > values are a. external-processor (since algorithm is too generic to me), b. > scheduler. > - And add a new PlacementProcessor just to pass SchedulingRequest to > scheduler without any modifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7930) Add configuration to initialize RM with configured labels.
[ https://issues.apache.org/jira/browse/YARN-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7930: Attachment: YARN-7930.002.patch > Add configuration to initialize RM with configured labels. > -- > > Key: YARN-7930 > URL: https://issues.apache.org/jira/browse/YARN-7930 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7930.001.patch, YARN-7930.002.patch > > > At present, the only way to create labels is using admin API. Sometimes, > there is a requirement to start the cluster with pre-configured node labels. > This Jira introduces yarn configurations to start RM with predefined node > labels. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
