[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700031#comment-16700031 ] Weiwei Yang commented on YARN-9037: --- Hi [~sunilg] The patch overall looks good. Something I am not quite sure about ResourceUtils has some other public methods, e.g getResourceTypesArray(), this is used in DomainResourceFairnessPolicy to compute shares, would that cause issue when the array has some non-countable resource? Apart from that, can we add some UT for verification? E.g to create some resource requests with such tag, verify the get methods as well as the allocation result. Thanks > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-9037-002.patch, YARN-9037.001.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695864#comment-16695864 ] Suma Shivaprasad edited comment on YARN-9034 at 11/27/18 7:24 AM: -- +1. Will commit this shortly was (Author: suma.shivaprasad): +1 > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch, YARN-9034.04.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9061) Improve the GPU/FPGA module log message of container-executor
[
https://issues.apache.org/jira/browse/YARN-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700029#comment-16700029
]
Hadoop QA commented on YARN-9061:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
30m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
16s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9061 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949610/YARN-9061-trunk.002.patch
|
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 15ffc73a9056 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33e0df4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22722/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22722/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Improve the GPU/FPGA module log message of container-executor
> -
>
> Key: YARN-9061
> URL: https://issues.apache.org/jira/browse/YARN-9061
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-9061-trunk.001.patch, YARN-9061-trunk.002.patch
>
>
> The log message is not clear when options value is missing.
> {code:java}
> fprintf(ERRORFILE, "is not specified, skip cgroups call.\n");{code}
--
This message was sent by Atlassian
[jira] [Commented] (YARN-9056) Yarn Service Upgrade: Instance state changes from UPGRADING to READY without performing a readiness check
[
https://issues.apache.org/jira/browse/YARN-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700014#comment-16700014
]
Hadoop QA commented on YARN-9056:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core:
The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was 41) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
7s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9056 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949609/YARN-9056.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 189b0841b6d8 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33e0df4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22721/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22721/testReport/ |
| Max. process+thread count | 772 (vs. ulimit of 1) |
| modules | C:
[jira] [Updated] (YARN-9061) Improve the GPU/FPGA module log message of container-executor
[
https://issues.apache.org/jira/browse/YARN-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-9061:
---
Attachment: YARN-9061-trunk.002.patch
> Improve the GPU/FPGA module log message of container-executor
> -
>
> Key: YARN-9061
> URL: https://issues.apache.org/jira/browse/YARN-9061
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-9061-trunk.001.patch, YARN-9061-trunk.002.patch
>
>
> The log message is not clear when options value is missing.
> {code:java}
> fprintf(ERRORFILE, "is not specified, skip cgroups call.\n");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[
https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699978#comment-16699978
]
Hadoop QA commented on YARN-9037:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 13s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 88 unchanged - 0 fixed = 91 total (was 88) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
46s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
33s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 22s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9037 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949600/YARN-9037-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6a920ba324fb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Updated] (YARN-9056) Yarn Service Upgrade: Instance state changes from UPGRADING to READY without performing a readiness check
[ https://issues.apache.org/jira/browse/YARN-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-9056: Attachment: YARN-9056.001.patch > Yarn Service Upgrade: Instance state changes from UPGRADING to READY without > performing a readiness check > - > > Key: YARN-9056 > URL: https://issues.apache.org/jira/browse/YARN-9056 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Critical > Attachments: YARN-9056.001.patch > > > Currently, when an instance is upgraded, the state of the instance changes to > UPGRADING. Once the NM informs AM that upgrade is finished, the state of the > instance changes to STABLE. > The instance state should be changes to STABLE only once readiness check > succeeds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9061) Improve the GPU/FPGA module log message of container-executor
[
https://issues.apache.org/jira/browse/YARN-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699959#comment-16699959
]
Hadoop QA commented on YARN-9061:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
31m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
37s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 37s{color} |
{color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 37s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 39s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9061 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949605/YARN-9061-trunk.001.patch
|
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 9d4a2f96be85 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33e0df4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/22720/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/22720/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/22720/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22720/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22720/testReport/ |
| Max. process+thread count | 411 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
[jira] [Commented] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation in native container-executor
[
https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699946#comment-16699946
]
Hadoop QA commented on YARN-9060:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
32m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 43s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9060 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949604/YARN-9060-trunk.001.patch
|
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 8ec05c83db2e 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33e0df4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22719/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22719/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22719/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> [YARN-8851] Phase 1 - Support device isolation in native container-executor
> ---
>
> Key: YARN-9060
> URL: https://issues.apache.org/jira/browse/YARN-9060
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9060-trunk.001.patch
>
>
> Due to the cgroups v1 implementation policy in linux kernel, we cannot update
> the value of the device cgroups controller unless we have the
[jira] [Created] (YARN-9061) Improve the GPU/FPGA module log message of container-executor
Zhankun Tang created YARN-9061:
--
Summary: Improve the GPU/FPGA module log message of
container-executor
Key: YARN-9061
URL: https://issues.apache.org/jira/browse/YARN-9061
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Zhankun Tang
Assignee: Zhankun Tang
The log message is not clear when options value is missing.
{code:java}
fprintf(ERRORFILE, "is not specified, skip cgroups call.\n");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9061) Improve the GPU/FPGA module log message of container-executor
[
https://issues.apache.org/jira/browse/YARN-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-9061:
---
Attachment: YARN-9061-trunk.001.patch
> Improve the GPU/FPGA module log message of container-executor
> -
>
> Key: YARN-9061
> URL: https://issues.apache.org/jira/browse/YARN-9061
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-9061-trunk.001.patch
>
>
> The log message is not clear when options value is missing.
> {code:java}
> fprintf(ERRORFILE, "is not specified, skip cgroups call.\n");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation in native container-executor
[ https://issues.apache.org/jira/browse/YARN-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-9060: --- Attachment: YARN-9060-trunk.001.patch > [YARN-8851] Phase 1 - Support device isolation in native container-executor > --- > > Key: YARN-9060 > URL: https://issues.apache.org/jira/browse/YARN-9060 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-9060-trunk.001.patch > > > Due to the cgroups v1 implementation policy in linux kernel, we cannot update > the value of the device cgroups controller unless we have the root permission > ([here|https://github.com/torvalds/linux/blob/6f0d349d922ba44e4348a17a78ea51b7135965b1/security/device_cgroup.c#L604]). > So we need to support this in container-executor for Java layer to invoke. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9058) YARN service fail to launch due to CSI changes
[
https://issues.apache.org/jira/browse/YARN-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699911#comment-16699911
]
Hadoop QA commented on YARN-9058:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
39m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9058 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949599/YARN-9058.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux 66a3aa7016f1 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33e0df4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22717/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C: hadoop-assemblies U: hadoop-assemblies |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22717/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> YARN service fail to launch due to CSI changes
> --
>
> Key: YARN-9058
> URL: https://issues.apache.org/jira/browse/YARN-9058
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-9058.001.patch
>
>
> YARN service AM fails to launch with error message:
> {code}
> 2018-11-26
[jira] [Commented] (YARN-9053) Support set environment variables for Docker Containers In nonEntryPoint mode
[
https://issues.apache.org/jira/browse/YARN-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699908#comment-16699908
]
Charo Zhang commented on YARN-9053:
---
[~Jim_Brennan] In the non-entrypoint case, we summit a GPU application by
{code:java}
yarn jar \
/data/bigdata/hadoop/client/hadoop-yarn-applications-distributedshell-3.1.1.jar
-jar \
/data/bigdata/hadoop/client/hadoop-yarn-applications-distributedshell-3.1.1.jar
\
-shell_env YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE=false \
-shell_env YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK=bridge \
-shell_env YARN_CONTAINER_RUNTIME_TYPE=docker \
-shell_env
hadoop_security_authentication_tbds_securekey=jNCmfGkKkvpRv48aN7X43UKRnCzyvYda \
-shell_env
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=docker.oa.com:8080/public/tensorflow1.8.0-gpu-py2:yarn311
\
-shell_env YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/cephfs/group/:/cephfs/group:rw
-timeout 214748367 \
-shell_command "tailf /etc/hosts" -container_resources
memory-mb=1024,vcores=1,yarn.io/gpu=1 -num_containers 1
{code}
When we access container by docker exec, we can't find
"hadoop_security_authentication_tbds_securekey " variable in "env" list. If we
set YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE=true, the
"hadoop_security_authentication_tbds_securekey " existed in "env" list.
And "hadoop_security_authentication_tbds_securekey " has configured in
white-listed variables.
Besides, we don't think white-listed way is good, because it must restart
nodemanger when white-listed be changed, and we can not make sure how many
environment variables will be used from the start.
> Support set environment variables for Docker Containers In nonEntryPoint mode
> -
>
> Key: YARN-9053
> URL: https://issues.apache.org/jira/browse/YARN-9053
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Priority: Major
> Labels: Docker
> Attachments: YARN-9053.patch
>
>
> In yarn 3.1.1, users can only set environment variables with "-shell_env" in
> ENTRYPOINT mode, and variables must be registered in
> yarn.nodemanager.env-whitelist.
> But in nonEntryPoint mode, we should allow users to set environment variables
> like "-e KEY=VAULE" in docker run command, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9058) YARN service fail to launch due to CSI changes
[
https://issues.apache.org/jira/browse/YARN-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699903#comment-16699903
]
Sunil Govindan commented on YARN-9058:
--
+1 for this change. Committing shortly
> YARN service fail to launch due to CSI changes
> --
>
> Key: YARN-9058
> URL: https://issues.apache.org/jira/browse/YARN-9058
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-9058.001.patch
>
>
> YARN service AM fails to launch with error message:
> {code}
> 2018-11-26 19:32:33,486 [main] INFO service.AbstractService - Service Client
> AM Service failed in state STARTED
> java.lang.ClassCastException:
> org.apache.hadoop.yarn.proto.ClientAMProtocol$ClientAMProtocolService$2
> cannot be cast to csi.com.google.protobuf.BlockingService
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.service.ClientAMService.serviceStart(ClientAMService.java:88)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.lambda$serviceStart$0(ServiceMaster.java:267)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.serviceStart(ServiceMaster.java:265)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.main(ServiceMaster.java:346)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8851) [Umbrella] A pluggable device plugin framework to ease vendor plugin development
[ https://issues.apache.org/jira/browse/YARN-8851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-8851: --- Summary: [Umbrella] A pluggable device plugin framework to ease vendor plugin development (was: [Umbrella] A new pluggable device plugin framework to ease vendor plugin development) > [Umbrella] A pluggable device plugin framework to ease vendor plugin > development > > > Key: YARN-8851 > URL: https://issues.apache.org/jira/browse/YARN-8851 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-8851-WIP2-trunk.001.patch, > YARN-8851-WIP3-trunk.001.patch, YARN-8851-WIP4-trunk.001.patch, > YARN-8851-WIP5-trunk.001.patch, YARN-8851-WIP6-trunk.001.patch, > YARN-8851-WIP7-trunk.001.patch, YARN-8851-WIP8-trunk.001.patch, > YARN-8851-WIP9-trunk.001.patch, YARN-8851-trunk.001.patch, > YARN-8851-trunk.002.patch, [YARN-8851] > YARN_New_Device_Plugin_Framework_Design_Proposal-3.pdf, [YARN-8851] > YARN_New_Device_Plugin_Framework_Design_Proposal-4.pdf, [YARN-8851] > YARN_New_Device_Plugin_Framework_Design_Proposal.pdf > > > At present, we support GPU/FPGA device in YARN through a native, coupling > way. But it's difficult for a vendor to implement such a device plugin > because the developer needs much knowledge of YARN internals. And this brings > burden to the community to maintain both YARN core and vendor-specific code. > Here we propose a new device plugin framework to ease vendor device plugin > development and provide a more flexible way to integrate with YARN NM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9060) [YARN-8851] Phase 1 - Support device isolation in native container-executor
Zhankun Tang created YARN-9060: -- Summary: [YARN-8851] Phase 1 - Support device isolation in native container-executor Key: YARN-9060 URL: https://issues.apache.org/jira/browse/YARN-9060 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhankun Tang Assignee: Zhankun Tang Due to the cgroups v1 implementation policy in linux kernel, we cannot update the value of the device cgroups controller unless we have the root permission ([here|https://github.com/torvalds/linux/blob/6f0d349d922ba44e4348a17a78ea51b7135965b1/security/device_cgroup.c#L604]). So we need to support this in container-executor for Java layer to invoke. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-9037: - Attachment: YARN-9037-002.patch > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-9037-002.patch, YARN-9037.001.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9058) YARN service fail to launch due to CSI changes
[
https://issues.apache.org/jira/browse/YARN-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-9058:
--
Attachment: YARN-9058.001.patch
> YARN service fail to launch due to CSI changes
> --
>
> Key: YARN-9058
> URL: https://issues.apache.org/jira/browse/YARN-9058
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
> Attachments: YARN-9058.001.patch
>
>
> YARN service AM fails to launch with error message:
> {code}
> 2018-11-26 19:32:33,486 [main] INFO service.AbstractService - Service Client
> AM Service failed in state STARTED
> java.lang.ClassCastException:
> org.apache.hadoop.yarn.proto.ClientAMProtocol$ClientAMProtocolService$2
> cannot be cast to csi.com.google.protobuf.BlockingService
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.service.ClientAMService.serviceStart(ClientAMService.java:88)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.lambda$serviceStart$0(ServiceMaster.java:267)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.serviceStart(ServiceMaster.java:265)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.main(ServiceMaster.java:346)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9059) Support RESTful API in NM for query FPGA allocation
Zhankun Tang created YARN-9059: -- Summary: Support RESTful API in NM for query FPGA allocation Key: YARN-9059 URL: https://issues.apache.org/jira/browse/YARN-9059 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhankun Tang Assignee: Zhankun Tang Support it for the user to be able to: curl :8042/ws/v1/node/resources/yarn.io%2Ffpga -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9058) YARN service fail to launch due to CSI changes
[
https://issues.apache.org/jira/browse/YARN-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699859#comment-16699859
]
Weiwei Yang commented on YARN-9058:
---
Hi [~eyang]
hadoop-yarn-csi module contains some shaded dependencies, adding this jar file
into hadoop classpath is programmatic. because hadoop has many places using
reflection to get class by name, this may cause to load wrong dependency. I
think we need to remove this jar file from hadoopclasspath. Will post a fix
today. Thanks
> YARN service fail to launch due to CSI changes
> --
>
> Key: YARN-9058
> URL: https://issues.apache.org/jira/browse/YARN-9058
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
>
> YARN service AM fails to launch with error message:
> {code}
> 2018-11-26 19:32:33,486 [main] INFO service.AbstractService - Service Client
> AM Service failed in state STARTED
> java.lang.ClassCastException:
> org.apache.hadoop.yarn.proto.ClientAMProtocol$ClientAMProtocolService$2
> cannot be cast to csi.com.google.protobuf.BlockingService
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.service.ClientAMService.serviceStart(ClientAMService.java:88)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.lambda$serviceStart$0(ServiceMaster.java:267)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.serviceStart(ServiceMaster.java:265)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.main(ServiceMaster.java:346)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9041) Optimize FSPreemptionThread#identifyContainersToPreempt method
[
https://issues.apache.org/jira/browse/YARN-9041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699843#comment-16699843
]
Steven Rand commented on YARN-9041:
---
Yes, the v2 patch resolves my concern -- thanks [~jiwq] for fixing that.
I'm curious, what's the motivation for reordering the conditions in the {{if}}
block?
> Optimize FSPreemptionThread#identifyContainersToPreempt method
> --
>
> Key: YARN-9041
> URL: https://issues.apache.org/jira/browse/YARN-9041
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler preemption
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Major
> Attachments: YARN-9041.001.patch, YARN-9041.002.patch
>
>
> In FSPreemptionThread#identifyContainersToPreempt method, I suggest if AM
> preemption, and locality relaxation is allowed, then the search space is
> expanded to all nodes changed to the remaining nodes. The remaining nodes are
> equal to all nodes minus the potential nodes.
> Judging condition changed to:
> # rr.getRelaxLocality()
> # !ResourceRequest.isAnyLocation(rr.getResourceName())
> # bestContainers != null
> # bestContainers.numAMContainers > 0
> If I understand the deviation, please criticize me. thx~
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9050) Usability improvements for scheduler activities
[
https://issues.apache.org/jira/browse/YARN-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-9050:
---
Component/s: capacityscheduler
> Usability improvements for scheduler activities
> ---
>
> Key: YARN-9050
> URL: https://issues.apache.org/jira/browse/YARN-9050
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: image-2018-11-23-16-46-38-138.png
>
>
> We have did some usability improvements for scheduler activities based on
> YARN3.1 in our cluster as follows:
> 1. Not available for multi-thread asynchronous scheduling. App and node
> activites maybe confused when multiple scheduling threads record activites of
> different allocation processes in the same variables like appsAllocation and
> recordingNodesAllocation in ActivitiesManager. I think these variables should
> be thread-local to make activities clear between multiple threads.
> 2. Incomplete activites for multi-node lookup machanism, since
> ActivitiesLogger will skip recording through {{if (node == null ||
> activitiesManager == null) return; }} when node is null which represents this
> allocation is for multi-nodes. We need support recording activities for
> multi-node lookup machanism.
> 3. Current app activites can not meet requirements of diagnostics, for
> example, we can know that node doesn't match request but hard to know why,
> especially when using placement constraints, it's difficult to make a
> detailed diagnosis manually. So I propose to improve the diagnoses of
> activites, add diagnosis for placement constraints check, update insufficient
> resource diagnosis with detailed info (like 'insufficient resource
> names:[memory-mb]') and so on.
> 4. Add more useful fields for app activities, in some scenarios we need to
> distinguish different requests but can't locate requests based on app
> activities info, there are some other fields can help to filter what we want
> such as allocation tags. We have added containerPriority, allocationRequestId
> and allocationTags fields in AppAllocation.
> 5. Filter app activities by key fields, sometimes the results of app
> activities is massive, it's hard to find what we want. We have support filter
> by allocation-tags to meet requirements from some apps, more over, we can
> take container-priority and allocation-request-id as candidates if necessary.
> 6. Aggragate app activities by diagnoses. For a single allocation process,
> activities still can be massive in a large cluster, we frequently want to
> know why request can't be allocated in cluster, it's hard to check every node
> manually in a large cluster, so that aggragation for app activities by
> diagnoses is neccessary to solve this trouble. We have added groupingType
> parameter for app-activities REST API for this, supports grouping by
> diagnositics and example like this:
> !image-2018-11-23-16-46-38-138.png!
> I think we can have a discuss about these points, useful improvements which
> can be accepted will be added into the patch. Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8488) YARN service/components/instances should have SUCCEEDED/FAILED states
[
https://issues.apache.org/jira/browse/YARN-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699784#comment-16699784
]
Hadoop QA commented on YARN-8488:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
23s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core:
The patch generated 0 new + 44 unchanged - 9 fixed = 44 total (was 53) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
25s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:080e9d0 |
| JIRA Issue | YARN-8488 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949551/YARN-8488-branch-3.1.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ec1e63c364a7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.1 / eb43772 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22716/testReport/ |
| Max. process+thread count | 760 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
|
|
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699779#comment-16699779 ] Hudson commented on YARN-8986: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15505/]) YARN-8986. Added port publish for Docker container running with bridge (eyang: rev 33e0df4b3558bea4a7977695d2c564af9a0c57e7) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRunCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerRunCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Fix For: 3.3.0 > > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch, > YARN-8986.009.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8986: Target Version/s: 3.2.1, 3.1.3 (was: 3.1.3) Fix Version/s: 3.3.0 > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Fix For: 3.3.0 > > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch, > YARN-8986.009.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8914) Add xtermjs to YARN UI2
[
https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699775#comment-16699775
]
Hadoop QA commented on YARN-8914:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 13m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
hadoop-client-modules/hadoop-client-minicluster . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 50s{color} | {color:orange} root: The patch generated 1 new + 2 unchanged -
0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
hadoop-client-modules/hadoop-client-minicluster . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}133m 29s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}274m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8914 |
| JIRA Patch URL |
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699773#comment-16699773 ] Eric Yang commented on YARN-8986: - [~Charo Zhang] I committed this to trunk. The patch doesn't apply to branch-3.2 and branch-3.1. Can you provide back port patches for the targeted branches? Thanks > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch, > YARN-8986.009.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9058) YARN service fail to launch due to CSI changes
Eric Yang created YARN-9058:
---
Summary: YARN service fail to launch due to CSI changes
Key: YARN-9058
URL: https://issues.apache.org/jira/browse/YARN-9058
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Eric Yang
YARN service AM fails to launch with error message:
{code}
2018-11-26 19:32:33,486 [main] INFO service.AbstractService - Service Client
AM Service failed in state STARTED
java.lang.ClassCastException:
org.apache.hadoop.yarn.proto.ClientAMProtocol$ClientAMProtocolService$2 cannot
be cast to csi.com.google.protobuf.BlockingService
at
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
at
org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
at
org.apache.hadoop.yarn.service.ClientAMService.serviceStart(ClientAMService.java:88)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.service.ServiceMaster.lambda$serviceStart$0(ServiceMaster.java:267)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.hadoop.yarn.service.ServiceMaster.serviceStart(ServiceMaster.java:265)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.yarn.service.ServiceMaster.main(ServiceMaster.java:346)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9058) YARN service fail to launch due to CSI changes
[
https://issues.apache.org/jira/browse/YARN-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reassigned YARN-9058:
---
Assignee: Weiwei Yang
> YARN service fail to launch due to CSI changes
> --
>
> Key: YARN-9058
> URL: https://issues.apache.org/jira/browse/YARN-9058
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Blocker
>
> YARN service AM fails to launch with error message:
> {code}
> 2018-11-26 19:32:33,486 [main] INFO service.AbstractService - Service Client
> AM Service failed in state STARTED
> java.lang.ClassCastException:
> org.apache.hadoop.yarn.proto.ClientAMProtocol$ClientAMProtocolService$2
> cannot be cast to csi.com.google.protobuf.BlockingService
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.service.ClientAMService.serviceStart(ClientAMService.java:88)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.lambda$serviceStart$0(ServiceMaster.java:267)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.serviceStart(ServiceMaster.java:265)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.main(ServiceMaster.java:346)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9055) Capacity Scheduler: allow larger queue level maximum-allocation-mb to override the cluster configuration
[
https://issues.apache.org/jira/browse/YARN-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699757#comment-16699757
]
Hadoop QA commented on YARN-9055:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 271 unchanged - 0 fixed = 273 total (was 271)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9055 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949554/YARN-9055.1.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d9d16791c294 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0d84061 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22715/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699740#comment-16699740 ] Chandni Singh commented on YARN-8298: - Thanks [~eyang] > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8298-branch-3.1.001.patch, YARN-8298.001.patch, > YARN-8298.002.patch, YARN-8298.003.patch, YARN-8298.004.patch, > YARN-8298.005.patch, YARN-8298.006.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade the service in one shot: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9003) Support multi-homed network for docker container
[ https://issues.apache.org/jira/browse/YARN-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699738#comment-16699738 ] Eric Yang commented on YARN-9003: - The failed unit test is not related to patch 002. > Support multi-homed network for docker container > > > Key: YARN-9003 > URL: https://issues.apache.org/jira/browse/YARN-9003 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Labels: docker > Attachments: YARN-9003.001.patch, YARN-9003.002.patch > > > Docker network can be defined as configuration properties - docker.network to > setup docker container to connect to a specific network in YARN service. > Docker can run multi-homed network by specifying --net=bridge > --net=private-net. This is useful to expose small number of front end > container and ports, while the rest of the infrastructure runs in private > network. This task is to add support for specifying multiple docker networks > to YARN service and docker support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9058) YARN service fail to launch due to CSI changes
[
https://issues.apache.org/jira/browse/YARN-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699727#comment-16699727
]
Eric Yang commented on YARN-9058:
-
[~cheersyang] [~sunilg] This bug is blocking me from making progress with trunk
code. Can you help?
> YARN service fail to launch due to CSI changes
> --
>
> Key: YARN-9058
> URL: https://issues.apache.org/jira/browse/YARN-9058
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Priority: Blocker
>
> YARN service AM fails to launch with error message:
> {code}
> 2018-11-26 19:32:33,486 [main] INFO service.AbstractService - Service Client
> AM Service failed in state STARTED
> java.lang.ClassCastException:
> org.apache.hadoop.yarn.proto.ClientAMProtocol$ClientAMProtocolService$2
> cannot be cast to csi.com.google.protobuf.BlockingService
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.service.ClientAMService.serviceStart(ClientAMService.java:88)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.lambda$serviceStart$0(ServiceMaster.java:267)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.serviceStart(ServiceMaster.java:265)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.service.ServiceMaster.main(ServiceMaster.java:346)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9057) CSI jar file should not bundle third party dependencies
Eric Yang created YARN-9057:
---
Summary: CSI jar file should not bundle third party dependencies
Key: YARN-9057
URL: https://issues.apache.org/jira/browse/YARN-9057
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Eric Yang
hadoop-yarn-csi-3.3.0-SNAPSHOT.jar bundles all third party classes like a
shaded jar instead of CSI only classes. This is generating error messages for
YARN cli:
{code}
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-csi-3.3.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9056) Yarn Service Upgrade: Instance state changes from UPGRADING to READY without performing a readiness check
[ https://issues.apache.org/jira/browse/YARN-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-9056: Affects Version/s: 3.1.2 3.2.0 > Yarn Service Upgrade: Instance state changes from UPGRADING to READY without > performing a readiness check > - > > Key: YARN-9056 > URL: https://issues.apache.org/jira/browse/YARN-9056 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Critical > > Currently, when an instance is upgraded, the state of the instance changes to > UPGRADING. Once the NM informs AM that upgrade is finished, the state of the > instance changes to STABLE. > The instance state should be changes to STABLE only once readiness check > succeeds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8867) Retrieve the status of resource localization
[
https://issues.apache.org/jira/browse/YARN-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699694#comment-16699694
]
Eric Yang commented on YARN-8867:
-
[~csingh] I get this error message during javadoc compile:
{code}
[ERROR]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalizationStatus.java:34:
error: unexpected end tag:
[ERROR] *
[ERROR]^
{code}
It looks like conflict between p tag, and javadock link tag.
> Retrieve the status of resource localization
>
>
> Key: YARN-8867
> URL: https://issues.apache.org/jira/browse/YARN-8867
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Attachments: YARN-8867.001.patch, YARN-8867.002.patch,
> YARN-8867.003.patch, YARN-8867.004.patch, YARN-8867.005.patch,
> YARN-8867.wip.patch
>
>
> Refer YARN-3854.
> Currently NM does not have an API to retrieve the status of localization.
> Unless the client can know when the localization of a resource is complete
> irrespective of the type of the resource, it cannot take any appropriate
> action.
> We need an API in {{ContainerManagementProtocol}} to retrieve the status on
> the localization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[
https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699684#comment-16699684
]
Hadoop QA commented on YARN-8298:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
9s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
46s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 19s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 8 new + 387 unchanged - 2 fixed = 395 total (was 389) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
34s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 25m
29s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
58s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
56s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:080e9d0 |
| JIRA Issue | YARN-8298 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949547/YARN-8298-branch-3.1.001.patch
|
| Optional Tests | dupname asflicense compile javac
[jira] [Commented] (YARN-9003) Support multi-homed network for docker container
[
https://issues.apache.org/jira/browse/YARN-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699661#comment-16699661
]
Hadoop QA commented on YARN-9003:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 23s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9003 |
| JIRA Patch URL |
[jira] [Created] (YARN-9056) Yarn Service Upgrade: Instance state changes from UPGRADING to READY without performing a readiness check
Chandni Singh created YARN-9056: --- Summary: Yarn Service Upgrade: Instance state changes from UPGRADING to READY without performing a readiness check Key: YARN-9056 URL: https://issues.apache.org/jira/browse/YARN-9056 Project: Hadoop YARN Issue Type: Bug Reporter: Chandni Singh Assignee: Chandni Singh Currently, when an instance is upgraded, the state of the instance changes to UPGRADING. Once the NM informs AM that upgrade is finished, the state of the instance changes to STABLE. The instance state should be changes to STABLE only once readiness check succeeds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9023) Pluggable YARN Auxiliary Services
[ https://issues.apache.org/jira/browse/YARN-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699649#comment-16699649 ] Eric Yang commented on YARN-9023: - [~oshevchenko] Thank you for the proposal. The proposal is an extension to make auxiliary service more generic for YARN daemons. However, the original design of YARN auxiliary service suffered from a couple short comings: # The injected service can not share the same jetty server. If the aux service like to add more REST API endpoint, this needs to route to a separate port. The end result is to add duplicated efforts to secure newly opened port. # Auxiliary service must upgrade at the same time when YARN framework upgrades. This creates risk for upgrading YARN framework that might break auxiliary service. # New functionality can only be added with restart of YARN framework. # Resource used by auxiliary service is not limited by resource management due to parasitic nature of the design. This can leads to resource leaks by auxiliary service and worst yet to bring down YARN framework. # YARN user credential is too powerful, mapreduce shuffle service should not run with YARN user credential to prevent third party to trigger code injection exploits via mapreduce. Those lessons learned from node manager auxiliary service, I think it would be better to redesign the auxiliary service from scratch. It would be better to launch the auxiliary service as yarn containers. This will ensure there is a way to add on new services without interruption to YARN framework, and reduce security holes that can be exposed to YARN daemons. ATS v2 deployed HBase auxiliary service is an example of the direction that has been taken to solve auxiliary service problem. This path allows the community to harden the core technology and move bell and whistle features into containers. Thoughts? > Pluggable YARN Auxiliary Services > - > > Key: YARN-9023 > URL: https://issues.apache.org/jira/browse/YARN-9023 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Oleksandr Shevchenko >Assignee: Oleksandr Shevchenko >Priority: Major > Attachments: Pluggable YARN Auxiliary Services Doc v1.pdf > > > The propose is to create customizable auxiliary services infrastructure for > all YARN services to provide abilities to extends or change functionality for > all YARN demons. > Currently, YARN has the ability to add auxiliary services only for > NodeManager. > There are some cases when we need to add auxiliary services for some other > YARN demons. For example, in YARN-2267 proposed to add Auxiliary Service > support in RM for implementation monitor plugins or some alarm services. We > faced with one more case, we need to create/setup some storage for YARN demon > data (for example for RM state store or AM intermediate data) before demon > started. Also, will be good if we can use some auxiliary service in different > YARN demons. > The end users can use YARN Auxiliary Services (YAS) to customize or extend > the functionality of all YARN demons. We can provide an ability to create aux > service both for one and for all demons. This will make easy creating of aux > service for shuffle or just to up some service within another YARN daemon. We > will get a unified and more extensible architecture for all aux services > which may need to create in the future. > Please look at the attached design doc for the details. [^Pluggable YARN > Auxiliary Services Doc v1.pdf] > Also, you can add your comments to this doc: > https://docs.google.com/document/d/1Pxs8eeWZwnlwryPj7gyN8Ej1O-FRBSX_C0ispV9ORT8/edit?usp=sharing > Thanks a lot for any comments and attention to this feature! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8962) Add ability to use interactive shell with normal yarn container
[
https://issues.apache.org/jira/browse/YARN-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699647#comment-16699647
]
Hadoop QA commented on YARN-8962:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
0s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8962 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949550/YARN-8962.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux dadc91f7332b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0d84061 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22714/testReport/ |
| Max. process+thread count | 435 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22714/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Updated] (YARN-8665) Yarn Service Upgrade: Support cancelling upgrade
[ https://issues.apache.org/jira/browse/YARN-8665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8665: Attachment: YARN-8665-branch-3.1.001.patch > Yarn Service Upgrade: Support cancelling upgrade > - > > Key: YARN-8665 > URL: https://issues.apache.org/jira/browse/YARN-8665 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8665-branch-3.1.001.patch, YARN-8665.001.patch, > YARN-8665.002.patch, YARN-8665.003.patch, YARN-8665.004.patch, > YARN-8665.005.patch > > > When a service is upgraded without auto-finalization or express upgrade, then > the upgrade can be cancelled. This provides the user ability to test upgrade > of a single instance and if that doesn't go well, they get a chance to cancel > it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9055) Capacity Scheduler: allow larger queue level maximum-allocation-mb to override the cluster configuration
[
https://issues.apache.org/jira/browse/YARN-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated YARN-9055:
---
Attachment: YARN-9055.1.patch
> Capacity Scheduler: allow larger queue level maximum-allocation-mb to
> override the cluster configuration
>
>
> Key: YARN-9055
> URL: https://issues.apache.org/jira/browse/YARN-9055
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.7.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: YARN-9055.1.patch
>
>
> YARN-1582 adds the support of maximum-allocation-mb configuration per queue.
> That feature gives the flexibility to give different memory requirements for
> different queues. Such patch adds the limitation that the queue level
> configuration can't exceed the cluster level default configuration, but I
> feel it may make more sense to remove such limitation to allow any overrides
> since
> # Such configuration is controlled by the admin so it shouldn't get abused;
> # It's common that typical queues require standard size containers while some
> job (queues) have requirements for larger containers. With current
> limitation, we have to set larger configuration on the cluster setting which
> will cause resource abuse unless we override them on all the queues.
> We can remove such limitation in CapacitySchedulerConfiguration.java so the
> cluster setting provides the default value and queue setting can override it.
> {noformat}
>if (maxAllocationMbPerQueue > clusterMax.getMemorySize()
> || maxAllocationVcoresPerQueue > clusterMax.getVirtualCores()) {
> throw new IllegalArgumentException(
> "Queue maximum allocation cannot be larger than the cluster setting"
> + " for queue " + queue
> + " max allocation per queue: " + result
> + " cluster setting: " + clusterMax);
> }
> {noformat}
> Let me know if it makes sense.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9032) Support sh shell for interactive shell
[
https://issues.apache.org/jira/browse/YARN-9032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699595#comment-16699595
]
Hadoop QA commented on YARN-9032:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-9032 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9032 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12948575/YARN-9032.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22712/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support sh shell for interactive shell
> --
>
> Key: YARN-9032
> URL: https://issues.apache.org/jira/browse/YARN-9032
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-9032.001.patch
>
>
> CLI can supply second parameter such as:
> yarn container -shell [container-id] sh
> This is supposed to launch sh instead of bash. This is not fully working yet
> in YARN-8778 patch and some organization of code clean up to support
> non-docker container in YARN-8962 making this change in YARN-8778 difficult.
> Hence, I file this as a separate sub-task to close the gap.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8488) YARN service/components/instances should have SUCCEEDED/FAILED states
[
https://issues.apache.org/jira/browse/YARN-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-8488:
Attachment: YARN-8488-branch-3.1.001.patch
> YARN service/components/instances should have SUCCEEDED/FAILED states
> -
>
> Key: YARN-8488
> URL: https://issues.apache.org/jira/browse/YARN-8488
> Project: Hadoop YARN
> Issue Type: Task
> Components: yarn-native-services
>Reporter: Wangda Tan
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8488-branch-3.1.001.patch, YARN-8488.1.patch,
> YARN-8488.2.patch, YARN-8488.3.patch, YARN-8488.4.patch, YARN-8488.5.patch,
> YARN-8488.6.patch, YARN-8488.7.patch, YARN-8488.8.patch
>
>
> Existing YARN service has following states:
> {code}
> public enum ServiceState {
> ACCEPTED, STARTED, STABLE, STOPPED, FAILED, FLEX, UPGRADING,
> UPGRADING_AUTO_FINALIZE;
> }
> {code}
> Ideally we should add "SUCCEEDED" state in order to support long running
> applications like Tensorflow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8963) Add flag to disable interactive shell
[
https://issues.apache.org/jira/browse/YARN-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699596#comment-16699596
]
Hadoop QA commented on YARN-8963:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-8963 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8963 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946757/YARN-8963.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22713/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add flag to disable interactive shell
> -
>
> Key: YARN-8963
> URL: https://issues.apache.org/jira/browse/YARN-8963
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-8963.001.patch
>
>
> For some production job, application admin might choose to disable debugging
> to production jobs to prevent developer or system admin from accessing the
> containers. It would be nice to add an environment variable flag to disable
> interactive shell during application submission.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8962) Add ability to use interactive shell with normal yarn container
[ https://issues.apache.org/jira/browse/YARN-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8962: Attachment: YARN-8962.003.patch > Add ability to use interactive shell with normal yarn container > --- > > Key: YARN-8962 > URL: https://issues.apache.org/jira/browse/YARN-8962 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8962.001.patch, YARN-8962.002.patch, > YARN-8962.003.patch > > > This task is focusing on extending interactive shell capability to yarn > container without docker. This will improve some aspect of debugging > mapreduce or spark applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8962) Add ability to use interactive shell with normal yarn container
[ https://issues.apache.org/jira/browse/YARN-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699587#comment-16699587 ] Eric Yang commented on YARN-8962: - Patch 003 fixed chdir return code check, checkstyle and white space errors. > Add ability to use interactive shell with normal yarn container > --- > > Key: YARN-8962 > URL: https://issues.apache.org/jira/browse/YARN-8962 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8962.001.patch, YARN-8962.002.patch, > YARN-8962.003.patch > > > This task is focusing on extending interactive shell capability to yarn > container without docker. This will improve some aspect of debugging > mapreduce or spark applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8488) YARN service/components/instances should have SUCCEEDED/FAILED states
[
https://issues.apache.org/jira/browse/YARN-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699585#comment-16699585
]
Chandni Singh commented on YARN-8488:
-
Re-opened to backport the changes to branch-3.1.
In order to backport https://issues.apache.org/jira/browse/YARN-8665 cleanly, I
will have to backport this change as well.
> YARN service/components/instances should have SUCCEEDED/FAILED states
> -
>
> Key: YARN-8488
> URL: https://issues.apache.org/jira/browse/YARN-8488
> Project: Hadoop YARN
> Issue Type: Task
> Components: yarn-native-services
>Reporter: Wangda Tan
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8488.1.patch, YARN-8488.2.patch, YARN-8488.3.patch,
> YARN-8488.4.patch, YARN-8488.5.patch, YARN-8488.6.patch, YARN-8488.7.patch,
> YARN-8488.8.patch
>
>
> Existing YARN service has following states:
> {code}
> public enum ServiceState {
> ACCEPTED, STARTED, STABLE, STOPPED, FAILED, FLEX, UPGRADING,
> UPGRADING_AUTO_FINALIZE;
> }
> {code}
> Ideally we should add "SUCCEEDED" state in order to support long running
> applications like Tensorflow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9003) Support multi-homed network for docker container
[ https://issues.apache.org/jira/browse/YARN-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699578#comment-16699578 ] Eric Yang commented on YARN-9003: - Patch 002 limit the number of networks that can be specified to 2. > Support multi-homed network for docker container > > > Key: YARN-9003 > URL: https://issues.apache.org/jira/browse/YARN-9003 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Labels: docker > Attachments: YARN-9003.001.patch, YARN-9003.002.patch > > > Docker network can be defined as configuration properties - docker.network to > setup docker container to connect to a specific network in YARN service. > Docker can run multi-homed network by specifying --net=bridge > --net=private-net. This is useful to expose small number of front end > container and ports, while the rest of the infrastructure runs in private > network. This task is to add support for specifying multiple docker networks > to YARN service and docker support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9003) Support multi-homed network for docker container
[ https://issues.apache.org/jira/browse/YARN-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-9003: Attachment: YARN-9003.002.patch > Support multi-homed network for docker container > > > Key: YARN-9003 > URL: https://issues.apache.org/jira/browse/YARN-9003 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Labels: docker > Attachments: YARN-9003.001.patch, YARN-9003.002.patch > > > Docker network can be defined as configuration properties - docker.network to > setup docker container to connect to a specific network in YARN service. > Docker can run multi-homed network by specifying --net=bridge > --net=private-net. This is useful to expose small number of front end > container and ports, while the rest of the infrastructure runs in private > network. This task is to add support for specifying multiple docker networks > to YARN service and docker support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-8488) YARN service/components/instances should have SUCCEEDED/FAILED states
[
https://issues.apache.org/jira/browse/YARN-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh reopened YARN-8488:
-
> YARN service/components/instances should have SUCCEEDED/FAILED states
> -
>
> Key: YARN-8488
> URL: https://issues.apache.org/jira/browse/YARN-8488
> Project: Hadoop YARN
> Issue Type: Task
> Components: yarn-native-services
>Reporter: Wangda Tan
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: YARN-8488.1.patch, YARN-8488.2.patch, YARN-8488.3.patch,
> YARN-8488.4.patch, YARN-8488.5.patch, YARN-8488.6.patch, YARN-8488.7.patch,
> YARN-8488.8.patch
>
>
> Existing YARN service has following states:
> {code}
> public enum ServiceState {
> ACCEPTED, STARTED, STABLE, STOPPED, FAILED, FLEX, UPGRADING,
> UPGRADING_AUTO_FINALIZE;
> }
> {code}
> Ideally we should add "SUCCEEDED" state in order to support long running
> applications like Tensorflow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-8665) Yarn Service Upgrade: Support cancelling upgrade
[ https://issues.apache.org/jira/browse/YARN-8665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh reopened YARN-8665: - Re-opening to backport the changes 3.1 > Yarn Service Upgrade: Support cancelling upgrade > - > > Key: YARN-8665 > URL: https://issues.apache.org/jira/browse/YARN-8665 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8665.001.patch, YARN-8665.002.patch, > YARN-8665.003.patch, YARN-8665.004.patch, YARN-8665.005.patch > > > When a service is upgraded without auto-finalization or express upgrade, then > the upgrade can be cancelled. This provides the user ability to test upgrade > of a single instance and if that doesn't go well, they get a chance to cancel > it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8298: Attachment: YARN-8298-branch-3.1.001.patch > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8298-branch-3.1.001.patch, YARN-8298.001.patch, > YARN-8298.002.patch, YARN-8298.003.patch, YARN-8298.004.patch, > YARN-8298.005.patch, YARN-8298.006.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade the service in one shot: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8914) Add xtermjs to YARN UI2
[ https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8914: Attachment: YARN-8914.007.patch > Add xtermjs to YARN UI2 > --- > > Key: YARN-8914 > URL: https://issues.apache.org/jira/browse/YARN-8914 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8914.001.patch, YARN-8914.002.patch, > YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch, > YARN-8914.006.patch, YARN-8914.007.patch > > > In the container listing from UI2, we can add a link to connect to docker > container using xtermjs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8914) Add xtermjs to YARN UI2
[ https://issues.apache.org/jira/browse/YARN-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699514#comment-16699514 ] Eric Yang commented on YARN-8914: - [~billie.rinaldi] Patch 007 resolves LICENSE.txt and NOTICE.txt conflicts. > Add xtermjs to YARN UI2 > --- > > Key: YARN-8914 > URL: https://issues.apache.org/jira/browse/YARN-8914 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8914.001.patch, YARN-8914.002.patch, > YARN-8914.003.patch, YARN-8914.004.patch, YARN-8914.005.patch, > YARN-8914.006.patch, YARN-8914.007.patch > > > In the container listing from UI2, we can add a link to connect to docker > container using xtermjs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9041) Optimize FSPreemptionThread#identifyContainersToPreempt method
[ https://issues.apache.org/jira/browse/YARN-9041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699511#comment-16699511 ] Yufei Gu commented on YARN-9041: Hi [~jiwq], thanks for the patch. I like the idea to shrink the search space, and your patch v2 seems to solve the concern raised by [~Steven Rand]. However, it is necessary to provide a unit test case for the change. > Optimize FSPreemptionThread#identifyContainersToPreempt method > -- > > Key: YARN-9041 > URL: https://issues.apache.org/jira/browse/YARN-9041 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler preemption >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Attachments: YARN-9041.001.patch, YARN-9041.002.patch > > > In FSPreemptionThread#identifyContainersToPreempt method, I suggest if AM > preemption, and locality relaxation is allowed, then the search space is > expanded to all nodes changed to the remaining nodes. The remaining nodes are > equal to all nodes minus the potential nodes. > Judging condition changed to: > # rr.getRelaxLocality() > # !ResourceRequest.isAnyLocation(rr.getResourceName()) > # bestContainers != null > # bestContainers.numAMContainers > 0 > If I understand the deviation, please criticize me. thx~ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-8298) Yarn Service Upgrade: Support express upgrade of a service
[ https://issues.apache.org/jira/browse/YARN-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh reopened YARN-8298: - Reopening to backport to branch-3.1 > Yarn Service Upgrade: Support express upgrade of a service > -- > > Key: YARN-8298 > URL: https://issues.apache.org/jira/browse/YARN-8298 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8298.001.patch, YARN-8298.002.patch, > YARN-8298.003.patch, YARN-8298.004.patch, YARN-8298.005.patch, > YARN-8298.006.patch > > > Currently service upgrade involves 2 steps > * initiate upgrade by providing new spec > * trigger upgrade of each instance/component > > We need to add the ability to upgrade the service in one shot: > # Aborting the upgrade will not be supported > # Upgrade finalization will be done automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
[ https://issues.apache.org/jira/browse/YARN-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699466#comment-16699466 ] Hudson commented on YARN-9054: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15504 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15504/]) YARN-9054. Fix (gifuma: rev 0d8406135f8b8ac427fd7f49f5faf20064ace121) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/utils/FederationStateStoreFacade.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/FederationStateStoreTestUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java > Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest > -- > > Key: YARN-9054 > URL: https://issues.apache.org/jira/browse/YARN-9054 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9054.001.patch, YARN-9054.002.patch > > > Currently cache key is create with null argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9055) Capacity Scheduler: allow larger queue level maximum-allocation-mb to override the cluster configuration
[
https://issues.apache.org/jira/browse/YARN-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699460#comment-16699460
]
Aihua Xu commented on YARN-9055:
[~jlowe], [~leftnoteasy] and [~tgraves] Can you guys have any opinions on this?
> Capacity Scheduler: allow larger queue level maximum-allocation-mb to
> override the cluster configuration
>
>
> Key: YARN-9055
> URL: https://issues.apache.org/jira/browse/YARN-9055
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.7.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>
> YARN-1582 adds the support of maximum-allocation-mb configuration per queue.
> That feature gives the flexibility to give different memory requirements for
> different queues. Such patch adds the limitation that the queue level
> configuration can't exceed the cluster level default configuration, but I
> feel it may make more sense to remove such limitation to allow any overrides
> since
> # Such configuration is controlled by the admin so it shouldn't get abused;
> # It's common that typical queues require standard size containers while some
> job (queues) have requirements for larger containers. With current
> limitation, we have to set larger configuration on the cluster setting which
> will cause resource abuse unless we override them on all the queues.
> We can remove such limitation in CapacitySchedulerConfiguration.java so the
> cluster setting provides the default value and queue setting can override it.
> {noformat}
>if (maxAllocationMbPerQueue > clusterMax.getMemorySize()
> || maxAllocationVcoresPerQueue > clusterMax.getVirtualCores()) {
> throw new IllegalArgumentException(
> "Queue maximum allocation cannot be larger than the cluster setting"
> + " for queue " + queue
> + " max allocation per queue: " + result
> + " cluster setting: " + clusterMax);
> }
> {noformat}
> Let me know if it makes sense.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
[ https://issues.apache.org/jira/browse/YARN-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699437#comment-16699437 ] Giovanni Matteo Fumarola commented on YARN-9054: Thanks [~bibinchundatt] for fixing it. +1. > Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest > -- > > Key: YARN-9054 > URL: https://issues.apache.org/jira/browse/YARN-9054 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-9054.001.patch, YARN-9054.002.patch > > > Currently cache key is create with null argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
[ https://issues.apache.org/jira/browse/YARN-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699437#comment-16699437 ] Giovanni Matteo Fumarola edited comment on YARN-9054 at 11/26/18 6:59 PM: -- Thanks [~bibinchundatt] for fixing it. +1 Committed to trunk. Let me know if we should push it in Branch-2 as well. was (Author: giovanni.fumarola): Thanks [~bibinchundatt] for fixing it. +1. > Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest > -- > > Key: YARN-9054 > URL: https://issues.apache.org/jira/browse/YARN-9054 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9054.001.patch, YARN-9054.002.patch > > > Currently cache key is create with null argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
[ https://issues.apache.org/jira/browse/YARN-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-9054: --- Fix Version/s: 3.3.0 > Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest > -- > > Key: YARN-9054 > URL: https://issues.apache.org/jira/browse/YARN-9054 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9054.001.patch, YARN-9054.002.patch > > > Currently cache key is create with null argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9055) Capacity Scheduler: allow larger queue level maximum-allocation-mb to override the cluster configuration
Aihua Xu created YARN-9055:
--
Summary: Capacity Scheduler: allow larger queue level
maximum-allocation-mb to override the cluster configuration
Key: YARN-9055
URL: https://issues.apache.org/jira/browse/YARN-9055
Project: Hadoop YARN
Issue Type: Improvement
Components: capacityscheduler
Affects Versions: 2.7.0
Reporter: Aihua Xu
Assignee: Aihua Xu
YARN-1582 adds the support of maximum-allocation-mb configuration per queue.
That feature gives the flexibility to give different memory requirements for
different queues. Such patch adds the limitation that the queue level
configuration can't exceed the cluster level default configuration, but I feel
it may make more sense to remove such limitation to allow any overrides since
# Such configuration is controlled by the admin so it shouldn't get abused;
# It's common that typical queues require standard size containers while some
job (queues) have requirements for larger containers. With current limitation,
we have to set larger configuration on the cluster setting which will cause
resource abuse unless we override them on all the queues.
We can remove such limitation in CapacitySchedulerConfiguration.java so the
cluster setting provides the default value and queue setting can override it.
{noformat}
if (maxAllocationMbPerQueue > clusterMax.getMemorySize()
|| maxAllocationVcoresPerQueue > clusterMax.getVirtualCores()) {
throw new IllegalArgumentException(
"Queue maximum allocation cannot be larger than the cluster setting"
+ " for queue " + queue
+ " max allocation per queue: " + result
+ " cluster setting: " + clusterMax);
}
{noformat}
Let me know if it makes sense.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7746) Fix PlacementProcessor to support app priority
[ https://issues.apache.org/jira/browse/YARN-7746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-7746: --- Attachment: YARN-7746.002.patch > Fix PlacementProcessor to support app priority > -- > > Key: YARN-7746 > URL: https://issues.apache.org/jira/browse/YARN-7746 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Manikandan R >Priority: Major > Attachments: YARN-7746.001.patch, YARN-7746.002.patch > > > The Threadpools used in the Processor should be modified to take a priority > blocking queue that respects application priority. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7746) Fix PlacementProcessor to support app priority
[ https://issues.apache.org/jira/browse/YARN-7746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699382#comment-16699382 ] Manikandan R commented on YARN-7746: Sorry for the delay. Attached .002 patch. Simplified based on earlier suggestion and added unit tests for the same. Please review. > Fix PlacementProcessor to support app priority > -- > > Key: YARN-7746 > URL: https://issues.apache.org/jira/browse/YARN-7746 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Manikandan R >Priority: Major > Attachments: YARN-7746.001.patch > > > The Threadpools used in the Processor should be modified to take a priority > blocking queue that respects application priority. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
[
https://issues.apache.org/jira/browse/YARN-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699364#comment-16699364
]
Hadoop QA commented on YARN-9054:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
4s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9054 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949515/YARN-9054.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e531524e46a4 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b098281 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22708/testReport/ |
| Max. process+thread count | 328 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22708/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
>
[jira] [Updated] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
[ https://issues.apache.org/jira/browse/YARN-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-9054: --- Attachment: YARN-9054.002.patch > Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest > -- > > Key: YARN-9054 > URL: https://issues.apache.org/jira/browse/YARN-9054 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-9054.001.patch, YARN-9054.002.patch > > > Currently cache key is create with null argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6523) Newly retrieved security Tokens are sent as part of each heartbeat to each node from RM which is not desirable in large cluster
[
https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699188#comment-16699188
]
Jason Lowe commented on YARN-6523:
--
Thanks for updating the patch! The whitespace and ASF warnings are related to
the patch. There are a lot of checkstyle warnings that should also be
addressed like unused imports, lacking braces on {{if}} statements, etc. The
unit test failure appears to be unrelated and tracked by YARN-8937.
I'm not sure all this PBImpl stuff is worth the boilerplate, especially in this
instance, and especially because it is not exposed to user code -- this is an
internal API between the RM and the NM. All we need to do here is cache the
list of SystemCredentialsForAppsProto values and have NodeHearbeatResponse take
that list of protos rather than a Map for the system
credentials. NodeHeartbeatResponsePBImpl can then call
addAllSystemCredentialsForApps on the builder when it builds its protocol
buffer. IMHO adding more abstract classes and PBImpls here just adds
boilerplate for no real benefit and just adds room for errors. It's not like
we support anything other than protocol buffers -- there's blind downcasting to
PBImpl all over the place, including in this patch. I say ditch all that and
just cache the list of SystemCredentialsForAppsProto directly. No new
interfaces or classes needed, especially because we're not sending anything we
weren't already sending before.
TestYarnServerApiClasses has a commented import.
TestYarnServerApiClasses#testNodeHeartbeatResponsePBImpl has two "// create
token2" comments, and I'm assuming only one of them is accurate.
ResourceTrackerService#populateTokenSequenceNo has a few debug statement that
build log strings before calling the logger. These either need to be protected
by log debug enabled checks or use the SLF4J positional argument brace syntax
to avoid constructing the log string for no benefit when debug logging is not
enabled (i.e.: the common case).
The very long unit test was removed but equivalent tests were not added. The
only tests that now exist are simple ones that verify
NodeHeartbeatReponsePBImpl properly preserves the system credentials for apps
payload. There should be two additional tests as I described earlier:
bq. One approach is to split the testing into two, separate unit tests for the
two prime units requiring testing as part of this feature. One test can verify
the DelegationTokenRenewer updates the sequence number when new delegation
tokens are requested, and another test can verify that ResourceTrackerService
sends system credentials in the heartbeat response only when the sequence
numbers mismatch.
> Newly retrieved security Tokens are sent as part of each heartbeat to each
> node from RM which is not desirable in large cluster
> ---
>
> Key: YARN-6523
> URL: https://issues.apache.org/jira/browse/YARN-6523
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: RM
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Naganarasimha G R
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-6523.001.patch, YARN-6523.002.patch,
> YARN-6523.003.patch, YARN-6523.004.patch, YARN-6523.005.patch,
> YARN-6523.006.patch, YARN-6523.007.patch
>
>
> Currently as part of heartbeat response RM sets all application's tokens
> though all applications might not be active on the node. On top of it
> NodeHeartbeatResponsePBImpl converts tokens for each app into
> SystemCredentialsForAppsProto. Hence for each node and each heartbeat too
> many SystemCredentialsForAppsProto objects were getting created.
> We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with
> 8GB RAM configured for RM
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8822) Nvidia-docker v2 support
[
https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699178#comment-16699178
]
Hadoop QA commented on YARN-8822:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 9m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 47s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 6 new + 217 unchanged - 0 fixed = 223 total (was 217) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
57s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
23s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8822 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949495/YARN-8822.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux 87be90ac95c2 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b098281 |
| maven | version:
[jira] [Commented] (YARN-3890) FairScheduler should show the scheduler health metrics similar to ones added in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699170#comment-16699170
]
Hadoop QA commented on YARN-3890:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 89 unchanged - 2 fixed = 91 total (was 91) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 54s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-3890 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949490/YARN-3890.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b6d510153fea 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b098281 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22704/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22704/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-9023) Pluggable YARN Auxiliary Services
[ https://issues.apache.org/jira/browse/YARN-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699158#comment-16699158 ] Oleksandr Shevchenko commented on YARN-9023: Could someone kindly review the proposal? Thanks! > Pluggable YARN Auxiliary Services > - > > Key: YARN-9023 > URL: https://issues.apache.org/jira/browse/YARN-9023 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Oleksandr Shevchenko >Assignee: Oleksandr Shevchenko >Priority: Major > Attachments: Pluggable YARN Auxiliary Services Doc v1.pdf > > > The propose is to create customizable auxiliary services infrastructure for > all YARN services to provide abilities to extends or change functionality for > all YARN demons. > Currently, YARN has the ability to add auxiliary services only for > NodeManager. > There are some cases when we need to add auxiliary services for some other > YARN demons. For example, in YARN-2267 proposed to add Auxiliary Service > support in RM for implementation monitor plugins or some alarm services. We > faced with one more case, we need to create/setup some storage for YARN demon > data (for example for RM state store or AM intermediate data) before demon > started. Also, will be good if we can use some auxiliary service in different > YARN demons. > The end users can use YARN Auxiliary Services (YAS) to customize or extend > the functionality of all YARN demons. We can provide an ability to create aux > service both for one and for all demons. This will make easy creating of aux > service for shuffle or just to up some service within another YARN daemon. We > will get a unified and more extensible architecture for all aux services > which may need to create in the future. > Please look at the attached design doc for the details. [^Pluggable YARN > Auxiliary Services Doc v1.pdf] > Also, you can add your comments to this doc: > https://docs.google.com/document/d/1Pxs8eeWZwnlwryPj7gyN8Ej1O-FRBSX_C0ispV9ORT8/edit?usp=sharing > Thanks a lot for any comments and attention to this feature! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8934) [GPG] Add JvmMetricsInfo and pause monitor
[
https://issues.apache.org/jira/browse/YARN-8934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699143#comment-16699143
]
Hadoop QA commented on YARN-8934:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-7402 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
28s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
46s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} YARN-7402 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
43s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-yarn-server-globalpolicygenerator in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8934 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949497/YARN-8934-YARN-7402.v3.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux b9d64e466064
[jira] [Commented] (YARN-9053) Support set environment variables for Docker Containers In nonEntryPoint mode
[
https://issues.apache.org/jira/browse/YARN-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699142#comment-16699142
]
Jim Brennan commented on YARN-9053:
---
{quote}
Environment variables are not written to docker container. After we added it in
non-entry point mode, all -shell_env values are added to containers. Maybe it's
really a bug, addEnv method only used in entry point mode by searching.
{quote}
[~Charo Zhang] It is not a bug. I don't think we need this change. In the
non-entrypoint case, environment variables are written to the launch_script in
LaunchContainer.call() where it calls exec.writeLaunchEnv(). Please do not
change this. We have to be careful about how we expose environment variables
to the container. We write the white-listed variables in a way that allows env
variables defined in the docker container to override them.
Can you provide more details about how things are failing for you? Are some
env variables not being written to the launch_container script? How are you
specifying these variables?
> Support set environment variables for Docker Containers In nonEntryPoint mode
> -
>
> Key: YARN-9053
> URL: https://issues.apache.org/jira/browse/YARN-9053
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Priority: Major
> Labels: Docker
> Attachments: YARN-9053.patch
>
>
> In yarn 3.1.1, users can only set environment variables with "-shell_env" in
> ENTRYPOINT mode, and variables must be registered in
> yarn.nodemanager.env-whitelist.
> But in nonEntryPoint mode, we should allow users to set environment variables
> like "-e KEY=VAULE" in docker run command, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8822) Nvidia-docker v2 support
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699114#comment-16699114 ] Zhankun Tang commented on YARN-8822: [~Charo Zhang] , the patch name should be like YARN-8822-branch-3.1.1.001.path to trigger the Yetus. And take a look at the code, it generally looks good to me beside some check style issues. Have you tested in a real GPU environment? > Nvidia-docker v2 support > > > Key: YARN-8822 > URL: https://issues.apache.org/jira/browse/YARN-8822 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Zhankun Tang >Assignee: Charo Zhang >Priority: Major > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8822.001.patch, YARN-8822.002.patch > > > To run a GPU container with Docker, we have nvdia-docker v1 support already > but is deprecated per > [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We > should support nvdia-docker v2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
[
https://issues.apache.org/jira/browse/YARN-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699055#comment-16699055
]
Hadoop QA commented on YARN-9054:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 1 new + 13 unchanged - 1 fixed = 14 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
57s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
20s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
|
| |
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade.buildGetSubClustersCacheRequest(boolean)
invokes inefficient Boolean constructor; use Boolean.valueOf(...) instead At
FederationStateStoreFacade.java:use Boolean.valueOf(...) instead At
FederationStateStoreFacade.java:[line 467] |
| | Primitive boxed just to call toString in
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade.buildGetSubClustersCacheRequest(boolean)
At FederationStateStoreFacade.java:toString in
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade.buildGetSubClustersCacheRequest(boolean)
At FederationStateStoreFacade.java:[line 468] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9054 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949499/YARN-9054.001.patch |
| Optional
[jira] [Assigned] (YARN-8823) Monitor the healthy state of GPU
[ https://issues.apache.org/jira/browse/YARN-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang reassigned YARN-8823: -- Assignee: Zhankun Tang > Monitor the healthy state of GPU > > > Key: YARN-8823 > URL: https://issues.apache.org/jira/browse/YARN-8823 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > > We have GPU resource discovered when the NM bootstrap but not updated through > later heatbeat with RM. There should be a monitoring mechanism to check GPU > healthy status from time to time and also the corresponding handling. > And YARN-8851 will also handle device's monitoring. There could be some > common part between the two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8821) GPU hierarchy scheduling support
[ https://issues.apache.org/jira/browse/YARN-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang reassigned YARN-8821: -- Assignee: Zhankun Tang > GPU hierarchy scheduling support > > > Key: YARN-8821 > URL: https://issues.apache.org/jira/browse/YARN-8821 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > > GPU topology affects performance dramatically. There's been a discussion in > YARN-7481. But we'd like to move related discussions here. > Please note that YARN-8851 will provide a pluggable device framework which > has a shared scheduler which could support default topology scheduling. And > Based on the framework, GPU plugin could have custom scheduler too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9053) Support set environment variables for Docker Containers In nonEntryPoint mode
[
https://issues.apache.org/jira/browse/YARN-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698983#comment-16698983
]
Hadoop QA commented on YARN-9053:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 58s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9053 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949484/YARN-9053.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a46947ad4ff4 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 23b441c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22702/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22702/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 1) |
| modules
[jira] [Commented] (YARN-8822) Nvidia-docker v2 support
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698978#comment-16698978 ] Charo Zhang commented on YARN-8822: --- [~tangzhankun] I got it. Maybe it's useful for users who are using 3.1.1 version. > Nvidia-docker v2 support > > > Key: YARN-8822 > URL: https://issues.apache.org/jira/browse/YARN-8822 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Zhankun Tang >Assignee: Charo Zhang >Priority: Major > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8822.001.patch, YARN-8822.002.patch > > > To run a GPU container with Docker, we have nvdia-docker v1 support already > but is deprecated per > [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We > should support nvdia-docker v2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
[ https://issues.apache.org/jira/browse/YARN-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698968#comment-16698968 ] Bibin A Chundatt commented on YARN-9054: cc: [~botong] > Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest > -- > > Key: YARN-9054 > URL: https://issues.apache.org/jira/browse/YARN-9054 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-9054.001.patch > > > Currently cache key is create with null argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
[ https://issues.apache.org/jira/browse/YARN-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-9054: --- Attachment: YARN-9054.001.patch > Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest > -- > > Key: YARN-9054 > URL: https://issues.apache.org/jira/browse/YARN-9054 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-9054.001.patch > > > Currently cache key is create with null argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9054) Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest
Bibin A Chundatt created YARN-9054: -- Summary: Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest Key: YARN-9054 URL: https://issues.apache.org/jira/browse/YARN-9054 Project: Hadoop YARN Issue Type: Sub-task Components: federation Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt Currently cache key is create with null argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8934) [GPG] Add JvmMetricsInfo and pause monitor
[ https://issues.apache.org/jira/browse/YARN-8934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated YARN-8934: Attachment: YARN-8934-YARN-7402.v3.patch > [GPG] Add JvmMetricsInfo and pause monitor > -- > > Key: YARN-8934 > URL: https://issues.apache.org/jira/browse/YARN-8934 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-8934-001.patch, YARN-8934-YARN-7402.v1.patch, > YARN-8934-YARN-7402.v2.patch, YARN-8934-YARN-7402.v3.patch, > image-2018-11-19-15-37-18-647.png > > > Similar to resourcemanager and nodemanager serivce we can add JvmMetricsInfo > to gpg service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8822) Nvidia-docker v2 support
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8822: -- Attachment: YARN-8822.002.patch > Nvidia-docker v2 support > > > Key: YARN-8822 > URL: https://issues.apache.org/jira/browse/YARN-8822 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Zhankun Tang >Assignee: Charo Zhang >Priority: Major > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8822.001.patch, YARN-8822.002.patch > > > To run a GPU container with Docker, we have nvdia-docker v1 support already > but is deprecated per > [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We > should support nvdia-docker v2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9049) Add application submit data to state store
[ https://issues.apache.org/jira/browse/YARN-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698931#comment-16698931 ] Bibin A Chundatt commented on YARN-9049: [~subru]/[~botong] Added patch inline with discussion had in YARN-8898 . Could you please help in review. > Add application submit data to state store > -- > > Key: YARN-9049 > URL: https://issues.apache.org/jira/browse/YARN-9049 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: YARN-9049.001.path > > > As per the discussion in YARN-8898 we need to persist trimmend > ApplicationSubmissionContext details to federation State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8821) GPU hierarchy scheduling support
[ https://issues.apache.org/jira/browse/YARN-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-8821: --- Description: GPU topology affects performance dramatically. There's been a discussion in YARN-7481. But we'd like to move related discussions here. Please note that YARN-8851 will provide a pluggable device framework which has a shared scheduler which could support default topology scheduling. And Based on the framework, GPU plugin could have custom scheduler too. was:GPU topology affects performance dramatically. There's been a discussion in YARN-7481. But we'd like to move related discussions here. > GPU hierarchy scheduling support > > > Key: YARN-8821 > URL: https://issues.apache.org/jira/browse/YARN-8821 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Priority: Major > > GPU topology affects performance dramatically. There's been a discussion in > YARN-7481. But we'd like to move related discussions here. > Please note that YARN-8851 will provide a pluggable device framework which > has a shared scheduler which could support default topology scheduling. And > Based on the framework, GPU plugin could have custom scheduler too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8823) Monitor the healthy state of GPU
[ https://issues.apache.org/jira/browse/YARN-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-8823: --- Description: We have GPU resource discovered when the NM bootstrap but not updated through later heatbeat with RM. There should be a monitoring mechanism to check GPU healthy status from time to time and also the corresponding handling. And YARN-8851 will also handle device's monitoring. There could be some common part between the two. was:We have GPU resource discovered when the NM bootstrap but not updated through later heatbeat with RM. There should be a monitoring mechanism to check GPU healthy status from time to time and also the corresponding handling. > Monitor the healthy state of GPU > > > Key: YARN-8823 > URL: https://issues.apache.org/jira/browse/YARN-8823 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Priority: Major > > We have GPU resource discovered when the NM bootstrap but not updated through > later heatbeat with RM. There should be a monitoring mechanism to check GPU > healthy status from time to time and also the corresponding handling. > And YARN-8851 will also handle device's monitoring. There could be some > common part between the two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9053) Support set environment variables for Docker Containers In nonEntryPoint mode
[ https://issues.apache.org/jira/browse/YARN-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-9053: -- Attachment: (was: YARN-9053.patch) > Support set environment variables for Docker Containers In nonEntryPoint mode > - > > Key: YARN-9053 > URL: https://issues.apache.org/jira/browse/YARN-9053 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Priority: Major > Labels: Docker > Attachments: YARN-9053.patch > > > In yarn 3.1.1, users can only set environment variables with "-shell_env" in > ENTRYPOINT mode, and variables must be registered in > yarn.nodemanager.env-whitelist. > But in nonEntryPoint mode, we should allow users to set environment variables > like "-e KEY=VAULE" in docker run command, too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8820) [Umbrella] GPU support on YARN - Phase 2
[ https://issues.apache.org/jira/browse/YARN-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-8820: --- Description: In YARN-6223, we've done a basic support for Nvidia GPU on YARN including resource discovery, allocation, cgroups isolation as well as docker support (Nvidia-docker v1). But there's still room for us to improve. For instance, multiple GPU cards in one host bring the requirements of GPU hierarchy scheduling. The Nvidia-docker v2 emerged and v1 has been deprecated. And we're planning a new device plugin framework in YARN which has relation to GPU support too. (maybe in the long term) So here we converge threads related to the above and open an umbrella here to track the next stage tasks for convenience. One thing to note is that a pluggable device framework is in progress (YARN-8851), once that framework is mature, we should prefer to utilize the ability of the framework to achieve these phase 2 support. was: In YARN-6223, we've done a basic support for Nvidia GPU on YARN including resource discovery, allocation, cgroups isolation as well as docker support (Nvidia-docker v1). But there's still room for us to improve. For instance, multiple GPU cards in one host bring the requirements of GPU hierarchy scheduling. The Nvidia-docker v2 emerged and v1 has been deprecated. And we're planning a new device plugin framework in YARN which has relation to GPU support too. (maybe in the long term) So here we converge threads related to the above and open an umbrella here to track the next stage tasks for convenience. > [Umbrella] GPU support on YARN - Phase 2 > > > Key: YARN-8820 > URL: https://issues.apache.org/jira/browse/YARN-8820 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: Zhankun Tang >Priority: Major > > In YARN-6223, we've done a basic support for Nvidia GPU on YARN including > resource discovery, allocation, cgroups isolation as well as docker support > (Nvidia-docker v1). But there's still room for us to improve. > For instance, multiple GPU cards in one host bring the requirements of GPU > hierarchy scheduling. The Nvidia-docker v2 emerged and v1 has been > deprecated. And we're planning a new device plugin framework in YARN which > has relation to GPU support too. (maybe in the long term) > So here we converge threads related to the above and open an umbrella here to > track the next stage tasks for convenience. > One thing to note is that a pluggable device framework is in progress > (YARN-8851), once that framework is mature, we should prefer to utilize the > ability of the framework to achieve these phase 2 support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8822) Nvidia-docker v2 support
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8822: -- Attachment: YARN-8822.001.patch > Nvidia-docker v2 support > > > Key: YARN-8822 > URL: https://issues.apache.org/jira/browse/YARN-8822 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Zhankun Tang >Assignee: Charo Zhang >Priority: Major > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8822.001.patch > > > To run a GPU container with Docker, we have nvdia-docker v1 support already > but is deprecated per > [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We > should support nvdia-docker v2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8822) Nvidia-docker v2 support
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698902#comment-16698902 ] Zhankun Tang commented on YARN-8822: [~Charo Zhang] , Thanks for the patch. Since a new pluggable device framework (YARN-8851) is in progress, we should not prefer to merge vendor specific code into the YARN code base in the future. I guess it would be better for us to implement this based on that framework. > Nvidia-docker v2 support > > > Key: YARN-8822 > URL: https://issues.apache.org/jira/browse/YARN-8822 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Zhankun Tang >Assignee: Charo Zhang >Priority: Major > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8822.001.patch > > > To run a GPU container with Docker, we have nvdia-docker v1 support already > but is deprecated per > [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We > should support nvdia-docker v2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8822) Nvidia-docker v2 support
[
https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698898#comment-16698898
]
Hadoop QA commented on YARN-8822:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-8822 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8822 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949492/YARN-8822.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22703/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Nvidia-docker v2 support
>
>
> Key: YARN-8822
> URL: https://issues.apache.org/jira/browse/YARN-8822
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Zhankun Tang
>Assignee: Charo Zhang
>Priority: Major
> Labels: Docker
> Fix For: 3.1.2
>
> Attachments: YARN-8822.001.patch
>
>
> To run a GPU container with Docker, we have nvdia-docker v1 support already
> but is deprecated per
> [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We
> should support nvdia-docker v2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8822) Nvidia-docker v2 support
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8822: -- Attachment: (was: YARN-8822.001.patch) > Nvidia-docker v2 support > > > Key: YARN-8822 > URL: https://issues.apache.org/jira/browse/YARN-8822 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Zhankun Tang >Assignee: Charo Zhang >Priority: Major > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8822.001.patch > > > To run a GPU container with Docker, we have nvdia-docker v1 support already > but is deprecated per > [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We > should support nvdia-docker v2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3890) FairScheduler should show the scheduler health metrics similar to ones added in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl updated YARN-3890: --- Attachment: YARN-3890.002.patch > FairScheduler should show the scheduler health metrics similar to ones added > in CapacityScheduler > - > > Key: YARN-3890 > URL: https://issues.apache.org/jira/browse/YARN-3890 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Zoltan Siegl >Priority: Major > Attachments: YARN-3890.001.patch, YARN-3890.002.patch > > > We should add information displayed in YARN-3293 in FairScheduler as well > possibly sharing the implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9051) Integrate multiple CustomResourceTypesConfigurationProvider implementations into one
[
https://issues.apache.org/jira/browse/YARN-9051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698889#comment-16698889
]
Antal Bálint Steinbach commented on YARN-9051:
--
LGTM (+1 Non-binding)
> Integrate multiple CustomResourceTypesConfigurationProvider implementations
> into one
>
>
> Key: YARN-9051
> URL: https://issues.apache.org/jira/browse/YARN-9051
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-9051.001.patch
>
>
> CustomResourceTypesConfigurationProvider (extends LocalConfigurationProvider)
> has 5 implementations on trunk nowadays.
> These could be integrated into 1 common class.
> Also,
> {{org.apache.hadoop.yarn.util.resource.TestResourceUtils#addNewTypesToResources}}
> has similar functionality so this can be considered as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8822) Nvidia-docker v2 support
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698876#comment-16698876 ] Charo Zhang edited comment on YARN-8822 at 11/26/18 12:52 PM: -- 1，nvidia-docker2 [Frequently Asked Questions|https://github.com/NVIDIA/nvidia-docker/wiki] 2，when yarn.nodemanager.resource-plugins.gpu.docker-plugin is nvidia-docker-v2, nodemanager will use nvdia-docker v2. 3，other nvidia-docker2 environment variables like NVIDIA_DRIVER_CAPABILITIES can set by -shell_env. 4，if non-entrypoint mode can not add environment variables, you could merge [YARN-9053|https://issues.apache.org/jira/browse/YARN-9053?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=16698231#comment-16698231]. was (Author: charo zhang): 1，nvidia-docker2 [Frequently Asked Questions|https://github.com/NVIDIA/nvidia-docker/wiki] 2，when yarn.nodemanager.resource-plugins.gpu.docker-plugin is nvidia-docker-v2, nodemanager will use nvdia-docker v2. 3，other nvidia-docker2 environment variables like NVIDIA_DRIVER_CAPABILITIES can set by -shell_env. 4，if non-entrypoint mode can not add environment variables, you could merge [YARN-9053|https://issues.apache.org/jira/browse/YARN-9053?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=16698231#comment-16698231] > Nvidia-docker v2 support > > > Key: YARN-8822 > URL: https://issues.apache.org/jira/browse/YARN-8822 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Zhankun Tang >Assignee: Charo Zhang >Priority: Major > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8822.001.patch > > > To run a GPU container with Docker, we have nvdia-docker v1 support already > but is deprecated per > [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We > should support nvdia-docker v2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8822) Nvidia-docker v2 support
[
https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698880#comment-16698880
]
Hadoop QA commented on YARN-8822:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-8822 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8822 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949488/YARN-8822.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22701/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Nvidia-docker v2 support
>
>
> Key: YARN-8822
> URL: https://issues.apache.org/jira/browse/YARN-8822
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Zhankun Tang
>Assignee: Charo Zhang
>Priority: Major
> Labels: Docker
> Fix For: 3.1.2
>
> Attachments: YARN-8822.001.patch
>
>
> To run a GPU container with Docker, we have nvdia-docker v1 support already
> but is deprecated per
> [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We
> should support nvdia-docker v2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8822) Nvidia-docker v2 support
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang reassigned YARN-8822: - Assignee: Charo Zhang Fix Version/s: 3.1.2 Affects Version/s: 3.1.1 Attachment: YARN-8822.001.patch Target Version/s: 3.1.2 1，nvidia-docker2 [Frequently Asked Questions|https://github.com/NVIDIA/nvidia-docker/wiki] 2，when yarn.nodemanager.resource-plugins.gpu.docker-plugin is nvidia-docker-v2, nodemanager will use nvdia-docker v2. 3，other nvidia-docker2 environment variables like NVIDIA_DRIVER_CAPABILITIES can set by -shell_env. 4，if non-entrypoint mode can not add environment variables, you could merge [YARN-9053|https://issues.apache.org/jira/browse/YARN-9053?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=16698231#comment-16698231] > Nvidia-docker v2 support > > > Key: YARN-8822 > URL: https://issues.apache.org/jira/browse/YARN-8822 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Zhankun Tang >Assignee: Charo Zhang >Priority: Major > Labels: Docker > Fix For: 3.1.2 > > Attachments: YARN-8822.001.patch > > > To run a GPU container with Docker, we have nvdia-docker v1 support already > but is deprecated per > [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We > should support nvdia-docker v2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
