[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695628#comment-16695628
]
Xun Liu edited comment on YARN-8986 at 11/22/18 7:49 AM:
-
hi, [~Charo Zhang], [~eyang]
{code:java}
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}{code}
Not a bridge network, should it be able to perform port mapping as well?
We now use docker to create a parallel network via calico.
In the Hadoop submarine project, we still need to expose the services in the
container (for example: tensorboard, zeppelin, etc.).
was (Author: liuxun323):
hi, [~Charo Zhang], [~eyang]
{code:java}
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }{code}
Not a bridge network, should it be able to perform port mapping as well?
We now use docker to create a parallel network via calico.
In the Hadoop submarine project, we still need to expose the services in the
container (for example: tensorboard, zeppelin, etc.).
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695628#comment-16695628
]
Xun Liu commented on YARN-8986:
---
hi, [~Charo Zhang], [~eyang]
{code:java}
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }{code}
Not a bridge network, should it be able to perform port mapping as well?
We now use docker to create a parallel network via calico.
In the Hadoop submarine project, we still need to expose the services in the
container (for example: tensorboard, zeppelin, etc.).
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8986: -- Attachment: (was: YARN-8986.008.patch) > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8986: -- Attachment: YARN-8986.008.patch > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9034: Attachment: YARN-9034.04.patch > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch, YARN-9034.04.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695589#comment-16695589
]
Hadoop QA commented on YARN-8986:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 27s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | TEST-cetest |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8986 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949141/YARN-8986.008.patch |
| Optional Tests | dupname asflicense
[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695586#comment-16695586
]
Hadoop QA commented on YARN-8714:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-submarine in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-submarine in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 22s{color}
| {color:red} hadoop-yarn-submarine in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine:
The patch generated 17 new + 47 unchanged - 5 fixed = 64 total (was 52) {color}
|
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-submarine in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} hadoop-yarn-submarine in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 24s{color}
| {color:red} hadoop-yarn-submarine in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8714 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949142/YARN-8714-trunk.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux daa84ae78c4b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f207e30 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/22674/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-submarine.txt
|
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/22674/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-submarine.txt
|
|
[jira] [Commented] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695578#comment-16695578 ] Rohith Sharma K S commented on YARN-9034: - I found test error and working on correcting it. > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8984) AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty
[ https://issues.apache.org/jira/browse/YARN-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695571#comment-16695571 ] Weiwei Yang commented on YARN-8984: --- Thanks [~fly_in_gis] and thanks [~kkaranasos] for the additional review, I will commit this today if no further comments. > AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty > -- > > Key: YARN-8984 > URL: https://issues.apache.org/jira/browse/YARN-8984 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yang Wang >Assignee: Yang Wang >Priority: Critical > Attachments: YARN-8984-001.patch, YARN-8984-002.patch, > YARN-8984-003.patch, YARN-8984-004.patch, YARN-8984-005.patch > > > In AMRMClient, outstandingSchedRequests should be removed or decreased when > container allocated. However, it could not work when allocation tag is null > or empty. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Deleted] (YARN-4803) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka deleted YARN-4803:
> ContainerMetrics drops records
> --
>
> Key: YARN-4803
> URL: https://issues.apache.org/jira/browse/YARN-4803
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Deleted] (YARN-4802) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka deleted YARN-4802:
> ContainerMetrics drops records
> --
>
> Key: YARN-4802
> URL: https://issues.apache.org/jira/browse/YARN-4802
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Deleted] (YARN-4801) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka deleted YARN-4801:
> ContainerMetrics drops records
> --
>
> Key: YARN-4801
> URL: https://issues.apache.org/jira/browse/YARN-4801
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Deleted] (YARN-4798) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka deleted YARN-4798:
> ContainerMetrics drops records
> --
>
> Key: YARN-4798
> URL: https://issues.apache.org/jira/browse/YARN-4798
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Deleted] (YARN-4799) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka deleted YARN-4799:
> ContainerMetrics drops records
> --
>
> Key: YARN-4799
> URL: https://issues.apache.org/jira/browse/YARN-4799
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Deleted] (YARN-4800) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka deleted YARN-4800:
> ContainerMetrics drops records
> --
>
> Key: YARN-4800
> URL: https://issues.apache.org/jira/browse/YARN-4800
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Deleted] (YARN-4797) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka deleted YARN-4797:
> ContainerMetrics drops records
> --
>
> Key: YARN-4797
> URL: https://issues.apache.org/jira/browse/YARN-4797
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-8714:
---
Description: See
[https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7],
{{job run --localization ...}} (was: See
https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
{{job run --localizations ...}})
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch
>
>
> See
> [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7],
> {{job run --localization ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-8714:
---
Attachment: YARN-8714-trunk.001.patch
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch, YARN-8714-trunk.001.patch
>
>
> See
> [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7],
> {{job run --localization ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8964) [UI2] YARN ui2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695564#comment-16695564
]
Hudson commented on YARN-8964:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15492 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15492/])
YARN-8964. [UI2] YARN ui2 should use clusters/{cluster name} for all (sunilg:
rev 5fe1dbf1959976d0dc5a8e614dd74836cfbee04c)
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-log-service.hbs
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-flow-activity.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-service-component.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/utils/converter.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/initializers/loader.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-service-info.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-timeline-appattempt.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/config/default-config.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-log.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-log.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-component-instance.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/router.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-timeline.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-timeline-container.js
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-log.js
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/yarn-log-service.js
> [UI2] YARN ui2 should use clusters/{cluster name} for all ATSv2 REST APIs
> -
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch,
> YARN-8964.003.patch, YARN-8964.004.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8964) [UI2] YARN ui2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated YARN-8964:
-
Summary: [UI2] YARN ui2 should use clusters/{cluster name} for all ATSv2
REST APIs (was: UI2 should use clusters/{cluster name} for all ATSv2 REST APIs)
> [UI2] YARN ui2 should use clusters/{cluster name} for all ATSv2 REST APIs
> -
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch,
> YARN-8964.003.patch, YARN-8964.004.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8986: -- Attachment: YARN-8986.008.patch > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8986: -- Attachment: (was: YARN-8986.008.patch) > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695554#comment-16695554
]
Sunil Govindan commented on YARN-8964:
--
+1 Committing shortly
> UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
> --
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch,
> YARN-8964.003.patch, YARN-8964.004.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[
https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695550#comment-16695550
]
Hadoop QA commented on YARN-9045:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 23s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 218 unchanged - 1 fixed = 223 total (was 219) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 50s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
28s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9045 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949136/YARN-9045.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f11c6829e064 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695549#comment-16695549
]
Hadoop QA commented on YARN-8986:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 31s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | TEST-cetest |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8986 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949137/YARN-8986.008.patch |
| Optional Tests | dupname asflicense
[jira] [Commented] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695514#comment-16695514 ] Rohith Sharma K S commented on YARN-9034: - UT failure is unrelated to this patch. I checked the UT logs and all tests are passed from TestYARNCLi.java. One test is skipped from TestAMRMClient. This may the reason jenkin is showing red. No failures in test run! Reg the checkstyle about "Line is longer than 80 characters", those are required as-is in test to compare the help messages. These could be ignored. Whitespace could be fixed during committing patch using "--whitespace=" in git apply command. > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8986: -- Attachment: YARN-8986.008.patch > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9042) Javadoc error in deviceplugin package
[
https://issues.apache.org/jira/browse/YARN-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang reassigned YARN-9042:
--
Assignee: Zhankun Tang
> Javadoc error in deviceplugin package
> -
>
> Key: YARN-9042
> URL: https://issues.apache.org/jira/browse/YARN-9042
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Zhankun Tang
>Priority: Major
>
> Many java doc errors are in deviceplugin
> {noformat}
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:29:
> error: bad HTML entity
> [ERROR] * This is a spec used to prepare & run container.
> [ERROR] ^
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:35:
> error: bad HTML entity
> [ERROR] * The volume & device mounts describes key isolation requirements
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
> error: unknown tag: domain
> [ERROR]* PCI Bus ID in format ]:]]:][][.[]].
> [ERROR] ^
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
> error: unknown tag: bus
> [ERROR]* PCI Bus ID in format ]:]]:][][.[]].
> [ERROR] ^
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
> error: unknown tag: slot
> [ERROR]* PCI Bus ID in format ]:]]:][][.[]].
> [ERROR] ^
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
> error: unknown tag: func
> [ERROR]* PCI Bus ID in format ]:]]:][][.[]].
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9042) Javadoc error in deviceplugin package
[
https://issues.apache.org/jira/browse/YARN-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695507#comment-16695507
]
Zhankun Tang commented on YARN-9042:
Thanks [~rohithsharma] for pointing this. I'll fix it. :)
> Javadoc error in deviceplugin package
> -
>
> Key: YARN-9042
> URL: https://issues.apache.org/jira/browse/YARN-9042
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Major
>
> Many java doc errors are in deviceplugin
> {noformat}
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:29:
> error: bad HTML entity
> [ERROR] * This is a spec used to prepare & run container.
> [ERROR] ^
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:35:
> error: bad HTML entity
> [ERROR] * The volume & device mounts describes key isolation requirements
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
> error: unknown tag: domain
> [ERROR]* PCI Bus ID in format ]:]]:][][.[]].
> [ERROR] ^
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
> error: unknown tag: bus
> [ERROR]* PCI Bus ID in format ]:]]:][][.[]].
> [ERROR] ^
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
> error: unknown tag: slot
> [ERROR]* PCI Bus ID in format ]:]]:][][.[]].
> [ERROR] ^
> [ERROR]
> /Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
> error: unknown tag: func
> [ERROR]* PCI Bus ID in format ]:]]:][][.[]].
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8882) Phase 1 - Add a shared device mapping manager for device plugin to use
[ https://issues.apache.org/jira/browse/YARN-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695504#comment-16695504 ] Zhankun Tang commented on YARN-8882: [~leftnoteasy] , Yeah. And it will include a list of device plugin scheduler if we merge the customized device scheduler interface. Do we have a better name than "DeviceScheduler"? the original "DeviceSchedulerManager"? > Phase 1 - Add a shared device mapping manager for device plugin to use > -- > > Key: YARN-8882 > URL: https://issues.apache.org/jira/browse/YARN-8882 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-8882-trunk.001.patch, YARN-8882-trunk.002.patch, > YARN-8882-trunk.003.patch, YARN-8882-trunk.004.patch, > YARN-8882-trunk.005.patch, YARN-8882-trunk.006.patch, > YARN-8882-trunk.007.patch > > > Since a few devices uses FIFO policy to assign devices to the container, we > use a shared device manager to handle all types of devices. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Attachment: (was: YARN-9045.patch) > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: YARN-9045.001.patch, YARN-9045.002.patch, > shared_mode.png, strict_mode.png, strict_relax_mode.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if available. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Attachments are different modes comparison. Especifically, the last picture > is the maximum percentage of container using CPU when the relaxation > coefficient is 1.5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Attachment: YARN-9045.001.patch YARN-9045.002.patch > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: YARN-9045.001.patch, YARN-9045.002.patch, > shared_mode.png, strict_mode.png, strict_relax_mode.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if available. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Attachments are different modes comparison. Especifically, the last picture > is the maximum percentage of container using CPU when the relaxation > coefficient is 1.5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695501#comment-16695501
]
Xun Liu commented on YARN-8714:
---
Now when submarine uses an external algorithm file, Only need to have read and
execute permissions to meet the needs. Either submit your current
implementation first.
I feel , From the perspective of YARN, the requirements for file permissions
after mount, It should be that the original file and the file permissions after
mount should be consistent. Still ask [~leftnoteasy] for advice.
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Description: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if available. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. Attachments are different modes comparison. Especifically, the last picture is the maximum percentage of container using CPU when the relaxation coefficient is 1.5. was: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. Attachments are different modes comparison. Especifically, the last picture is the maximum percentage of container using CPU when the relaxation coefficient is 1.5. > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: YARN-9045.patch, shared_mode.png, strict_mode.png, > strict_relax_mode.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if available. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Attachments are different modes comparison. Especifically, the last picture > is the maximum percentage of container using CPU when the relaxation > coefficient is 1.5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8984) AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty
[ https://issues.apache.org/jira/browse/YARN-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695464#comment-16695464 ] Konstantinos Karanasos commented on YARN-8984: -- Looks good to me, thanks [~fly_in_gis]. > AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty > -- > > Key: YARN-8984 > URL: https://issues.apache.org/jira/browse/YARN-8984 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yang Wang >Assignee: Yang Wang >Priority: Critical > Attachments: YARN-8984-001.patch, YARN-8984-002.patch, > YARN-8984-003.patch, YARN-8984-004.patch, YARN-8984-005.patch > > > In AMRMClient, outstandingSchedRequests should be removed or decreased when > container allocated. However, it could not work when allocation tag is null > or empty. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9030) Log aggregation changes to handle filesystems which do not support setting permissions
[
https://issues.apache.org/jira/browse/YARN-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-9030:
-
Summary: Log aggregation changes to handle filesystems which do not support
setting permissions (was: Log aggregation changes to handle filesystems which
do not support permissions)
> Log aggregation changes to handle filesystems which do not support setting
> permissions
> --
>
> Key: YARN-9030
> URL: https://issues.apache.org/jira/browse/YARN-9030
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
> Attachments: YARN-9030.1.patch, YARN-9030.2.patch
>
>
> Some cloud storages like ADLS do not support permissions in which case they
> throw an UnsupportedOperationException. Log aggregation code should
> log/ignore these exceptions and not set permissions henceforth for log
> aggregation base dir/sub dirs
> {noformat}
> 2018-11-12 15:37:28,726 WARN logaggregation.LogAggregationService
> (LogAggregationService.java:initApp(209)) - Application failed to init
> aggregation
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to check
> permissions for dir [abfs://testc...@test.blob.core.windows.net/app-logs]
> at
> org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController.verifyAndCreateRemoteLogDir(LogAggregationFileController.java:277)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initAppAggregator(LogAggregationService.java:238)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:204)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:347)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:69)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8882) Phase 1 - Add a shared device mapping manager for device plugin to use
[ https://issues.apache.org/jira/browse/YARN-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695450#comment-16695450 ] Wangda Tan commented on YARN-8882: -- [~tangzhankun], why rename "device-scheduler" to "device-mapping-manager"? :) I understand what it do is mapping devices to containers. But I would prefer to call it DeviceScheduler instead of DeviceMappingManager given name "scheduler" means booking, ordering, etc. > Phase 1 - Add a shared device mapping manager for device plugin to use > -- > > Key: YARN-8882 > URL: https://issues.apache.org/jira/browse/YARN-8882 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-8882-trunk.001.patch, YARN-8882-trunk.002.patch, > YARN-8882-trunk.003.patch, YARN-8882-trunk.004.patch, > YARN-8882-trunk.005.patch, YARN-8882-trunk.006.patch, > YARN-8882-trunk.007.patch > > > Since a few devices uses FIFO policy to assign devices to the container, we > use a shared device manager to handle all types of devices. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8867) Retrieve the status of resource localization
[
https://issues.apache.org/jira/browse/YARN-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695412#comment-16695412
]
Hadoop QA commented on YARN-8867:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
5s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 58s{color} | {color:orange} root: The patch generated 2 new + 516 unchanged
- 1 fixed = 518 total (was 517) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
24s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
28s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
2s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 26s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 26m
7s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 56s{color}
| {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
32s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color}
[jira] [Assigned] (YARN-8118) Better utilize gracefully decommissioning node managers
[ https://issues.apache.org/jira/browse/YARN-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash reassigned YARN-8118: -- Assignee: Karthik Palaniappan > Better utilize gracefully decommissioning node managers > --- > > Key: YARN-8118 > URL: https://issues.apache.org/jira/browse/YARN-8118 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 2.8.2 > Environment: * Google Compute Engine (Dataproc) > * Java 8 > * Hadoop 2.8.2 using client-mode graceful decommissioning >Reporter: Karthik Palaniappan >Assignee: Karthik Palaniappan >Priority: Major > Attachments: YARN-8118-branch-2.001.patch > > > Proposal design doc with background + details (please comment directly on > doc): > [https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7] > tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications > to complete before shutting down, but they cannot run new containers from > those in-progress applications. This is wasteful, particularly in > environments where you are billed by resource usage (e.g. EC2). > Proposal: YARN should schedule containers from in-progress applications on > DECOMMISSIONING nodes, but should still avoid scheduling containers from new > applications. That will make in-progress applications complete faster and let > nodes decommission faster. Overall, this should be cheaper. > I have a working patch without unit tests that's surprisingly just a few real > lines of code (patch 001). If folks are happy with the proposal, I'll write > unit tests and also write a patch targeted at trunk. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6523) Newly retrieved security Tokens are sent as part of each heartbeat to each node from RM which is not desirable in large cluster
[
https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695280#comment-16695280
]
Hadoop QA commented on YARN-6523:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 29s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 12 new + 171 unchanged - 4 fixed = 183 total (was 175) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
20s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
24s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
35s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 17s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
51s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}222m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-6523 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949085/YARN-6523.007.patch |
| Optional Tests | dupname
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695233#comment-16695233
]
Hadoop QA commented on YARN-8948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949084/YARN-8948.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 14346a6eb061 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 34b6fa7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22668/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Updated] (YARN-8867) Retrieve the status of resource localization
[
https://issues.apache.org/jira/browse/YARN-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-8867:
Attachment: YARN-8867.005.patch
> Retrieve the status of resource localization
>
>
> Key: YARN-8867
> URL: https://issues.apache.org/jira/browse/YARN-8867
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Attachments: YARN-8867.001.patch, YARN-8867.002.patch,
> YARN-8867.003.patch, YARN-8867.004.patch, YARN-8867.005.patch,
> YARN-8867.wip.patch
>
>
> Refer YARN-3854.
> Currently NM does not have an API to retrieve the status of localization.
> Unless the client can know when the localization of a resource is complete
> irrespective of the type of the resource, it cannot take any appropriate
> action.
> We need an API in {{ContainerManagementProtocol}} to retrieve the status on
> the localization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9046) Fix overlapping netty classes warnings when building YARN CSI module
[
https://issues.apache.org/jira/browse/YARN-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated YARN-9046:
-
Labels: build (was: )
> Fix overlapping netty classes warnings when building YARN CSI module
>
>
> Key: YARN-9046
> URL: https://issues.apache.org/jira/browse/YARN-9046
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Priority: Major
> Labels: build
>
> {code}
> [*WARNING*] netty-codec-4.1.27.Final.jar, netty-all-4.1.27.Final.jar define
> 197 overlapping classes:
> [*WARNING*] - io.netty.handler.codec.compression.ZlibUtil
> [*WARNING*] - io.netty.handler.codec.CharSequenceValueConverter
> [*WARNING*] - io.netty.handler.codec.compression.Bzip2Encoder$3
> [*WARNING*] - io.netty.handler.codec.DefaultHeaders$1
> [*WARNING*] - io.netty.handler.codec.compression.Bzip2BlockDecompressor
> [*WARNING*] - io.netty.handler.codec.compression.Lz4FrameDecoder$1
> [*WARNING*] - io.netty.handler.codec.compression.SnappyFrameDecoder
> [*WARNING*] - io.netty.handler.codec.compression.DecompressionException
> [*WARNING*] - io.netty.handler.codec.compression.Bzip2DivSufSort$TRBudget
> [*WARNING*] - io.netty.handler.codec.compression.Bzip2Decoder$State
> [*WARNING*] - 187 more...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9046) Fix overlapping netty classes warnings when building YARN CSI module
Xiaoyu Yao created YARN-9046:
Summary: Fix overlapping netty classes warnings when building YARN
CSI module
Key: YARN-9046
URL: https://issues.apache.org/jira/browse/YARN-9046
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xiaoyu Yao
{code}
[*WARNING*] netty-codec-4.1.27.Final.jar, netty-all-4.1.27.Final.jar define 197
overlapping classes:
[*WARNING*] - io.netty.handler.codec.compression.ZlibUtil
[*WARNING*] - io.netty.handler.codec.CharSequenceValueConverter
[*WARNING*] - io.netty.handler.codec.compression.Bzip2Encoder$3
[*WARNING*] - io.netty.handler.codec.DefaultHeaders$1
[*WARNING*] - io.netty.handler.codec.compression.Bzip2BlockDecompressor
[*WARNING*] - io.netty.handler.codec.compression.Lz4FrameDecoder$1
[*WARNING*] - io.netty.handler.codec.compression.SnappyFrameDecoder
[*WARNING*] - io.netty.handler.codec.compression.DecompressionException
[*WARNING*] - io.netty.handler.codec.compression.Bzip2DivSufSort$TRBudget
[*WARNING*] - io.netty.handler.codec.compression.Bzip2Decoder$State
[*WARNING*] - 187 more...
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695040#comment-16695040
]
Eric Yang edited comment on YARN-8986 at 11/21/18 6:04 PM:
---
[~Charo Zhang] Jenkins runs with docker 17.05.0-ce. Hadoop compilation runs
inside docker container, which does not have docker binary available to unit
test. This is the reason that it fails. If you change
add_ports_mapping_to_command, then it will work correctly:
{code}
+ char *network_name = get_configuration_value("net",
DOCKER_COMMAND_FILE_SECTION, command_config);
+ if (network_name == NULL) {
+network_name = make_string("%s", "bridge");
+ }
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+ FILE* docker_network = popen(docker_network_command, "r");
+ ret = fscanf(docker_network, "%s", network_type);
+ if (pclose (docker_network) != 0 || ret <= 0) {
+fprintf (ERRORFILE, "Could not inspect docker network to get type %s.\n",
docker_network_command);
+goto cleanup;
+ }
{code}
to
{code}
+ char *network_name = get_configuration_value("net",
DOCKER_COMMAND_FILE_SECTION, command_config);
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }
{code}
If network name is not specified, it will be allowed to use -p and -P flags.
This side step the requirement to have docker binary available to unit test and
validate the default case is working. It would be possible the refactor the
first 3 lines into another function and test the docker_network_command for
null and also artificial name for generating docker_network_command for maximum
coverage, but it is okay if you don't do the refactoring because some existing
unit tests should not fail when net parameter is not passed. It would be okay
to skip the actual docker command execution in unit test, since we already unit
test that part manually in our discussions.
was (Author: eyang):
[~Charo Zhang] Jenkins runs with docker 17.05.0-ce. Hadoop compilation runs
inside docker container, which does not have docker binary available to unit
test. This is the reason that it fails. If you change
add_ports_mapping_to_command, then it will work correctly:
{code}
+ if (network_name == NULL) {
+network_name = make_string("%s", "bridge");
+ }
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+ FILE* docker_network = popen(docker_network_command, "r");
+ ret = fscanf(docker_network, "%s", network_type);
+ if (pclose (docker_network) != 0 || ret <= 0) {
+fprintf (ERRORFILE, "Could not inspect docker network to get type %s.\n",
docker_network_command);
+goto cleanup;
+ }
{code}
to
{code}
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }
{code}
If network name is not specified, it will be allowed to use -p and -P flags.
This side step the requirement to have docker binary available to unit test and
validate the default case is working. It would be possible the refactor the
first 3 lines into another function and test the docker_network_command for
null and also artificial name for generating
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695040#comment-16695040
]
Eric Yang commented on YARN-8986:
-
[~Charo Zhang] Jenkins runs with docker 17.05.0-ce. Hadoop compilation runs
inside docker container, which does not have docker binary available to unit
test. This is the reason that it fails. If you change
add_ports_mapping_to_command, then it will work correctly:
{code}
+ if (network_name == NULL) {
+network_name = make_string("%s", "bridge");
+ }
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+ FILE* docker_network = popen(docker_network_command, "r");
+ ret = fscanf(docker_network, "%s", network_type);
+ if (pclose (docker_network) != 0 || ret <= 0) {
+fprintf (ERRORFILE, "Could not inspect docker network to get type %s.\n",
docker_network_command);
+goto cleanup;
+ }
{code}
to
{code}
+ char **ports_mapping_values =
get_configuration_values_delimiter("ports-mapping",
DOCKER_COMMAND_FILE_SECTION, command_config, ",");
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }
{code}
If network name is not specified, it will be allowed to use -p and -P flags.
This side step the requirement to have docker binary available to unit test and
validate the default case is working. It would be possible the refactor the
first 3 lines into another function and test the docker_network_command for
null and also artificial name for generating docker_network_command for maximum
coverage, but it is okay if you don't do the refactoring because some existing
unit tests should not fail when net parameter is not passed. It would be okay
to skip the actual docker command execution in unit test, since we already unit
test that part manually in our discussions.
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6523) Newly retrieved security Tokens are sent as part of each heartbeat to each node from RM which is not desirable in large cluster
[ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-6523: --- Attachment: YARN-6523.007.patch > Newly retrieved security Tokens are sent as part of each heartbeat to each > node from RM which is not desirable in large cluster > --- > > Key: YARN-6523 > URL: https://issues.apache.org/jira/browse/YARN-6523 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Affects Versions: 2.8.0, 2.7.3 >Reporter: Naganarasimha G R >Assignee: Manikandan R >Priority: Major > Attachments: YARN-6523.001.patch, YARN-6523.002.patch, > YARN-6523.003.patch, YARN-6523.004.patch, YARN-6523.005.patch, > YARN-6523.006.patch, YARN-6523.007.patch > > > Currently as part of heartbeat response RM sets all application's tokens > though all applications might not be active on the node. On top of it > NodeHeartbeatResponsePBImpl converts tokens for each app into > SystemCredentialsForAppsProto. Hence for each node and each heartbeat too > many SystemCredentialsForAppsProto objects were getting created. > We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with > 8GB RAM configured for RM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6523) Newly retrieved security Tokens are sent as part of each heartbeat to each node from RM which is not desirable in large cluster
[
https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695037#comment-16695037
]
Manikandan R commented on YARN-6523:
[~jlowe] Thanks for your detailed explanation.
I've made changes to compute SystemCredentialsForAppsProto in case of any
changes in credentials only once in
NodeHeartbeatResponsePBImpl.setSystemCredentialsForApps itself so that
SystemCredentialsForAppsProto computation can be avoided when every time
NodeHeartbeatResponsePBImpl.getProto() call happens. Thoughts?
{quote}Checking for hasTokenSequenceNo and returning zero if not present is
redundant.{quote}
Taken care
{quote}My concerns about the unit test duration have not been addressed. This
single unit test takes almost two minutes to execute{quote}
Sorry, thought of configuring appropriate token expiry time but missed this in
earlier patch. Taken care now. It has reduced overall unit tests duration
significantly.
> Newly retrieved security Tokens are sent as part of each heartbeat to each
> node from RM which is not desirable in large cluster
> ---
>
> Key: YARN-6523
> URL: https://issues.apache.org/jira/browse/YARN-6523
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: RM
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Naganarasimha G R
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-6523.001.patch, YARN-6523.002.patch,
> YARN-6523.003.patch, YARN-6523.004.patch, YARN-6523.005.patch,
> YARN-6523.006.patch
>
>
> Currently as part of heartbeat response RM sets all application's tokens
> though all applications might not be active on the node. On top of it
> NodeHeartbeatResponsePBImpl converts tokens for each app into
> SystemCredentialsForAppsProto. Hence for each node and each heartbeat too
> many SystemCredentialsForAppsProto objects were getting created.
> We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with
> 8GB RAM configured for RM
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-8948:
---
Attachment: YARN-8948.004.patch
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695027#comment-16695027
]
Bibin A Chundatt commented on YARN-8948:
[~wilfreds]/[~suma.shivaprasad]/[~sunilg]
Attached patch.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[
https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694996#comment-16694996
]
Hadoop QA commented on YARN-9045:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 31s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 5 new + 218 unchanged - 1 fixed = 223 total (was 219) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 49s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
47s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9045 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949067/YARN-9045.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 587bcd79f46b 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-9039) App ACLs are not validated when serving logs from Logs CLI/Yarn UI2
[ https://issues.apache.org/jira/browse/YARN-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694929#comment-16694929 ] Bibin A Chundatt commented on YARN-9039: Thank you [~suma.shivaprasad] for discussion. # We might have to fix LogAggregationHtmlBlock#checkAcls too. About solution for the problem, I was reading though S3 documentation. Can we make use of IAM folder and permission control. # Configure separate bucket for logaggregation.(remote log directory) # Following IAM configuration for nodemanager and JHS(Deletion service runs in JHS) https://aws.amazon.com/blogs/security/writing-iam-policies-how-to-grant-access-to-an-amazon-s3-bucket/ For individual users folder access control using. https://aws.amazon.com/blogs/security/writing-iam-policies-grant-access-to-user-specific-folders-in-an-amazon-s3-bucket/ Will this work ?? > App ACLs are not validated when serving logs from Logs CLI/Yarn UI2 > --- > > Key: YARN-9039 > URL: https://issues.apache.org/jira/browse/YARN-9039 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-9039.1.patch, YARN-9039.2.patch > > > App Acls are not being validated when serving logs through YARN CLI. > This also applies while serving logs through YARN UIV2 through ATSV2 Log > Webservice -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9039) App ACLs are not validated when serving logs from Logs CLI/Yarn UI2
[ https://issues.apache.org/jira/browse/YARN-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694912#comment-16694912 ] Suma Shivaprasad commented on YARN-9039: [~bibinchundatt] Thats correct. Discussed with [~rohithsharma] and [~sunilg] and they confirmed the same - If we remove the line adminAcl.addUser , it would break the "process started user" as admin user usecase and may break third party code as well which may use this class. Hence will need to think of another approach of addressing this. > App ACLs are not validated when serving logs from Logs CLI/Yarn UI2 > --- > > Key: YARN-9039 > URL: https://issues.apache.org/jira/browse/YARN-9039 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-9039.1.patch, YARN-9039.2.patch > > > App Acls are not being validated when serving logs through YARN CLI. > This also applies while serving logs through YARN UIV2 through ATSV2 Log > Webservice -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694901#comment-16694901 ] Suma Shivaprasad commented on YARN-9034: [~rohithsharma] Patch LGTM. Can you pls check UT failures/whitespace/CS issues > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694895#comment-16694895
]
Hadoop QA commented on YARN-8986:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 18s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | TEST-cetest |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8986 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949052/YARN-8986.007.patch |
| Optional Tests | dupname asflicense
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Description: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. Attachments are different modes comparison. Especifically, the last one is the maximum percentage of container using CPU when the relaxation coefficient is 1.5. was: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. Attachments are different modes comparison. Epecifically, the last one is the maximum percentage of container using CPU when the relaxation coefficient is 1.5. > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: shared_mode.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Attachments are different modes comparison. Especifically, the last one is > the maximum percentage of container using CPU when the relaxation coefficient > is 1.5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Attachment: shared_mode.png > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: shared_mode.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Attachments are different modes comparison. Especifically, the last one is > the maximum percentage of container using CPU when the relaxation coefficient > is 1.5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Description: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. Attachments are different modes comparison. Especifically, the last picture is the maximum percentage of container using CPU when the relaxation coefficient is 1.5. was: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. Attachments are different modes comparison. Especifically, the last one is the maximum percentage of container using CPU when the relaxation coefficient is 1.5. > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: shared_mode.png, strict_mode.png, strict_relax_mode.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Attachments are different modes comparison. Especifically, the last picture > is the maximum percentage of container using CPU when the relaxation > coefficient is 1.5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Attachment: strict_relax_mode.png > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: shared_mode.png, strict_mode.png, strict_relax_mode.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Attachments are different modes comparison. Especifically, the last one is > the maximum percentage of container using CPU when the relaxation coefficient > is 1.5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Attachment: image-2018-11-21-23-18-41-366.png > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: image-2018-11-21-23-18-41-366.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Share Mode， > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Attachment: strict_mode.png > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: shared_mode.png, strict_mode.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Attachments are different modes comparison. Especifically, the last one is > the maximum percentage of container using CPU when the relaxation coefficient > is 1.5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Description: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. Attachments are different modes comparison. Epecifically, the last one is the maximum percentage of container using CPU when the relaxation coefficient is 1.5. was: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. Share Mode， > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Attachments are different modes comparison. Epecifically, the last one is the > maximum percentage of container using CPU when the relaxation coefficient is > 1.5. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Attachment: (was: image-2018-11-21-23-18-41-366.png) > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Share Mode， > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Description: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. Share Mode， was: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: image-2018-11-21-23-18-41-366.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > > Share Mode， > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
[ https://issues.apache.org/jira/browse/YARN-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohui Xin updated YARN-9045: -- Description: There are two modes in yarn container cpu limitation. The default is Shared mode which guarantees the requested resource and can also use other spare CPU if avaliable. Another one is Strict mode, container's resource is limited to allocated resource strictly. If we want to limit the resource usage of the container and can use a certain percentage of spare CPU if available, the above two modes cannot satisfy our demand. We can define a relax coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) to relax the restrictions on resources in strict mode. > Container's cpu usage limit should be soft with a relaxation factor > --- > > Key: YARN-9045 > URL: https://issues.apache.org/jira/browse/YARN-9045 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.6.0, 3.1.1 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: image-2018-11-21-23-18-41-366.png > > > There are two modes in yarn container cpu limitation. The default is Shared > mode which guarantees the requested resource and can also use other spare CPU > if avaliable. Another one is Strict mode, container's resource is limited to > allocated resource strictly. > If we want to limit the resource usage of the container and can use a certain > percentage of spare CPU if available, the above two modes cannot satisfy our > demand. We can define a relax > coefficient(yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage.ratio) > to relax the restrictions on resources in strict mode. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9034) ApplicationCLI should have option to take clusterId
[
https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694800#comment-16694800
]
Hadoop QA commented on YARN-9034:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 7 new +
454 unchanged - 0 fixed = 461 total (was 454) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 54s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9034 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949016/YARN-9034.03.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 01e9b85b4df9 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f63e4e4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22665/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/22665/artifact/out/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22665/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22665/testReport/ |
| Max.
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8986: -- Attachment: YARN-8986.007.patch > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8986: -- Attachment: (was: YARN-8986.007.patch) > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694740#comment-16694740
]
Charo Zhang edited comment on YARN-8986 at 11/21/18 2:36 PM:
-
[~eyang] what's version of Docker in Jenkins node. Does it support "docker
network inspect bridge --format='{{.Driver}}'"? unit test still failed after
default network specified.
the first test case exit code is -1, which means docker network inspect
command execute failed.
was (Author: charo zhang):
[~eyang] what's version of Docker in Jenkins node. Does it support "docker
network inspect bridge --format='{{.Driver}}'"? unit test still failed after
default network specified.
-1 exit cod mean docker network inspect command execute failed.
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694740#comment-16694740
]
Charo Zhang edited comment on YARN-8986 at 11/21/18 2:35 PM:
-
[~eyang] what's version of Docker in Jenkins node. Does it support "docker
network inspect bridge --format='{{.Driver}}'"? unit test still failed after
default network specified.
-1 exit cod mean docker network inspect command execute failed.
was (Author: charo zhang):
[~eyang] what's version of Docker in Jenkins node. Does it support "docker
network inspect bridge --format='{{.Driver}}'"? unit test still failed after
default network specified.
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694779#comment-16694779
]
Zhankun Tang commented on YARN-8714:
[~liuxun323] , One question from me. Do you need to support bind-mount
localized file/dir into a container with RO permission?
Because YARN mounts the application/container dir with hard-coded RW permission
so that all locolized file/dir is writable to Docker container process. In this
case, RO permission for localized file/dir seems impossible until we improve
YARN's current implementation on this.
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9045) Container's cpu usage limit should be soft with a relaxation factor
Zhaohui Xin created YARN-9045: - Summary: Container's cpu usage limit should be soft with a relaxation factor Key: YARN-9045 URL: https://issues.apache.org/jira/browse/YARN-9045 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 3.1.1, 2.6.0 Reporter: Zhaohui Xin Assignee: Zhaohui Xin -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694740#comment-16694740
]
Charo Zhang commented on YARN-8986:
---
[~eyang] what's version of Docker in Jenkins node. Does it support "docker
network inspect bridge --format='{{.Driver}}'"? unit test still failed after
default network specified.
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8867) Retrieve the status of resource localization
[
https://issues.apache.org/jira/browse/YARN-8867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694705#comment-16694705
]
Hadoop QA commented on YARN-8867:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 6m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 8s{color} | {color:orange} root: The patch generated 7 new + 517 unchanged
- 1 fixed = 524 total (was 518) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
57s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
46s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
26s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 19s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 25m
42s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 43s{color}
| {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694692#comment-16694692
]
Hadoop QA commented on YARN-8986:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 57s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | TEST-cetest |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8986 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949029/YARN-8986.007.patch |
| Optional Tests | dupname asflicense
[jira] [Comment Edited] (YARN-9039) App ACLs are not validated when serving logs from Logs CLI/Yarn UI2
[
https://issues.apache.org/jira/browse/YARN-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694652#comment-16694652
]
Bibin A Chundatt edited comment on YARN-9039 at 11/21/18 12:28 PM:
---
[~suma.shivaprasad]
Process started user is supposed to be admin user, IIUC thts the reason we add
{{71adminAcl.addUser(owner.getShortUserName());}}
This could break that rt ?? IIUC ApplicationACLManager should be initialized by
the abstract services only rt ??
About S3 i havn't analyzed much that part.
{quote}
while testing with S3, unless there are object level ACLs, any user can access
any other user's logs if this change is not done.
{quote}
Even if we add the acl check we are restricting only LogAggregationFileReader.
Users are always allowed to read data direclty from object store and read its
using custom readers rt ? if bucket access is available ..
was (Author: bibinchundatt):
[~suma.shivaprasad]
Process started user is supposed to be admin user, IIUC thts the reason we add
{{71adminAcl.addUser(owner.getShortUserName());}}
This could break that rt ?? IIUC ApplicationACLManager should by initialized by
the abstract services only rt ??
About S3 i havn't analyzed much that part.
{quote}
while testing with S3, unless there are object level ACLs, any user can access
any other user's logs if this change is not done.
{quote}
Even if we add the acl check we are restricting only LogAggregationFileReader.
Users are always allowed to read data direclty from object store and read its
using custom readers rt ? if bucket access is available ..
> App ACLs are not validated when serving logs from Logs CLI/Yarn UI2
> ---
>
> Key: YARN-9039
> URL: https://issues.apache.org/jira/browse/YARN-9039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Critical
> Attachments: YARN-9039.1.patch, YARN-9039.2.patch
>
>
> App Acls are not being validated when serving logs through YARN CLI.
> This also applies while serving logs through YARN UIV2 through ATSV2 Log
> Webservice
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9039) App ACLs are not validated when serving logs from Logs CLI/Yarn UI2
[ https://issues.apache.org/jira/browse/YARN-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694653#comment-16694653 ] Bibin A Chundatt commented on YARN-9039: If my first point is valid we might be having multiple wrong usages. > App ACLs are not validated when serving logs from Logs CLI/Yarn UI2 > --- > > Key: YARN-9039 > URL: https://issues.apache.org/jira/browse/YARN-9039 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-9039.1.patch, YARN-9039.2.patch > > > App Acls are not being validated when serving logs through YARN CLI. > This also applies while serving logs through YARN UIV2 through ATSV2 Log > Webservice -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9039) App ACLs are not validated when serving logs from Logs CLI/Yarn UI2
[
https://issues.apache.org/jira/browse/YARN-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694652#comment-16694652
]
Bibin A Chundatt commented on YARN-9039:
[~suma.shivaprasad]
Process started user is supposed to be admin user, IIUC thts the reason we add
{{71adminAcl.addUser(owner.getShortUserName());}}
This could break that rt ?? IIUC ApplicationACLManager should by initialized by
the abstract services only rt ??
About S3 i havn't analyzed much that part.
{quote}
while testing with S3, unless there are object level ACLs, any user can access
any other user's logs if this change is not done.
{quote}
Even if we add the acl check we are restricting only LogAggregationFileReader.
Users are always allowed to read data direclty from object store and read its
using custom readers rt ? if bucket access is available ..
> App ACLs are not validated when serving logs from Logs CLI/Yarn UI2
> ---
>
> Key: YARN-9039
> URL: https://issues.apache.org/jira/browse/YARN-9039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Critical
> Attachments: YARN-9039.1.patch, YARN-9039.2.patch
>
>
> App Acls are not being validated when serving logs through YARN CLI.
> This also applies while serving logs through YARN UIV2 through ATSV2 Log
> Webservice
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9044) LogsCLI should contact ATSv2 for "-am" option
Rohith Sharma K S created YARN-9044: --- Summary: LogsCLI should contact ATSv2 for "-am" option Key: YARN-9044 URL: https://issues.apache.org/jira/browse/YARN-9044 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S *yarn logs -applicationId appId -am 1* contact ATS1.5 even though it is not configured. Rather LogsCLI should contact ATSv2 for AM container info. Alternative to above one can use *yarn logs -containerId * to fetch logs. But -am option should also work along with ATSv2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694637#comment-16694637
]
Hadoop QA commented on YARN-8964:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
30m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8964 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949028/YARN-8964.004.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux e3bad7eae3ed 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f63e4e4 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 443 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22663/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
> --
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch,
> YARN-8964.003.patch, YARN-8964.004.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9040) LevelDBCacheTimelineStore in ATS 1.5 leaks native memory
[
https://issues.apache.org/jira/browse/YARN-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694610#comment-16694610
]
Hadoop QA commented on YARN-9040:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
18s{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
19s{color} | {color:green} hadoop-yarn-server-timeline-pluginstorage in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9040 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949023/YARN-9040.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux aa1c9ecb937f 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f63e4e4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
|
[jira] [Commented] (YARN-8984) AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty
[ https://issues.apache.org/jira/browse/YARN-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694602#comment-16694602 ] Yang Wang commented on YARN-8984: - Hi, [~kkaranasos], [~botong], [~asuresh] Could you please take a look about this patch. It is very important when use SchedulingRequest instead of ResourceRequest. > AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty > -- > > Key: YARN-8984 > URL: https://issues.apache.org/jira/browse/YARN-8984 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yang Wang >Assignee: Yang Wang >Priority: Critical > Attachments: YARN-8984-001.patch, YARN-8984-002.patch, > YARN-8984-003.patch, YARN-8984-004.patch, YARN-8984-005.patch > > > In AMRMClient, outstandingSchedRequests should be removed or decreased when > container allocated. However, it could not work when allocation tag is null > or empty. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694595#comment-16694595
]
Hadoop QA commented on YARN-8964:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
30m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8964 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949026/YARN-8964.003.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux 49c658e818ee 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f63e4e4 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 446 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22662/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
> --
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch,
> YARN-8964.003.patch, YARN-8964.004.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694587#comment-16694587
]
Sushil Ks commented on YARN-9016:
-
Hi [~vrushalic] and [~rohithsharma], based on our discussion in the weekly
ATSv2 call, I have updated the patch. Kindly review it.
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akhil PB updated YARN-8964:
---
Attachment: (was: YARN-8964.004.patch)
> UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
> --
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch,
> YARN-8964.003.patch, YARN-8964.004.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8986: -- Attachment: YARN-8986.007.patch > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charo Zhang updated YARN-8986: -- Attachment: (was: YARN-8986.007.patch) > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akhil PB updated YARN-8964:
---
Attachment: YARN-8964.004.patch
> UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
> --
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch,
> YARN-8964.003.patch, YARN-8964.004.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akhil PB updated YARN-8964:
---
Attachment: YARN-8964.004.patch
> UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
> --
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch,
> YARN-8964.003.patch, YARN-8964.004.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694570#comment-16694570
]
Hadoop QA commented on YARN-9016:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}137m 44s{color}
| {color:red} hadoop-yarn-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-server-timelineservice-documentstore in
the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} |
[jira] [Commented] (YARN-9034) ApplicationCLI should have option to take clusterId
[
https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694565#comment-16694565
]
Hadoop QA commented on YARN-9034:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 7 new +
454 unchanged - 0 fixed = 461 total (was 454) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 20s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9034 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949016/YARN-9034.03.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6ebaa684eeba 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c8b3dfa |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22659/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/22659/artifact/out/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22659/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22659/testReport/ |
| Max.
[jira] [Commented] (YARN-9041) Optimize FSPreemptionThread#identifyContainersToPreempt method
[
https://issues.apache.org/jira/browse/YARN-9041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694564#comment-16694564
]
Hadoop QA commented on YARN-9041:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 11s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9041 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949006/YARN-9041.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e8d7f359298c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c8b3dfa |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22658/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22658/testReport/ |
| Max. process+thread count | 968 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-8936) Bump up Atsv2 hbase versions
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694553#comment-16694553 ] Hudson commented on YARN-8936: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15482 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15482/]) YARN-8936. Bump up Atsv2 hbase versions. Contributed by Vrushali C. (rohithsharmaks: rev f63e4e4f2eee6b75c3a6c3fb5926d44c7e846145) * (edit) hadoop-project/pom.xml > Bump up Atsv2 hbase versions > > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8936-yarn-7055.0001.patch, YARN-8936.0001.patch, > YARN-8936.0002.patch, YARN-8936.0003.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akhil PB updated YARN-8964:
---
Attachment: YARN-8964.003.patch
> UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
> --
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch,
> YARN-8964.003.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694524#comment-16694524
]
Hadoop QA commented on YARN-8964:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8964 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949014/YARN-8964.002.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux ae6808c69892 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c8b3dfa |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 472 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22660/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
> --
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch, YARN-8964.002.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9043) Inter-queue preemption sometimes starves an underserved queue when using DominantResourceCalculator
[
https://issues.apache.org/jira/browse/YARN-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-9043:
---
Description:
To reproduce this problem in UT, we can setup a cluster with resource <40,18>
and create 3 queues and apps:
* queue a: guaranteed=<10,10>, used=<6,10> by app1
* queue b: guaranteed=<20,6>, used=<20,8> by app2
* queue c: guaranteed=<10,2>, used=<0,0>, pending=<1,1>
Queue c is an underserved queue, queue b overuses 2 cpu resource, we expect
app2 in queue b can be preempted but nothing happens.
This problem is related to Resources#greaterThan/lessThan, comparation between
two resources is based on the resource/cluster-resource ratio inside
DominantResourceCalculator#compare, in this way, the low weight resource may be
ignored, for the scenario in UT, take comparation between ideal assgined
resource and used resource:
* cluster resource is <40,18>
* ideal assigned resource of queue b is <20,6>, ideal-assigned-resource /
cluster-resource = <20, 6> / <40, 18> = max(20/40, 6/18) = 0.5
* used resource of queue b is <20, 8>, used-resource / cluster-resource = <20,
8> / <40, 18> = max(20/40, 8/18) = 0.5
The results of {{Resources.greaterThan(rc, clusterResource, used,
idealAssigned)}} will be false instead of true, and there are some other
similar places have the same problem, so that preemption can't happen in
current logic.
To solve this problem, I propose to add
ResourceCalculator#isAnyMajorResourceGreaterThan method, inside
DominantResourceCalculator implements, it will compare every resource type
between two resources and return true if any major resource types of left
resource is greater than that of right resource, then replace
Resources#greaterThan with it in some places of inter-queue preemption with
this problem.
Other places called Resources#greaterThan and other comparations in scheduler
and other preemption processes may encounter the same problem, perhaps need to
check through all resource comparation places in YARN, we need further discuss
about this.
was:
To reproduce this problem in UT, we can setup a cluster with resource <40,18>
and create 3 queues and apps:
* queue a: guaranteed=<10,10>, used=<6,10> by app1
* queue b: guaranteed=<20,6>, used=<20,8> by app2
* queue c: guaranteed=<10,2>, used=<0,0>, pending=<1,1>
Queue c is an underserved queue, queue b overuses 2 cpu resource, we expect
app2 in queue b can be preempted but nothing happens.
This problem is related to Resources#greaterThan/lessThan, comparation between
two resources is based on the resource/cluster-resource ratio inside
DominantResourceCalculator#compare, in this way, the low weight resource may be
ignored, for the scenario in UT, take comparation between ideal assgined
resource and used resource:
* cluster resource is <40,18>
* ideal assigned resource of queue b is <20,6>, ideal-assigned-resource /
cluster-resource = <20, 6> / <40, 18> = max(20/40, 6/18) = 0.5
* used resource of queue b is <20, 8>, used-resource / cluster-resource = <20,
8> / <40, 18> = max(20/40, 8/18) = 0.5
The results of {{Resources.greaterThan(rc, clusterResource, used,
idealAssigned)}} will be false instead of true, and there are some other
similar places have the same problem, so that preemption can't happen in
current logic.
To solve this problem, I propose to add
ResourceCalculator#isAnyMajorResourceGreaterThan method, inside
DominantResourceCalculator implements, it will compare every resource type
between two resources and return true if any major resource types of left
resource is greater than that of right resource, then replace
Resources#greaterThan with it in some places of inter-queue preemption with
this problem.
> Inter-queue preemption sometimes starves an underserved queue when using
> DominantResourceCalculator
> ---
>
> Key: YARN-9043
> URL: https://issues.apache.org/jira/browse/YARN-9043
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
>
> To reproduce this problem in UT, we can setup a cluster with resource <40,18>
> and create 3 queues and apps:
> * queue a: guaranteed=<10,10>, used=<6,10> by app1
> * queue b: guaranteed=<20,6>, used=<20,8> by app2
> * queue c: guaranteed=<10,2>, used=<0,0>, pending=<1,1>
> Queue c is an underserved queue, queue b overuses 2 cpu resource, we expect
> app2 in queue b can be preempted but nothing happens.
> This problem is related to Resources#greaterThan/lessThan, comparation
> between two resources is based on the resource/cluster-resource ratio inside
> DominantResourceCalculator#compare, in this way, the low weight resource may
> be
[jira] [Updated] (YARN-9040) LevelDBCacheTimelineStore in ATS 1.5 leaks native memory
[
https://issues.apache.org/jira/browse/YARN-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tarun Parimi updated YARN-9040:
---
Attachment: YARN-9040.002.patch
> LevelDBCacheTimelineStore in ATS 1.5 leaks native memory
>
>
> Key: YARN-9040
> URL: https://issues.apache.org/jira/browse/YARN-9040
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.8.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: YARN-9040.001.patch, YARN-9040.002.patch
>
>
> When LevelDBCacheTimelineStore from YARN-4219 is used as ATS 1.5 entity
> caching storage, we observe memory leak due to leveldb files even after the
> fix of YARN-5368 .
> Top output shows 0.024TB (25GB) RES, even though heap size is only 8GB.
>
>
> {code:java}
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 25519 yarn 20 0 33.024g 0.024t 41468 S 6.2 26.0 21:07.39
> /usr/java/default/bin/java -Dproc_timelineserver -Xmx8192m
> {code}
>
> Lsof shows a lot of open timeline-cache.ldb files which are referenced by
> ATS, even though are deleted (DEL), since they are not present when listing
> them .
>
> {code:java}
> java 25519 yarn DEL REG 253,28 9438452
> /var/yarn/timeline/timelineEntityGroupId_1542280269959_55569_dag_1542280269959_55569_2-timeline-cache.ldb/07.sst
> java 25519 yarn DEL REG 253,28 9438438
> /var/yarn/timeline/timelineEntityGroupId_1542280269959_55569_dag_1542280269959_55569_2-timeline-cache.ldb/07.sst
> java 25519 yarn DEL REG 253,28 9438437
> /var/yarn/timeline/timelineEntityGroupId_1542280269959_55569_dag_1542280269959_55569_2-timeline-cache.ldb/05.sst
> {code}
>
> Looks like LevelDBCacheTimelineStore is not closing these files as the
> LevelDB DBIterator is not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9043) Inter-queue preemption sometimes starves an underserved queue when using DominantResourceCalculator
Tao Yang created YARN-9043:
--
Summary: Inter-queue preemption sometimes starves an underserved
queue when using DominantResourceCalculator
Key: YARN-9043
URL: https://issues.apache.org/jira/browse/YARN-9043
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 3.3.0
Reporter: Tao Yang
Assignee: Tao Yang
To reproduce this problem in UT, we can setup a cluster with resource <40,18>
and create 3 queues and apps:
* queue a: guaranteed=<10,10>, used=<6,10> by app1
* queue b: guaranteed=<20,6>, used=<20,8> by app2
* queue c: guaranteed=<10,2>, used=<0,0>, pending=<1,1>
Queue c is an underserved queue, queue b overuses 2 cpu resource, we expect
app2 in queue b can be preempted but nothing happens.
This problem is related to Resources#greaterThan/lessThan, comparation between
two resources is based on the resource/cluster-resource ratio inside
DominantResourceCalculator#compare, in this way, the low weight resource may be
ignored, for the scenario in UT, take comparation between ideal assgined
resource and used resource:
* cluster resource is <40,18>
* ideal assigned resource of queue b is <20,6>, ideal-assigned-resource /
cluster-resource = <20, 6> / <40, 18> = max(20/40, 6/18) = 0.5
* used resource of queue b is <20, 8>, used-resource / cluster-resource = <20,
8> / <40, 18> = max(20/40, 8/18) = 0.5
The results of {{Resources.greaterThan(rc, clusterResource, used,
idealAssigned)}} will be false instead of true, and there are some other
similar places have the same problem, so that preemption can't happen in
current logic.
To solve this problem, I propose to add
ResourceCalculator#isAnyMajorResourceGreaterThan method, inside
DominantResourceCalculator implements, it will compare every resource type
between two resources and return true if any major resource types of left
resource is greater than that of right resource, then replace
Resources#greaterThan with it in some places of inter-queue preemption with
this problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9042) Javadoc error in deviceplugin package
Rohith Sharma K S created YARN-9042:
---
Summary: Javadoc error in deviceplugin package
Key: YARN-9042
URL: https://issues.apache.org/jira/browse/YARN-9042
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Many java doc errors are in deviceplugin
{noformat}
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:29:
error: bad HTML entity
[ERROR] * This is a spec used to prepare & run container.
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:35:
error: bad HTML entity
[ERROR] * The volume & device mounts describes key isolation requirements
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: domain
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: bus
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: slot
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: func
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8882) Phase 1 - Add a shared device mapping manager for device plugin to use
[ https://issues.apache.org/jira/browse/YARN-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694457#comment-16694457 ] Zhankun Tang commented on YARN-8882: [~goyal.sunil] , [~leftnoteasy], Could you please help to review? > Phase 1 - Add a shared device mapping manager for device plugin to use > -- > > Key: YARN-8882 > URL: https://issues.apache.org/jira/browse/YARN-8882 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-8882-trunk.001.patch, YARN-8882-trunk.002.patch, > YARN-8882-trunk.003.patch, YARN-8882-trunk.004.patch, > YARN-8882-trunk.005.patch, YARN-8882-trunk.006.patch, > YARN-8882-trunk.007.patch > > > Since a few devices uses FIFO policy to assign devices to the container, we > use a shared device manager to handle all types of devices. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
