[jira] [Commented] (YARN-8638) Allow linux container runtimes to be pluggable
[
https://issues.apache.org/jira/browse/YARN-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575768#comment-16575768
]
genericqa commented on YARN-8638:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
4s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8638 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935071/YARN-8638.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2c5936d00b5a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 08d5060 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| javadoc |
[jira] [Commented] (YARN-8575) CapacityScheduler should check node state before committing reserve/allocate proposals
[
https://issues.apache.org/jira/browse/YARN-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575756#comment-16575756
]
genericqa commented on YARN-8575:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m
28s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8575 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935069/YARN-8575.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 52cee73e2aa9 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 08d5060 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21556/testReport/ |
| Max. process+thread count | 951 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21556/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> CapacityScheduler should check node state before
[jira] [Updated] (YARN-8638) Allow linux container runtimes to be pluggable
[
https://issues.apache.org/jira/browse/YARN-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Craig Condit updated YARN-8638:
---
Attachment: (was: YARN-8638.002.patch)
> Allow linux container runtimes to be pluggable
> --
>
> Key: YARN-8638
> URL: https://issues.apache.org/jira/browse/YARN-8638
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 3.2.0
>Reporter: Craig Condit
>Assignee: Craig Condit
>Priority: Minor
> Attachments: YARN-8638.001.patch, YARN-8638.002.patch
>
>
> YARN currently supports three different Linux container runtimes (default,
> docker, and javasandbox). However, it would be relatively straightforward to
> support arbitrary runtime implementations. This would enable easier
> experimentation with new and emerging runtime technologies (runc, containerd,
> etc.) without requiring a rebuild and redeployment of Hadoop.
> This could be accomplished via a simple configuration change:
> {code:xml}
>
> yarn.nodemanager.runtime.linux.allowed-runtimes
> default,docker,experimental
>
>
>
> yarn.nodemanager.runtime.linux.experimental.class
> com.somecompany.yarn.runtime.ExperimentalLinuxContainerRuntime
> {code}
>
> In this example, {{yarn.nodemanager.runtime.linux.allowed-runtimes}} would
> now allow arbitrary values. Additionally,
> {{yarn.nodemanager.runtime.linux.\{RUNTIME_KEY}.class}} would indicate the
> {{LinuxContainerRuntime}} implementation to instantiate. A no-argument
> constructor should be sufficient, as {{LinuxContainerRuntime}} already
> provides an {{initialize()}} method.
> {{DockerLinuxContainerRuntime.isDockerContainerRequested(Map
> env)}} and {{JavaSandboxLinuxContainerRuntime.isSandboxContainerRequested()}}
> could be generalized to {{isRuntimeRequested(Map env)}} and
> added to the {{LinuxContainerRuntime}} interface. This would allow
> {{DelegatingLinuxContainerRuntime}} to select an appropriate runtime based on
> whether that runtime claimed ownership of the current container execution.
> For backwards compatibility, the existing values (default,docker,javasandbox)
> would continue to be supported as-is. Under the current logic, the evaluation
> order is javasandbox, docker, default (with default being chosen if no other
> candidates are available). Under the new evaluation logic, pluggable runtimes
> would be evaluated after docker and before default, in the order in which
> they are defined in the allowed-runtimes list. This will change no behavior
> on current clusters (as there would be no pluggable runtimes defined), and
> preserves behavior with respect to ordering of existing runtimes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8638) Allow linux container runtimes to be pluggable
[
https://issues.apache.org/jira/browse/YARN-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575734#comment-16575734
]
genericqa commented on YARN-8638:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
37s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
34s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
52s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8638 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935064/YARN-8638.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0c0adf286c3d 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 08d5060 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| javadoc |
[jira] [Created] (YARN-8645) Yarn NM fail to start when remount cpu control group
Jiandan Yang created YARN-8645:
---
Summary: Yarn NM fail to start when remount cpu control group
Key: YARN-8645
URL: https://issues.apache.org/jira/browse/YARN-8645
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: Jiandan Yang
NM failed to start when we update Yarn to latest version. NM logs are as
follows:
{code:java}
2018-08-08 16:07:01,244 INFO [main]
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl:
Mounting controller cpu at /sys/fs/cgroup/cpu
2018-08-08 16:07:01,246 WARN [main]
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
Shell execution returned exit code: 32. Privileged Execution Operation Stderr:
Feature disabled: mount cgroup
Stdout:
Full command array for failed execution:
[/home/hadoop/hadoop_hbase/hadoop-current/bin/container-executor,
--mount-cgroups, hadoop-yarn, cpu,cpuset,cpuacct=/sys/fs/cgroup/cpu]
2018-08-08 16:07:01,247 ERROR [main]
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl:
Failed to mount controller: cpu
2018-08-08 16:07:01,247 ERROR [main]
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Failed to
bootstrap configured resource subsystems!
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerException:
Failed to mount controller: cpu
{code}
The cause of error is that 351cf87c92872d90f62c476f85ae4d02e485769c disable
mounting cgroups by default in container-executor, which make
container-executor return non-zero when executing mount-cgroups
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8575) CapacityScheduler should check node state before committing reserve/allocate proposals
[
https://issues.apache.org/jira/browse/YARN-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575699#comment-16575699
]
Tao Yang commented on YARN-8575:
Thanks [~cheersyang] for the review. Attached v2 patch for the latest code.
> CapacityScheduler should check node state before committing reserve/allocate
> proposals
> --
>
> Key: YARN-8575
> URL: https://issues.apache.org/jira/browse/YARN-8575
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-8575.001.patch, YARN-8575.002.patch
>
>
> Recently we found a new error as follows:
> {noformat}
> ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp:
> node to unreserve doesn't exist, nodeid: host1:45454
> {noformat}
> Reproduce this problem:
> (1) Create a reserve proposal for app1 on node1
> (2) node1 is successfully decommissioned and removed from node tracker
> (3) Try to commit this outdated reserve proposal, it will be accepted and
> applied.
> This error may be occurred after decommissioning some NMs. The application
> who print the error log will always have a reserved container on non-exist
> (decommissioned) NM and the pending request will never be satisfied.
> To solve this problem, scheduler should check node state in
> FiCaSchedulerApp#accept to avoid committing outdated proposals on unusable
> nodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8575) CapacityScheduler should check node state before committing reserve/allocate proposals
[
https://issues.apache.org/jira/browse/YARN-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-8575:
---
Attachment: YARN-8575.002.patch
> CapacityScheduler should check node state before committing reserve/allocate
> proposals
> --
>
> Key: YARN-8575
> URL: https://issues.apache.org/jira/browse/YARN-8575
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-8575.001.patch, YARN-8575.002.patch
>
>
> Recently we found a new error as follows:
> {noformat}
> ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp:
> node to unreserve doesn't exist, nodeid: host1:45454
> {noformat}
> Reproduce this problem:
> (1) Create a reserve proposal for app1 on node1
> (2) node1 is successfully decommissioned and removed from node tracker
> (3) Try to commit this outdated reserve proposal, it will be accepted and
> applied.
> This error may be occurred after decommissioning some NMs. The application
> who print the error log will always have a reserved container on non-exist
> (decommissioned) NM and the pending request will never be satisfied.
> To solve this problem, scheduler should check node state in
> FiCaSchedulerApp#accept to avoid committing outdated proposals on unusable
> nodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8575) CapacityScheduler should check node state before committing reserve/allocate proposals
[
https://issues.apache.org/jira/browse/YARN-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575678#comment-16575678
]
Weiwei Yang commented on YARN-8575:
---
Thanks [~Tao Yang] for the patch, the patch looks good overall. But it doesn't
apply to latest patch, could you please rebase it?
> CapacityScheduler should check node state before committing reserve/allocate
> proposals
> --
>
> Key: YARN-8575
> URL: https://issues.apache.org/jira/browse/YARN-8575
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-8575.001.patch
>
>
> Recently we found a new error as follows:
> {noformat}
> ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp:
> node to unreserve doesn't exist, nodeid: host1:45454
> {noformat}
> Reproduce this problem:
> (1) Create a reserve proposal for app1 on node1
> (2) node1 is successfully decommissioned and removed from node tracker
> (3) Try to commit this outdated reserve proposal, it will be accepted and
> applied.
> This error may be occurred after decommissioning some NMs. The application
> who print the error log will always have a reserved container on non-exist
> (decommissioned) NM and the pending request will never be satisfied.
> To solve this problem, scheduler should check node state in
> FiCaSchedulerApp#accept to avoid committing outdated proposals on unusable
> nodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8638) Allow linux container runtimes to be pluggable
[
https://issues.apache.org/jira/browse/YARN-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Craig Condit updated YARN-8638:
---
Attachment: YARN-8638.001.patch
> Allow linux container runtimes to be pluggable
> --
>
> Key: YARN-8638
> URL: https://issues.apache.org/jira/browse/YARN-8638
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 3.2.0
>Reporter: Craig Condit
>Assignee: Craig Condit
>Priority: Minor
> Attachments: YARN-8638.001.patch, YARN-8638.002.patch
>
>
> YARN currently supports three different Linux container runtimes (default,
> docker, and javasandbox). However, it would be relatively straightforward to
> support arbitrary runtime implementations. This would enable easier
> experimentation with new and emerging runtime technologies (runc, containerd,
> etc.) without requiring a rebuild and redeployment of Hadoop.
> This could be accomplished via a simple configuration change:
> {code:xml}
>
> yarn.nodemanager.runtime.linux.allowed-runtimes
> default,docker,experimental
>
>
>
> yarn.nodemanager.runtime.linux.experimental.class
> com.somecompany.yarn.runtime.ExperimentalLinuxContainerRuntime
> {code}
>
> In this example, {{yarn.nodemanager.runtime.linux.allowed-runtimes}} would
> now allow arbitrary values. Additionally,
> {{yarn.nodemanager.runtime.linux.\{RUNTIME_KEY}.class}} would indicate the
> {{LinuxContainerRuntime}} implementation to instantiate. A no-argument
> constructor should be sufficient, as {{LinuxContainerRuntime}} already
> provides an {{initialize()}} method.
> {{DockerLinuxContainerRuntime.isDockerContainerRequested(Map
> env)}} and {{JavaSandboxLinuxContainerRuntime.isSandboxContainerRequested()}}
> could be generalized to {{isRuntimeRequested(Map env)}} and
> added to the {{LinuxContainerRuntime}} interface. This would allow
> {{DelegatingLinuxContainerRuntime}} to select an appropriate runtime based on
> whether that runtime claimed ownership of the current container execution.
> For backwards compatibility, the existing values (default,docker,javasandbox)
> would continue to be supported as-is. Under the current logic, the evaluation
> order is javasandbox, docker, default (with default being chosen if no other
> candidates are available). Under the new evaluation logic, pluggable runtimes
> would be evaluated after docker and before default, in the order in which
> they are defined in the allowed-runtimes list. This will change no behavior
> on current clusters (as there would be no pluggable runtimes defined), and
> preserves behavior with respect to ordering of existing runtimes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8638) Allow linux container runtimes to be pluggable
[jira] [Commented] (YARN-8639) Sort queue and apps in fair schduler using a separate thread
[ https://issues.apache.org/jira/browse/YARN-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575646#comment-16575646 ] wan kun commented on YARN-8639: --- [~yufeigu] I'm sorry, I find the sort operation costing too much time in my product environment. But our hadoop version is too old. The newly code change to use TreeSet instead of Collections.sort() method,and only deal with the applications which have resource request. The sort operation will be more faster. But this is still a possible problem in the future. Now I don't know how many applications will be considered as too many . Maybe this issue can be closed. > Sort queue and apps in fair schduler using a separate thread > > > Key: YARN-8639 > URL: https://issues.apache.org/jira/browse/YARN-8639 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: wan kun >Priority: Minor > > If fair scheduler have many queue and each queue have many active > applications . > For each assignContainer function, we need to sort all the queue, and all the > applications in each queue. For a large system,this may be cost too much > time. So we can sort the queue and applications using a separate thread > asynchronous. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8521) NPE in AllocationTagsManager when a container is removed more than once
[
https://issues.apache.org/jira/browse/YARN-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575628#comment-16575628
]
Hudson commented on YARN-8521:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14744 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14744/])
YARN-8521. NPE in AllocationTagsManager when a container is removed more (wwei:
rev 08d5060605af81a3d6048044176dc656c0dad56c)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/TestPlacementConstraintsUtil.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/TestAllocationTagsManager.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/constraint/AllocationTagsManager.java
> NPE in AllocationTagsManager when a container is removed more than once
> ---
>
> Key: YARN-8521
> URL: https://issues.apache.org/jira/browse/YARN-8521
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8521.001.patch, YARN-8521.002.patch,
> YARN-8521.003.patch
>
>
> We've seen sometimes there is NPE in AllocationTagsManager
> {code:java}
> private void removeTagFromInnerMap(Map innerMap, String tag) {
> Long count = innerMap.get(tag);
> if (count > 1) { // NPE!!
> ...
> {code}
> it seems {{AllocationTagsManager#removeContainer}} somehow gets called more
> than once for a same container.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8521) NPE in AllocationTagsManager when a container is removed more than once
[
https://issues.apache.org/jira/browse/YARN-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575620#comment-16575620
]
Weiwei Yang commented on YARN-8521:
---
Just committed this to trunk and branch-3.1, thanks [~suma.shivaprasad]!
> NPE in AllocationTagsManager when a container is removed more than once
> ---
>
> Key: YARN-8521
> URL: https://issues.apache.org/jira/browse/YARN-8521
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
> Attachments: YARN-8521.001.patch, YARN-8521.002.patch,
> YARN-8521.003.patch
>
>
> We've seen sometimes there is NPE in AllocationTagsManager
> {code:java}
> private void removeTagFromInnerMap(Map innerMap, String tag) {
> Long count = innerMap.get(tag);
> if (count > 1) { // NPE!!
> ...
> {code}
> it seems {{AllocationTagsManager#removeContainer}} somehow gets called more
> than once for a same container.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8559) Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint
[
https://issues.apache.org/jira/browse/YARN-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575607#comment-16575607
]
genericqa commented on YARN-8559:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
56s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
2s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} branch-3.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 15s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 14 unchanged - 0 fixed = 17 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 58s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:1776208 |
| JIRA Issue | YARN-8559 |
| JIRA Patch URL |
[jira] [Commented] (YARN-8521) NPE in AllocationTagsManager when a container is removed more than once
[
https://issues.apache.org/jira/browse/YARN-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575604#comment-16575604
]
Weiwei Yang commented on YARN-8521:
---
Hi [~suma.shivaprasad]
Thanks for helping review this. I don't think the UT failure was related, I've
seen this failure in other jenkins result too, looks like a flaky one. I am
going to commit this shortly.
> NPE in AllocationTagsManager when a container is removed more than once
> ---
>
> Key: YARN-8521
> URL: https://issues.apache.org/jira/browse/YARN-8521
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8521.001.patch, YARN-8521.002.patch,
> YARN-8521.003.patch
>
>
> We've seen sometimes there is NPE in AllocationTagsManager
> {code:java}
> private void removeTagFromInnerMap(Map innerMap, String tag) {
> Long count = innerMap.get(tag);
> if (count > 1) { // NPE!!
> ...
> {code}
> it seems {{AllocationTagsManager#removeContainer}} somehow gets called more
> than once for a same container.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8638) Allow linux container runtimes to be pluggable
[
https://issues.apache.org/jira/browse/YARN-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575602#comment-16575602
]
genericqa commented on YARN-8638:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
30s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
48s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
51s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
50s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8638 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935052/YARN-8638.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0eef50805ac6 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b2517dd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| javadoc |
[jira] [Commented] (YARN-8559) Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint
[ https://issues.apache.org/jira/browse/YARN-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575594#comment-16575594 ] Weiwei Yang commented on YARN-8559: --- It looks good to me, thanks [~jhung] for taking care of these! > Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint > > > Key: YARN-8559 > URL: https://issues.apache.org/jira/browse/YARN-8559 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Anna Savarin >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8559-branch-2.001.patch, > YARN-8559-branch-3.0.001.patch, YARN-8559.001.patch, YARN-8559.002.patch, > YARN-8559.003.patch, YARN-8559.004.patch > > > All Hadoop services provide a set of common endpoints (/stacks, /logLevel, > /metrics, /jmx, /conf). In the case of the Resource Manager, part of the > configuration comes from the scheduler being used. Currently, these > configuration key/values are not exposed through the /conf endpoint, thereby > revealing an incomplete configuration picture. > Make an improvement and expose the scheduling configuration info through the > RM's /conf endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6924) Metrics for Federation AMRMProxy
[ https://issues.apache.org/jira/browse/YARN-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang reassigned YARN-6924: -- Assignee: Young Chen (was: Botong Huang) > Metrics for Federation AMRMProxy > > > Key: YARN-6924 > URL: https://issues.apache.org/jira/browse/YARN-6924 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Young Chen >Priority: Major > > This JIRA proposes addition of metrics for Federation AMRMProxy -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8561) [Submarine] Add initial implementation: training job submission and job history retrieve.
[
https://issues.apache.org/jira/browse/YARN-8561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575544#comment-16575544
]
genericqa commented on YARN-8561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 5 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 29m 5s{color}
| {color:red} hadoop-yarn-applications in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} hadoop-yarn-submarine in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.applications.distributedshell.TestDistributedShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA
[jira] [Commented] (YARN-4946) RM should not consider an application as COMPLETED when log aggregation is not in a terminal state
[ https://issues.apache.org/jira/browse/YARN-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575510#comment-16575510 ] Hudson commented on YARN-4946: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14742 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14742/]) YARN-4946. RM should not consider an application as COMPLETED when log (rkanter: rev b2517dd66b3c88fdd478411cf208921bd3023755) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java > RM should not consider an application as COMPLETED when log aggregation is > not in a terminal state > -- > > Key: YARN-4946 > URL: https://issues.apache.org/jira/browse/YARN-4946 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-4946.001.patch, YARN-4946.002.patch, > YARN-4946.003.patch, YARN-4946.004.patch > > > MAPREDUCE-6415 added a tool that combines the aggregated log files for each > Yarn App into a HAR file. When run, it seeds the list by looking at the > aggregated logs directory, and then filters out ineligible apps. One of the > criteria involves checking with the RM that an Application's log aggregation > status is not still running and has not failed. When the RM "forgets" about > an older completed Application (e.g. RM failover, enough time has passed, > etc), the tool won't find the Application in the RM and will just assume that > its log aggregation succeeded, even if it actually failed or is still running. > We can solve this problem by doing the following: > The RM should not consider an app to be fully completed (and thus removed > from its history) until the aggregation status has reached a terminal state > (e.g. SUCCEEDED, FAILED, TIME_OUT). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6972) Adding RM ClusterId in AppInfo
[
https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575508#comment-16575508
]
Giovanni Matteo Fumarola commented on YARN-6972:
Thanks [~tanujnay]. {{federationSubClusterId}} is wrong, it should be called
{{rmClusterId}}.
Since it is a public api change, it would be better if someone outside our
group will review it.
cc [~sunilg] , [~leftnoteasy]
> Adding RM ClusterId in AppInfo
> --
>
> Key: YARN-6972
> URL: https://issues.apache.org/jira/browse/YARN-6972
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Tanuj Nayak
>Priority: Major
> Attachments: YARN-6972.001.patch, YARN-6972.002.patch,
> YARN-6972.003.patch, YARN-6972.004.patch, YARN-6972.005.patch,
> YARN-6972.006.patch, YARN-6972.007.patch, YARN-6972.008.patch,
> YARN-6972.009.patch, YARN-6972.010.patch, YARN-6972.011.patch,
> YARN-6972.012.patch, YARN-6972.013.patch, YARN-6972.014.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8559) Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint
[ https://issues.apache.org/jira/browse/YARN-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575465#comment-16575465 ] Jonathan Hung commented on YARN-8559: - Hi [~cheersyang], thanks for the commit! I attached a branch-3.0 and branch-2 patch. Could you take a look? If it looks OK I can commit these patches. Thanks! Also reopening this ticket to trigger jenkins. > Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint > > > Key: YARN-8559 > URL: https://issues.apache.org/jira/browse/YARN-8559 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Anna Savarin >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8559-branch-2.001.patch, > YARN-8559-branch-3.0.001.patch, YARN-8559.001.patch, YARN-8559.002.patch, > YARN-8559.003.patch, YARN-8559.004.patch > > > All Hadoop services provide a set of common endpoints (/stacks, /logLevel, > /metrics, /jmx, /conf). In the case of the Resource Manager, part of the > configuration comes from the scheduler being used. Currently, these > configuration key/values are not exposed through the /conf endpoint, thereby > revealing an incomplete configuration picture. > Make an improvement and expose the scheduling configuration info through the > RM's /conf endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8559) Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint
[ https://issues.apache.org/jira/browse/YARN-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-8559: Attachment: YARN-8559-branch-3.0.001.patch YARN-8559-branch-2.001.patch > Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint > > > Key: YARN-8559 > URL: https://issues.apache.org/jira/browse/YARN-8559 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Anna Savarin >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8559-branch-2.001.patch, > YARN-8559-branch-3.0.001.patch, YARN-8559.001.patch, YARN-8559.002.patch, > YARN-8559.003.patch, YARN-8559.004.patch > > > All Hadoop services provide a set of common endpoints (/stacks, /logLevel, > /metrics, /jmx, /conf). In the case of the Resource Manager, part of the > configuration comes from the scheduler being used. Currently, these > configuration key/values are not exposed through the /conf endpoint, thereby > revealing an incomplete configuration picture. > Make an improvement and expose the scheduling configuration info through the > RM's /conf endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-8559) Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint
[ https://issues.apache.org/jira/browse/YARN-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung reopened YARN-8559: - > Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint > > > Key: YARN-8559 > URL: https://issues.apache.org/jira/browse/YARN-8559 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Anna Savarin >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8559.001.patch, YARN-8559.002.patch, > YARN-8559.003.patch, YARN-8559.004.patch > > > All Hadoop services provide a set of common endpoints (/stacks, /logLevel, > /metrics, /jmx, /conf). In the case of the Resource Manager, part of the > configuration comes from the scheduler being used. Currently, these > configuration key/values are not exposed through the /conf endpoint, thereby > revealing an incomplete configuration picture. > Make an improvement and expose the scheduling configuration info through the > RM's /conf endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8561) [Submarine] Add initial implementation: training job submission and job history retrieve.
[ https://issues.apache.org/jira/browse/YARN-8561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8561: - Attachment: YARN-8561.005.patch > [Submarine] Add initial implementation: training job submission and job > history retrieve. > - > > Key: YARN-8561 > URL: https://issues.apache.org/jira/browse/YARN-8561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-8561.001.patch, YARN-8561.002.patch, > YARN-8561.003.patch, YARN-8561.004.patch, YARN-8561.005.patch > > > Added following parts: > 1) New subcomponent of YARN, under applications/ project. > 2) Tensorflow training job submission, including training (single node and > distributed). > - Supported Docker container. > - Support GPU isolation. > - Support YARN registry DNS. > 3) Retrieve job history. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8561) [Submarine] Add initial implementation: training job submission and job history retrieve.
[ https://issues.apache.org/jira/browse/YARN-8561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575464#comment-16575464 ] Wangda Tan commented on YARN-8561: -- Thanks [~sunilg] For your addition comments: 1. I think we can register Cli plugin for this. Like a RunJobCli is registered to "job" "run", same for help msg. I would prefer to do this in a separated patch. 2. I'm not sure if we have a better solution for this, maybe a better idea is to use YAML to define types and auto generate ...Parameter class and parser. If we don't move ...Parameter to the same definition, we need to do a parsing and setting in any case. Python will be easier to do such things. I would prefer to think more about this. Given CLI parsing is a one-time effort. We need to change 2-3 classes if we want to change and parameters, I think we can live with it. 3. Now it is only for remote FS. 4. Basically, if user specified --checkpoint_dir in the commandline, they don't need to pass the same parameter to launch_command of worker/ps again. User can say: '--worker_launch_command "--ckpt-dir=%checkpoint_dir%' 5. I'm not quite sure about differences of them, could u explain? 6. Done. 7. Can we do it in a separate patch? I don't want to touch YARN stuffs within the patch? 8. It may not be the highest priority given res profile is not easy to use. We can revisit this in the future. 9. That is right, we can have separate JIRAs for this, and we may need to design the CLI options carefully to not mix YARN-specific paramters and DL-related parameters together. 10. That's right. 11. That's right, I think we can have follow up JIRA for that. 13. I think JobStatusBuilder#fromServiceState is for that, is that what you meant? > [Submarine] Add initial implementation: training job submission and job > history retrieve. > - > > Key: YARN-8561 > URL: https://issues.apache.org/jira/browse/YARN-8561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-8561.001.patch, YARN-8561.002.patch, > YARN-8561.003.patch, YARN-8561.004.patch, YARN-8561.005.patch > > > Added following parts: > 1) New subcomponent of YARN, under applications/ project. > 2) Tensorflow training job submission, including training (single node and > distributed). > - Supported Docker container. > - Support GPU isolation. > - Support YARN registry DNS. > 3) Retrieve job history. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8644) Make RMAppImpl$FinalTransition more readable + add more test coverage
[
https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575461#comment-16575461
]
genericqa commented on YARN-8644:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 10 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 35s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8644 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935029/YARN-8644.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f0b095c099d7 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8244abb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/21549/artifact/out/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/21549/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21549/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-4946) RM should not consider an application as COMPLETED when log aggregation is not in a terminal state
[ https://issues.apache.org/jira/browse/YARN-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575462#comment-16575462 ] Robert Kanter commented on YARN-4946: - +1 > RM should not consider an application as COMPLETED when log aggregation is > not in a terminal state > -- > > Key: YARN-4946 > URL: https://issues.apache.org/jira/browse/YARN-4946 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-4946.001.patch, YARN-4946.002.patch, > YARN-4946.003.patch, YARN-4946.004.patch > > > MAPREDUCE-6415 added a tool that combines the aggregated log files for each > Yarn App into a HAR file. When run, it seeds the list by looking at the > aggregated logs directory, and then filters out ineligible apps. One of the > criteria involves checking with the RM that an Application's log aggregation > status is not still running and has not failed. When the RM "forgets" about > an older completed Application (e.g. RM failover, enough time has passed, > etc), the tool won't find the Application in the RM and will just assume that > its log aggregation succeeded, even if it actually failed or is still running. > We can solve this problem by doing the following: > The RM should not consider an app to be fully completed (and thus removed > from its history) until the aggregation status has reached a terminal state > (e.g. SUCCEEDED, FAILED, TIME_OUT). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4946) RM should not consider an application as COMPLETED when log aggregation is not in a terminal state
[
https://issues.apache.org/jira/browse/YARN-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575460#comment-16575460
]
genericqa commented on YARN-4946:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m
6s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-4946 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935028/YARN-4946.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9acbdb2c72e9 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8244abb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21550/testReport/ |
| Max. process+thread count | 850 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21550/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RM should not consider an application as
[jira] [Comment Edited] (YARN-8569) Create an interface to provide cluster information to application
[
https://issues.apache.org/jira/browse/YARN-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575445#comment-16575445
]
Eric Yang edited comment on YARN-8569 at 8/9/18 9:52 PM:
-
Sysfs is a pseudo file system provided by Linux Kernel to expose system related
information to user space. YARN can mimic the same ideology to export cluster
information to container. The proposal is to expose cluster information to:
{code}
/hadoop/yarn/fs/cluster.json
{code}
This basically have the runtime information about the deployed application, and
getting updated when state changes happen. The file is replicated from YARN
service AM to host system in appcache for the application.
was (Author: eyang):
Sysfs is a pseudo file system provided by Linux Kernel to expose system related
information to user space. YARN can mimic the same ideology to export cluster
information to container. The proposal is to expose cluster information to:
{code}
/hadoop/yarn/fs/cluster.json
{code}
This basically have the runtime information about the deployed application, and
getting updated when state changes happen. The file is replicated from hdfs to
host system in appcache for the application.
> Create an interface to provide cluster information to application
> -
>
> Key: YARN-8569
> URL: https://issues.apache.org/jira/browse/YARN-8569
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Priority: Major
> Labels: Docker
>
> Some program requires container hostnames to be known for application to run.
> For example, distributed tensorflow requires launch_command that looks like:
> {code}
> # On ps0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=0
> # On ps1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=1
> # On worker0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=0
> # On worker1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=1
> {code}
> This is a bit cumbersome to orchestrate via Distributed Shell, or YARN
> services launch_command. In addition, the dynamic parameters do not work
> with YARN flex command. This is the classic pain point for application
> developer attempt to automate system environment settings as parameter to end
> user application.
> It would be great if YARN Docker integration can provide a simple option to
> expose hostnames of the yarn service via a mounted file. The file content
> gets updated when flex command is performed. This allows application
> developer to consume system environment settings via a standard interface.
> It is like /proc/devices for Linux, but for Hadoop. This may involve
> updating a file in distributed cache, and allow mounting of the file via
> container-executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8569) Create an interface to provide cluster information to application
[
https://issues.apache.org/jira/browse/YARN-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575445#comment-16575445
]
Eric Yang commented on YARN-8569:
-
Sysfs is a pseudo file system provided by Linux Kernel to expose system related
information to user space. YARN can mimic the same ideology to export cluster
information to container. The proposal is to expose cluster information to:
{code}
/hadoop/yarn/fs/cluster.json
{code}
This basically have the runtime information about the deployed application, and
getting updated when state changes happen. The file is replicated from hdfs to
host system in appcache for the application.
> Create an interface to provide cluster information to application
> -
>
> Key: YARN-8569
> URL: https://issues.apache.org/jira/browse/YARN-8569
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Priority: Major
> Labels: Docker
>
> Some program requires container hostnames to be known for application to run.
> For example, distributed tensorflow requires launch_command that looks like:
> {code}
> # On ps0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=0
> # On ps1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=1
> # On worker0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=0
> # On worker1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=1
> {code}
> This is a bit cumbersome to orchestrate via Distributed Shell, or YARN
> services launch_command. In addition, the dynamic parameters do not work
> with YARN flex command. This is the classic pain point for application
> developer attempt to automate system environment settings as parameter to end
> user application.
> It would be great if YARN Docker integration can provide a simple option to
> expose hostnames of the yarn service via a mounted file. The file content
> gets updated when flex command is performed. This allows application
> developer to consume system environment settings via a standard interface.
> It is like /proc/devices for Linux, but for Hadoop. This may involve
> updating a file in distributed cache, and allow mounting of the file via
> container-executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575421#comment-16575421
]
genericqa commented on YARN-8160:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
33s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-8160 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935036/YARN-8160.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux bee7b19bad00 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8244abb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21551/testReport/ |
| Max. process+thread count | 441 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21551/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Commented] (YARN-7974) Allow updating application tracking url after registration
[ https://issues.apache.org/jira/browse/YARN-7974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575384#comment-16575384 ] Jonathan Hung commented on YARN-7974: - Still can't seem to get jenkins build working. But I ran all the hadoop-yarn-client tests locally and they passed. Also ran the modified TestRMRestart and TestApplicationMasterService locally and they all passed. [~leftnoteasy] do you mind taking a look at branch-2.001 patch? If it looks OK to you then I can commit it. Thanks! > Allow updating application tracking url after registration > -- > > Key: YARN-7974 > URL: https://issues.apache.org/jira/browse/YARN-7974 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: YARN-7974-branch-2.001.patch, YARN-7974.001.patch, > YARN-7974.002.patch, YARN-7974.003.patch, YARN-7974.004.patch, > YARN-7974.005.patch, YARN-7974.006.patch > > > Normally an application's tracking url is set on AM registration. We have a > use case for updating the tracking url after registration (e.g. the UI is > hosted on one of the containers). > Approach is for AM to update tracking url on heartbeat to RM, and add related > API in AMRMClient. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8136) Add version attribute to site doc examples and quickstart
[ https://issues.apache.org/jira/browse/YARN-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575365#comment-16575365 ] Eric Yang commented on YARN-8136: - Thank you [~leftnoteasy]! > Add version attribute to site doc examples and quickstart > - > > Key: YARN-8136 > URL: https://issues.apache.org/jira/browse/YARN-8136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: site >Reporter: Gour Saha >Assignee: Eric Yang >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8136.001.patch > > > version attribute is missing in the following 2 site doc files - > src/site/markdown/yarn-service/Examples.md > src/site/markdown/yarn-service/QuickStart.md -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-8160:
Attachment: YARN-8160.001.patch
> Yarn Service Upgrade: Support upgrade of service that use docker containers
>
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-8160.001.patch,
> container_e02_1533231998644_0009_01_03.nm.log
>
>
> Ability to upgrade dockerized yarn native services.
> Ref: YARN-5637
> *Background*
> Container upgrade is supported by the NM via {{reInitializeContainer}} api.
> {{reInitializeContainer}} does *NOT* change the ContainerId of the upgraded
> container.
> NM performs the following steps during {{reInitializeContainer}}:
> - kills the existing process
> - cleans up the container
> - launches another container with the new {{ContainerLaunchContext}}
> NOTE: {{ContainerLaunchContext}} holds all the information that needs to
> upgrade the container.
> With {{reInitializeContainer}}, the following does *NOT* change
> - container ID. This is not created by NM. It is provided to it and here RM
> is not creating another container allocation.
> - {{localizedResources}} this stays the same if the upgrade does *NOT*
> require additional resources IIUC.
>
> The following changes with {{reInitializeContainer}}
> - the working directory of the upgraded container changes. It is *NOT* a
> relaunch.
> *Changes required in the case of docker container*
> - {{reInitializeContainer}} seems to not be working with Docker containers.
> Investigate and fix this.
> - [Future change] Add an additional api to NM to pull the images and modify
> {{reInitializeContainer}} to trigger docker container launch without pulling
> the image first which could be based on a flag.
> -- When the service upgrade is initialized, we can provide the user with
> an option to just pull the images on the NMs.
> -- When a component instance is upgrade, it calls the
> {{reInitializeContainer}} with the flag pull-image set to false, since the NM
> will have already pulled the images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-8160:
Attachment: (was: YARN-8610.001.patch)
> Yarn Service Upgrade: Support upgrade of service that use docker containers
>
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: container_e02_1533231998644_0009_01_03.nm.log
>
>
> Ability to upgrade dockerized yarn native services.
> Ref: YARN-5637
> *Background*
> Container upgrade is supported by the NM via {{reInitializeContainer}} api.
> {{reInitializeContainer}} does *NOT* change the ContainerId of the upgraded
> container.
> NM performs the following steps during {{reInitializeContainer}}:
> - kills the existing process
> - cleans up the container
> - launches another container with the new {{ContainerLaunchContext}}
> NOTE: {{ContainerLaunchContext}} holds all the information that needs to
> upgrade the container.
> With {{reInitializeContainer}}, the following does *NOT* change
> - container ID. This is not created by NM. It is provided to it and here RM
> is not creating another container allocation.
> - {{localizedResources}} this stays the same if the upgrade does *NOT*
> require additional resources IIUC.
>
> The following changes with {{reInitializeContainer}}
> - the working directory of the upgraded container changes. It is *NOT* a
> relaunch.
> *Changes required in the case of docker container*
> - {{reInitializeContainer}} seems to not be working with Docker containers.
> Investigate and fix this.
> - [Future change] Add an additional api to NM to pull the images and modify
> {{reInitializeContainer}} to trigger docker container launch without pulling
> the image first which could be based on a flag.
> -- When the service upgrade is initialized, we can provide the user with
> an option to just pull the images on the NMs.
> -- When a component instance is upgrade, it calls the
> {{reInitializeContainer}} with the flag pull-image set to false, since the NM
> will have already pulled the images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8644) Make RMAppImpl$FinalTransition more readable + add more test coverage
[ https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8644: - Attachment: YARN-8644.001.patch > Make RMAppImpl$FinalTransition more readable + add more test coverage > - > > Key: YARN-8644 > URL: https://issues.apache.org/jira/browse/YARN-8644 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-8644.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4946) RM should not consider an application as COMPLETED when log aggregation is not in a terminal state
[ https://issues.apache.org/jira/browse/YARN-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575312#comment-16575312 ] Szilard Nemeth commented on YARN-4946: -- Hi [~rkanter]! Thanks for your comments. 1. Fixed 2. Good point, somehow my IDE generates xxTest class names by default. 3. Removed reflection logic 4. Good point, removed those unnecessary log aggregation related testcases. Exactly, the logic is the same and it was mainly a readability improvement so I created a follow-up jira: YARN-8644 6. Fixed > RM should not consider an application as COMPLETED when log aggregation is > not in a terminal state > -- > > Key: YARN-4946 > URL: https://issues.apache.org/jira/browse/YARN-4946 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-4946.001.patch, YARN-4946.002.patch, > YARN-4946.003.patch, YARN-4946.004.patch > > > MAPREDUCE-6415 added a tool that combines the aggregated log files for each > Yarn App into a HAR file. When run, it seeds the list by looking at the > aggregated logs directory, and then filters out ineligible apps. One of the > criteria involves checking with the RM that an Application's log aggregation > status is not still running and has not failed. When the RM "forgets" about > an older completed Application (e.g. RM failover, enough time has passed, > etc), the tool won't find the Application in the RM and will just assume that > its log aggregation succeeded, even if it actually failed or is still running. > We can solve this problem by doing the following: > The RM should not consider an app to be fully completed (and thus removed > from its history) until the aggregation status has reached a terminal state > (e.g. SUCCEEDED, FAILED, TIME_OUT). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4946) RM should not consider an application as COMPLETED when log aggregation is not in a terminal state
[ https://issues.apache.org/jira/browse/YARN-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-4946: - Attachment: YARN-4946.004.patch > RM should not consider an application as COMPLETED when log aggregation is > not in a terminal state > -- > > Key: YARN-4946 > URL: https://issues.apache.org/jira/browse/YARN-4946 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-4946.001.patch, YARN-4946.002.patch, > YARN-4946.003.patch, YARN-4946.004.patch > > > MAPREDUCE-6415 added a tool that combines the aggregated log files for each > Yarn App into a HAR file. When run, it seeds the list by looking at the > aggregated logs directory, and then filters out ineligible apps. One of the > criteria involves checking with the RM that an Application's log aggregation > status is not still running and has not failed. When the RM "forgets" about > an older completed Application (e.g. RM failover, enough time has passed, > etc), the tool won't find the Application in the RM and will just assume that > its log aggregation succeeded, even if it actually failed or is still running. > We can solve this problem by doing the following: > The RM should not consider an app to be fully completed (and thus removed > from its history) until the aggregation status has reached a terminal state > (e.g. SUCCEEDED, FAILED, TIME_OUT). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8644) Make RMAppImpl$FinalTransition more readable + add more test coverage
Szilard Nemeth created YARN-8644: Summary: Make RMAppImpl$FinalTransition more readable + add more test coverage Key: YARN-8644 URL: https://issues.apache.org/jira/browse/YARN-8644 Project: Hadoop YARN Issue Type: Improvement Reporter: Szilard Nemeth Assignee: Szilard Nemeth -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8639) Sort queue and apps in fair schduler using a separate thread
[ https://issues.apache.org/jira/browse/YARN-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575302#comment-16575302 ] Yufei Gu commented on YARN-8639: We need to quantify a little bit before we'd make any non-trivial change. How many sub-queues/applications will be considered as too many, and cause performance issue. The result won't only justify why we need the change but also provide the guideline to tuning the queue settings. > Sort queue and apps in fair schduler using a separate thread > > > Key: YARN-8639 > URL: https://issues.apache.org/jira/browse/YARN-8639 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: wan kun >Priority: Minor > > If fair scheduler have many queue and each queue have many active > applications . > For each assignContainer function, we need to sort all the queue, and all the > applications in each queue. For a large system,this may be cost too much > time. So we can sort the queue and applications using a separate thread > asynchronous. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8588) Logging improvements for better debuggability
[ https://issues.apache.org/jira/browse/YARN-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575296#comment-16575296 ] Hudson commented on YARN-8588: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14741 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14741/]) YARN-8588. Logging improvements for better debuggability. (Suma (wangda: rev 344c335a920e6f32a35ebace0a118a9dc4a22fb7) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueManagementChange.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/queuemanagement/GuaranteedOrZeroCapacityOverTimePolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedLeafQueueConfig.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueManagementDynamicEditPolicy.java > Logging improvements for better debuggability > - > > Key: YARN-8588 > URL: https://issues.apache.org/jira/browse/YARN-8588 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-8588.1.patch, YARN-8588.2.patch > > > Capacity allocations decided in GuaranteedCapacityOvertimePolicy are > available via AutoCreatedLeafQueueConfig. However this class lacks a toString > and some other DEBUG level logs are needed for better debuggability -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8136) Add version attribute to site doc examples and quickstart
[ https://issues.apache.org/jira/browse/YARN-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575297#comment-16575297 ] Hudson commented on YARN-8136: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14741 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14741/]) YARN-8136. Add version attribute to site doc examples and quickstart. (wangda: rev 8244abb7aeb768b73682b8c9a26516a9cf06bca5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/Examples.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/QuickStart.md > Add version attribute to site doc examples and quickstart > - > > Key: YARN-8136 > URL: https://issues.apache.org/jira/browse/YARN-8136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: site >Reporter: Gour Saha >Assignee: Eric Yang >Priority: Major > Attachments: YARN-8136.001.patch > > > version attribute is missing in the following 2 site doc files - > src/site/markdown/yarn-service/Examples.md > src/site/markdown/yarn-service/QuickStart.md -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7129) Application Catalog for YARN applications
[
https://issues.apache.org/jira/browse/YARN-7129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575292#comment-16575292
]
genericqa commented on YARN-7129:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 19m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications .
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 20m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:orange}-0{color} | {color:orange} shelldocs {color} | {color:orange}
0m 13s{color} | {color:orange} The patch generated 158 new + 114 unchanged - 0
fixed = 272 total (was 114) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
14s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog
.
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-docker
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 38s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License
[jira] [Commented] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers
[
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575255#comment-16575255
]
genericqa commented on YARN-8160:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-8160 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8160 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934878/YARN-8610.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21548/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Yarn Service Upgrade: Support upgrade of service that use docker containers
>
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-8610.001.patch,
> container_e02_1533231998644_0009_01_03.nm.log
>
>
> Ability to upgrade dockerized yarn native services.
> Ref: YARN-5637
> *Background*
> Container upgrade is supported by the NM via {{reInitializeContainer}} api.
> {{reInitializeContainer}} does *NOT* change the ContainerId of the upgraded
> container.
> NM performs the following steps during {{reInitializeContainer}}:
> - kills the existing process
> - cleans up the container
> - launches another container with the new {{ContainerLaunchContext}}
> NOTE: {{ContainerLaunchContext}} holds all the information that needs to
> upgrade the container.
> With {{reInitializeContainer}}, the following does *NOT* change
> - container ID. This is not created by NM. It is provided to it and here RM
> is not creating another container allocation.
> - {{localizedResources}} this stays the same if the upgrade does *NOT*
> require additional resources IIUC.
>
> The following changes with {{reInitializeContainer}}
> - the working directory of the upgraded container changes. It is *NOT* a
> relaunch.
> *Changes required in the case of docker container*
> - {{reInitializeContainer}} seems to not be working with Docker containers.
> Investigate and fix this.
> - [Future change] Add an additional api to NM to pull the images and modify
> {{reInitializeContainer}} to trigger docker container launch without pulling
> the image first which could be based on a flag.
> -- When the service upgrade is initialized, we can provide the user with
> an option to just pull the images on the NMs.
> -- When a component instance is upgrade, it calls the
> {{reInitializeContainer}} with the flag pull-image set to false, since the NM
> will have already pulled the images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8643) Docker image life cycle management on HDFS
[ https://issues.apache.org/jira/browse/YARN-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8643: Description: This JIRA is forking an idea proposed in YARN-3854 to manage trusted docker images on HDFS. YARN-3854 is going to focus on rearranging the node manager and container-executor logic to download docker image during file localization phase for better error handling and reporting. The new proposal is to create a docker container which host Docker registry and NFS gateway, and configure docker images to store on HDFS. The docker container can run in YARN framework to serve as privileged or trusted registries. HDFS_Docker_registry_design.png describes the components required to create a docker image that can store docker images on HDFS. Deploy_HDFS_Docker_registry_workflow.png describes the operation to perform to setup a trusted docker registry that write to HDFS. was:This JIRA is forking an idea proposed in YARN-3854 to manage trusted docker images on HDFS. YARN-3854 is going to focus on rearranging the node manager and container-executor logic to download docker image during file localization phase for better error handling and reporting. The new proposal is to create a docker container which host Docker registry and NFS gateway, and configure docker images to store on HDFS. The docker container can run in YARN framework to serve as privileged or trusted registries. > Docker image life cycle management on HDFS > -- > > Key: YARN-8643 > URL: https://issues.apache.org/jira/browse/YARN-8643 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Eric Yang >Priority: Major > Labels: Docker > Attachments: Deploy_HDFS_Docker_registry_workflow.png, > HDFS_Docker_registry_design.png > > > This JIRA is forking an idea proposed in YARN-3854 to manage trusted docker > images on HDFS. YARN-3854 is going to focus on rearranging the node manager > and container-executor logic to download docker image during file > localization phase for better error handling and reporting. The new proposal > is to create a docker container which host Docker registry and NFS gateway, > and configure docker images to store on HDFS. The docker container can run > in YARN framework to serve as privileged or trusted registries. > HDFS_Docker_registry_design.png describes the components required to create a > docker image that can store docker images on HDFS. > Deploy_HDFS_Docker_registry_workflow.png describes the operation to perform > to setup a trusted docker registry that write to HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8643) Docker image life cycle management on HDFS
[ https://issues.apache.org/jira/browse/YARN-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8643: Attachment: HDFS_Docker_registry_design.png > Docker image life cycle management on HDFS > -- > > Key: YARN-8643 > URL: https://issues.apache.org/jira/browse/YARN-8643 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Eric Yang >Priority: Major > Labels: Docker > Attachments: Deploy_HDFS_Docker_registry_workflow.png, > HDFS_Docker_registry_design.png > > > This JIRA is forking an idea proposed in YARN-3854 to manage trusted docker > images on HDFS. YARN-3854 is going to focus on rearranging the node manager > and container-executor logic to download docker image during file > localization phase for better error handling and reporting. The new proposal > is to create a docker container which host Docker registry and NFS gateway, > and configure docker images to store on HDFS. The docker container can run > in YARN framework to serve as privileged or trusted registries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8643) Docker image life cycle management on HDFS
[ https://issues.apache.org/jira/browse/YARN-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8643: Attachment: Deploy_HDFS_Docker_registry_workflow.png > Docker image life cycle management on HDFS > -- > > Key: YARN-8643 > URL: https://issues.apache.org/jira/browse/YARN-8643 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Eric Yang >Priority: Major > Labels: Docker > Attachments: Deploy_HDFS_Docker_registry_workflow.png > > > This JIRA is forking an idea proposed in YARN-3854 to manage trusted docker > images on HDFS. YARN-3854 is going to focus on rearranging the node manager > and container-executor logic to download docker image during file > localization phase for better error handling and reporting. The new proposal > is to create a docker container which host Docker registry and NFS gateway, > and configure docker images to store on HDFS. The docker container can run > in YARN framework to serve as privileged or trusted registries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8643) Docker image life cycle management on HDFS
Eric Yang created YARN-8643: --- Summary: Docker image life cycle management on HDFS Key: YARN-8643 URL: https://issues.apache.org/jira/browse/YARN-8643 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Eric Yang This JIRA is forking an idea proposed in YARN-3854 to manage trusted docker images on HDFS. YARN-3854 is going to focus on rearranging the node manager and container-executor logic to download docker image during file localization phase for better error handling and reporting. The new proposal is to create a docker container which host Docker registry and NFS gateway, and configure docker images to store on HDFS. The docker container can run in YARN framework to serve as privileged or trusted registries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8642) Add support for tmpfs mounts with the Docker runtime
Shane Kumpf created YARN-8642:
-
Summary: Add support for tmpfs mounts with the Docker runtime
Key: YARN-8642
URL: https://issues.apache.org/jira/browse/YARN-8642
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Shane Kumpf
Add support to the existing Docker runtime to allow the user to request tmpfs
mounts for their containers. For example:
{code}/usr/bin/docker run --name=container_name --tmpfs /run image
/bootstrap/start-systemd
{code}
One use case is to allow systemd to run as PID 1 in a non-privileged container,
/run is expected to be a tmpfs mount in the container for that to work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8559) Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint
[ https://issues.apache.org/jira/browse/YARN-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575123#comment-16575123 ] Hudson commented on YARN-8559: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14739 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14739/]) YARN-8559. Expose mutable-conf scheduler's configuration in RM (wwei: rev d352f167ebb865a6486afbbdac8e2a5e97a7bbad) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/MutableConfigurationProvider.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ConfInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesConfigurationMutation.java > Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint > > > Key: YARN-8559 > URL: https://issues.apache.org/jira/browse/YARN-8559 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Anna Savarin >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8559.001.patch, YARN-8559.002.patch, > YARN-8559.003.patch, YARN-8559.004.patch > > > All Hadoop services provide a set of common endpoints (/stacks, /logLevel, > /metrics, /jmx, /conf). In the case of the Resource Manager, part of the > configuration comes from the scheduler being used. Currently, these > configuration key/values are not exposed through the /conf endpoint, thereby > revealing an incomplete configuration picture. > Make an improvement and expose the scheduling configuration info through the > RM's /conf endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8521) NPE in AllocationTagsManager when a container is removed more than once
[
https://issues.apache.org/jira/browse/YARN-8521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575125#comment-16575125
]
Suma Shivaprasad commented on YARN-8521:
Thanks [~cheersyang] +1. Patch 003 LGTM. Can you pls check if UT failures are
related?
> NPE in AllocationTagsManager when a container is removed more than once
> ---
>
> Key: YARN-8521
> URL: https://issues.apache.org/jira/browse/YARN-8521
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8521.001.patch, YARN-8521.002.patch,
> YARN-8521.003.patch
>
>
> We've seen sometimes there is NPE in AllocationTagsManager
> {code:java}
> private void removeTagFromInnerMap(Map innerMap, String tag) {
> Long count = innerMap.get(tag);
> if (count > 1) { // NPE!!
> ...
> {code}
> it seems {{AllocationTagsManager#removeContainer}} somehow gets called more
> than once for a same container.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8559) Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint
[ https://issues.apache.org/jira/browse/YARN-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575111#comment-16575111 ] Weiwei Yang commented on YARN-8559: --- Pushed to trunk, branch-3.1. > Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint > > > Key: YARN-8559 > URL: https://issues.apache.org/jira/browse/YARN-8559 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Anna Savarin >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8559.001.patch, YARN-8559.002.patch, > YARN-8559.003.patch, YARN-8559.004.patch > > > All Hadoop services provide a set of common endpoints (/stacks, /logLevel, > /metrics, /jmx, /conf). In the case of the Resource Manager, part of the > configuration comes from the scheduler being used. Currently, these > configuration key/values are not exposed through the /conf endpoint, thereby > revealing an incomplete configuration picture. > Make an improvement and expose the scheduling configuration info through the > RM's /conf endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8641) Refresh Queues ResourceManager - Failed to re-init queues : Unable to construct queue ordering policy=fifo queue=root.llap
Diego created YARN-8641:
Summary: Refresh Queues ResourceManager - Failed to re-init queues
: Unable to construct queue ordering policy=fifo queue=root.llap
Key: YARN-8641
URL: https://issues.apache.org/jira/browse/YARN-8641
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler, resourcemanager
Affects Versions: 3.1.0
Environment: -Ambari 2.7
-HDP3
-YARN
{{$ yarn version}}
{{Hadoop 3.1.0.3.0.0.0-1634}}
{{Source code repository g...@github.com:hortonworks/hadoop.git -r
ab6fed463d44250e0f1cc433987c9de0592db596}}
{{Compiled by jenkins on 2018-07-12T20:32Z}}
{{Compiled with protoc 2.5.0}}
{{From source with checksum 13431af4b6ec467e2496f7dd95d9dd}}
{{This command was run using
/usr/hdp/3.0.0.0-1634/hadoop/hadoop-common-3.1.0.3.0.0.0-1634.jar}}
Reporter: Diego
Steps:
in Ambari (2.7) Yarn Queue manager, I've created a leaf queue as follows:
{{/root}}
{{--/default}}
{{--/llap}}
{{/test}}
After doing save and refresh, the refresh YARN Capacity Scheduler fails. It
does take the changes and the queue seams to work fine if Yarn is restarted.
the refresh YARN Capacity Scheduler error is:
resource_management.core.exceptions.ExecutionFailed: Execution of ' export
HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.0.0-1634/hadoop/libexec &&
/usr/hdp/3.0.0.0-1634/hadoop-yarn/bin/yarn rmadmin -refreshQueues' returned
255. 18/08/09 15:08:51 INFO client.ConfiguredRMFailoverProxyProvider: Failing
over to rm2
refreshQueues: java.io.IOException: Failed to re-init queues : Unable to
construct queue ordering policy=fifo queue=root.llap
at
org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.logAndWrapException(AdminService.java:913)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:399)
at
org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceManagerAdministrationProtocolPBServiceImpl.refreshQueues(ResourceManagerAdministrationProtocolPBServiceImpl.java:114)
at
org.apache.hadoop.yarn.proto.ResourceManagerAdministrationProtocol$ResourceManagerAdministrationProtocolService$2.callBlockingMethod(ResourceManagerAdministrationProtocol.java:271)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
Caused by: java.io.IOException: Failed to re-init queues : Unable to construct
queue ordering policy=fifo queue=root.llap
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:476)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:394)
... 10 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to
construct queue ordering policy=fifo queue=root.llap
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1505)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:143)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:110)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:275)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:171)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:725)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:471)
... 12 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Commented] (YARN-8331) Race condition in NM container launched after done
[
https://issues.apache.org/jira/browse/YARN-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575031#comment-16575031
]
Hudson commented on YARN-8331:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14738 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14738/])
YARN-8331. Race condition in NM container launched after done. (jlowe: rev
cd04e954d2db27f0a15b7d1c492b7cdb656a51db)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
> Race condition in NM container launched after done
> --
>
> Key: YARN-8331
> URL: https://issues.apache.org/jira/browse/YARN-8331
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1, 3.0.2
>Reporter: Yang Wang
>Assignee: Pradeep Ambati
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
>
> Attachments: YARN-8331.001.patch, YARN-8331.002.patch
>
>
> When a container is launching, in ContainerLaunch#launchContainer, state is
> SCHEDULED,
> kill event was sent to this container, state : SCHEDULED->KILLING->DONE
> Then ContainerLaunch send CONTAINER_LAUNCHED event and start the container
> processes. These absent container processes will not be cleaned up anymore.
>
> {code:java}
> 2018-05-21 13:11:56,114 INFO [Thread-11] nodemanager.NMAuditLogger
> (NMAuditLogger.java:logSuccess(94)) - USER=nobody OPERATION=Start Container
> Request TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_0_CONTAINERID=container_0__01_00
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from NEW to INITING
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:transition(446)) - Adding
> container_0__01_00 to application application_0_
> 2018-05-21 13:11:56,118 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from INITING to RUNNING
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from NEW to SCHEDULED
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> containermanager.AuxServices (AuxServices.java:handle(220)) - Got event
> CONTAINER_INIT for appId application_0_
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> scheduler.ContainerScheduler (ContainerScheduler.java:startContainer(504)) -
> Starting container [container_0__01_00]
> 2018-05-21 13:11:56,226 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from SCHEDULED to KILLING
> 2018-05-21 13:11:56,227 INFO [NM ContainerManager dispatcher]
> containermanager.TestContainerManager
> (BaseContainerManagerTest.java:delete(287)) - Psuedo delete: user - nobody,
> type - FILE
> 2018-05-21 13:11:56,227 INFO [NM ContainerManager dispatcher]
> nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(94)) - USER=nobody
> OPERATION=Container Finished - Killed TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_0_
> CONTAINERID=container_0__01_00
> 2018-05-21 13:11:56,238 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from KILLING to DONE
> 2018-05-21 13:11:56,238 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:transition(489)) - Removing
> container_0__01_00 from application application_0_
> 2018-05-21 13:11:56,239 INFO [NM ContainerManager dispatcher]
> monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:onStopMonitoringContainer(932)) - Stopping
> resource-monitoring for container_0__01_00
> 2018-05-21
[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement
[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575020#comment-16575020
]
Weiwei Yang commented on YARN-7494:
---
Hi [~sunilg]
I think to get this not a CS only feature was from this comment commented by
[~leftnoteasy],
{quote}Ideally, MultiNodeLookupPolicy initialization should not determined by
Scheduler, and implementation of MultiNodeSortingManager should not bind to
specific scheduler.
{quote}
my opinion is if this causes burden ending up with changing a lot of files, it
seems better to limit this feature in CS. Just let the sorting manager be a
common service would be enough.
*LocalityAppPlacementAllocator*
line 90-94: we should not print this debug message if nodeLookupPolicy is null.
line 107: if multi-nodes lookup enabled and an app is giving an incorrect name,
then singleNode is null and nodeLookupPolicy is also null, then looks like
this will cause NPE.
*TestCapacityScheduler*
line 33: this seems to be an unintentional change
line 145-146: unused imports
*ResourceUsageMultiNodeLookupPolicy*
Currently this policy is using synchronized methods for thread safe, but this
could be sub-optimal. Scheduler alloc happens very fast, it may cause alloc
thread be waiting for sorting to be completed. This will get worse when there
is multiple alloc threads and a large number of nodes. In another word, we need
to decouple the SORT and GET.
*ResourceUsageMultiNodeLookupPolicyWithNoAutoSort*
I don't think we need another policy to achieve NoAutoSort. It is nature to
configure the {{sorting interval}} to 0 for
{{ResourceUsageMultiNodeLookupPolicy}} for this. How hard to get it done in
this way?
Hope it helps.
Thanks
> Add muti node lookup support for better placement
> -
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Sunil Govindan
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch,
> YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch,
> YARN-7494.006.patch, YARN-7494.007.patch, YARN-7494.008.patch,
> YARN-7494.009.patch, YARN-7494.010.patch, YARN-7494.11.patch,
> YARN-7494.12.patch, YARN-7494.v0.patch, YARN-7494.v1.patch,
> multi-node-designProposal.png
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup
> based on partition to start with.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8561) [Submarine] Add initial implementation: training job submission and job history retrieve.
[
https://issues.apache.org/jira/browse/YARN-8561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574994#comment-16574994
]
Sunil Govindan commented on YARN-8561:
--
Thanks [~leftnoteasy]
>> I'm not quite sure about this suggestion, it seems to me that we should add
>> the getServiceResourceFromYarnResource method to service.Resource instead. I
>> don't want to touch any service classes in this patch. Should we do it in a
>> separate JIRA?
Yes. This makes sense.
>> I think we can push this to the future patch, one possible solution is to
>>include a yaml file to describe job configs and user can reuse it instead of
>>passing 10+ params to CLI.
This could be a followup jira. I agree.
Few more comments:
# In {{Cli}} class, could we use Options for CLI parsing. This will help to do
paring and add help much better.
# We have a bunch of constants defined in {{CliConstants}}> going forward we
will more here. I am not sure whether this is a good idea. Could we load this
from a config file something like resource-types.xml where all such commands
can be loaded. A new command can be added with a definition in this spec file
without code change.
# RemoteDirectoryManager holds information about dirs and fs. Currently its
only in HDFS, or could we also support local file?
# CliUtils#replacePatternsInLaunchCommand
{code:java}
65 String newCli = specifiedCli;
66 for (Map.Entry replace : replacePattern.entrySet()) {
67 newCli = newCli.replace(replace.getKey(), replace.getValue());
68 }{code}
I didnt understand this very cleanly. We want to replace the value in
the specifiedCli to newCli, correct? Key will be the string which starts with %.
5. I think StringUtils#strip is better to trim chars like '[' and ']' in
{{parseResourcesString}}
6. {{!resource.matches("^[^=]+=\\d+\\s?\\w*$"}} In parseResourcesString, its
better to put this in const string
7. Yes, UnitsConversionUtil is more exhaustive and might have a bit diff
meaning. But could we refactor the unit conversion code in
parseResourcesString to a util, as i think it might help for some other apps
too.
8. Resource profile support in cli? May help to avoid specify whole resources?
9. Submarine cli might need to support app priority, app timeout, queue mapping
etc ?
10. Submarine kerberos support will be in subsequent jira, correct?
11. In cli, CliConstants.ENV helps to add envs to job. But we also need to
specify ENV's per component level, correct?
12. May be checksum verification needed for FSBasedSubmarineStorageImpl
13. I think we need a translation from ServiceState to JobState. Because more
and more states are added native service, and it will be tough to map to
submarine.
14. In general i think this is a great effort in getting cli, job tracking,
store etc in to one framework. Parameter validation and error handling is
always tougher, but is there a way where we can cleanly show error comes from
native service if the image doesnt exist or corrupted or no permission etc to
be popped up in submarine level.
> [Submarine] Add initial implementation: training job submission and job
> history retrieve.
> -
>
> Key: YARN-8561
> URL: https://issues.apache.org/jira/browse/YARN-8561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
> Attachments: YARN-8561.001.patch, YARN-8561.002.patch,
> YARN-8561.003.patch, YARN-8561.004.patch
>
>
> Added following parts:
> 1) New subcomponent of YARN, under applications/ project.
> 2) Tensorflow training job submission, including training (single node and
> distributed).
> - Supported Docker container.
> - Support GPU isolation.
> - Support YARN registry DNS.
> 3) Retrieve job history.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8331) Race condition in NM container launched after done
[
https://issues.apache.org/jira/browse/YARN-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574989#comment-16574989
]
Jason Lowe commented on YARN-8331:
--
Thanks for updating the patch! +1 lgtm. Commiting this.
> Race condition in NM container launched after done
> --
>
> Key: YARN-8331
> URL: https://issues.apache.org/jira/browse/YARN-8331
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.9.1, 3.0.2
>Reporter: Yang Wang
>Assignee: Pradeep Ambati
>Priority: Major
> Attachments: YARN-8331.001.patch, YARN-8331.002.patch
>
>
> When a container is launching, in ContainerLaunch#launchContainer, state is
> SCHEDULED,
> kill event was sent to this container, state : SCHEDULED->KILLING->DONE
> Then ContainerLaunch send CONTAINER_LAUNCHED event and start the container
> processes. These absent container processes will not be cleaned up anymore.
>
> {code:java}
> 2018-05-21 13:11:56,114 INFO [Thread-11] nodemanager.NMAuditLogger
> (NMAuditLogger.java:logSuccess(94)) - USER=nobody OPERATION=Start Container
> Request TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_0_CONTAINERID=container_0__01_00
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from NEW to INITING
> 2018-05-21 13:11:56,114 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:transition(446)) - Adding
> container_0__01_00 to application application_0_
> 2018-05-21 13:11:56,118 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:handle(632)) - Application
> application_0_ transitioned from INITING to RUNNING
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from NEW to SCHEDULED
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> containermanager.AuxServices (AuxServices.java:handle(220)) - Got event
> CONTAINER_INIT for appId application_0_
> 2018-05-21 13:11:56,119 INFO [NM ContainerManager dispatcher]
> scheduler.ContainerScheduler (ContainerScheduler.java:startContainer(504)) -
> Starting container [container_0__01_00]
> 2018-05-21 13:11:56,226 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from SCHEDULED to KILLING
> 2018-05-21 13:11:56,227 INFO [NM ContainerManager dispatcher]
> containermanager.TestContainerManager
> (BaseContainerManagerTest.java:delete(287)) - Psuedo delete: user - nobody,
> type - FILE
> 2018-05-21 13:11:56,227 INFO [NM ContainerManager dispatcher]
> nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(94)) - USER=nobody
> OPERATION=Container Finished - Killed TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_0_
> CONTAINERID=container_0__01_00
> 2018-05-21 13:11:56,238 INFO [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2111)) - Container
> container_0__01_00 transitioned from KILLING to DONE
> 2018-05-21 13:11:56,238 INFO [NM ContainerManager dispatcher]
> application.ApplicationImpl (ApplicationImpl.java:transition(489)) - Removing
> container_0__01_00 from application application_0_
> 2018-05-21 13:11:56,239 INFO [NM ContainerManager dispatcher]
> monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:onStopMonitoringContainer(932)) - Stopping
> resource-monitoring for container_0__01_00
> 2018-05-21 13:11:56,239 INFO [NM ContainerManager dispatcher]
> containermanager.AuxServices (AuxServices.java:handle(220)) - Got event
> CONTAINER_STOP for appId application_0_
> 2018-05-21 13:11:56,274 WARN [NM ContainerManager dispatcher]
> container.ContainerImpl (ContainerImpl.java:handle(2106)) - Can't handle this
> event at current state: Current: [DONE], eventType: [CONTAINER_LAUNCHED],
> container: [container_0__01_00]
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> CONTAINER_LAUNCHED at DONE
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:2104)
> at
>
[jira] [Created] (YARN-8640) Restore previous state in container-executor if write_exit_code_file_as_nm fails
Jim Brennan created YARN-8640:
-
Summary: Restore previous state in container-executor if
write_exit_code_file_as_nm fails
Key: YARN-8640
URL: https://issues.apache.org/jira/browse/YARN-8640
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jim Brennan
Assignee: Jim Brennan
The container-executor function {{write_exit_code_file_as_nm}} had a number of
failure conditions where it just returns -1 without restoring previous state.
This is not a problem in any of the places where it is currently called, but it
could be a problem if future code changes call it before code that depends on
the previous state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement
[
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574790#comment-16574790
]
genericqa commented on YARN-7494:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 1s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerMultiNodes
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-7494 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934958/YARN-7494.12.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 60a687ddab39 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3d96bc6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/21546/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21546/testReport/ |
| Max. process+thread count | 920 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Issue Comment Deleted] (YARN-8450) Blocking resources such as GPU/FPGA etc tend to release actual device slowly even after RM identifies it as COMPLETED
[
https://issues.apache.org/jira/browse/YARN-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-8450:
---
Comment: was deleted
(was: [~sunilg]/[~eepayne]
During kill scenarios/preemption cases this issue mainly gets exposed.
Thoughts on moving the resource check to {{ResourceHandlerChain}}.
Solution could be wait until the resource is released by {{resourceHandlers}}
which has strict binding.
or Adding {{canAssign}} interface to resource handlers, and Query canAssign
till timeout. Thoughts?)
> Blocking resources such as GPU/FPGA etc tend to release actual device slowly
> even after RM identifies it as COMPLETED
> -
>
> Key: YARN-8450
> URL: https://issues.apache.org/jira/browse/YARN-8450
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.2
>Reporter: Sunil Govindan
>Assignee: Bilwa S T
>Priority: Major
>
> For resources such as GPU/FPGA or similar resources, sometimes we have seen
> that device is not released from a container even after container is in
> completed states.
> In such cases, we need a common way of handling from NM level. YARN-8423 is
> only handling this for GPU.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8450) Blocking resources such as GPU/FPGA etc tend to release actual device slowly even after RM identifies it as COMPLETED
[
https://issues.apache.org/jira/browse/YARN-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574711#comment-16574711
]
Bibin A Chundatt commented on YARN-8450:
[~sunilg]/[~eepayne]
During kill scenarios/preemption cases this issue mainly gets exposed.
Thoughts on moving the resource check to {{ResourceHandlerChain}}.
Solution could be wait until the resource is released by {{resourceHandlers}}
which has strict binding.
or Adding {{canAssign}} interface to resource handlers, and Query canAssign
till timeout. Thoughts?
> Blocking resources such as GPU/FPGA etc tend to release actual device slowly
> even after RM identifies it as COMPLETED
> -
>
> Key: YARN-8450
> URL: https://issues.apache.org/jira/browse/YARN-8450
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.2
>Reporter: Sunil Govindan
>Assignee: Bilwa S T
>Priority: Major
>
> For resources such as GPU/FPGA or similar resources, sometimes we have seen
> that device is not released from a container even after container is in
> completed states.
> In such cases, we need a common way of handling from NM level. YARN-8423 is
> only handling this for GPU.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8450) Blocking resources such as GPU/FPGA etc tend to release actual device slowly even after RM identifies it as COMPLETED
[
https://issues.apache.org/jira/browse/YARN-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574710#comment-16574710
]
Bibin A Chundatt commented on YARN-8450:
[~sunilg]/[~eepayne]
During kill scenarios/preemption cases this issue mainly gets exposed.
Thoughts on moving the resource check to {{ResourceHandlerChain}}.
Solution could be wait until the resource is released by {{resourceHandlers}}
which has strict binding.
or Adding {{canAssign}} interface to resource handlers, and Query canAssign
till timeout. Thoughts?
> Blocking resources such as GPU/FPGA etc tend to release actual device slowly
> even after RM identifies it as COMPLETED
> -
>
> Key: YARN-8450
> URL: https://issues.apache.org/jira/browse/YARN-8450
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.2
>Reporter: Sunil Govindan
>Assignee: Bilwa S T
>Priority: Major
>
> For resources such as GPU/FPGA or similar resources, sometimes we have seen
> that device is not released from a container even after container is in
> completed states.
> In such cases, we need a common way of handling from NM level. YARN-8423 is
> only handling this for GPU.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7957) [UI2] Yarn service delete option disappears after stopping application
[
https://issues.apache.org/jira/browse/YARN-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574687#comment-16574687
]
Akhil PB commented on YARN-7957:
Below are the service states enums defined.
{code:java}
public enum ServiceState {
ACCEPTED, STARTED, STABLE, STOPPED, FAILED, FLEX, UPGRADING,
UPGRADING_AUTO_FINALIZE;
}
{code}
So for a deployed service,
1. Show both Stop and Delete button when states are
{code:java}
ACCEPTED, STARTED, STABLE, FLEX, UPGRADING, UPGRADING_AUTO_FINALIZE
{code}
2. Show Delete button only when state is
{code:java}
STOPPED
{code}
3. Do not show any buttons when state is
{code:java}
FAILED
{code}
OR if response code is 404 error.
cc [~sunilg] [~gsaha] please share your thoughts.
> [UI2] Yarn service delete option disappears after stopping application
> --
>
> Key: YARN-7957
> URL: https://issues.apache.org/jira/browse/YARN-7957
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Akhil PB
>Priority: Critical
> Attachments: YARN-7957.001.patch
>
>
> Steps:
> 1) Launch yarn service
> 2) Go to service page and click on Setting button->"Stop Service". The
> application will be stopped.
> 3) Refresh page
> Here, setting button disappears. Thus, user can not delete service from UI
> after stopping application
> Expected behavior:
> Setting button should be present on UI page after application is stopped. If
> application is stopped, setting button should only have "Delete Service"
> action available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement
[ https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574680#comment-16574680 ] Sunil Govindan commented on YARN-7494: -- Thanks [~cheersyang] and [~leftnoteasy] Almost fixed all comments. 5) It's better to add getMultiNodeSortingPolicyName CSQueue, inside FiCaSchedulerApp, you can check the instanceof. With this we can limit the changes to CS. Sunil: We initially had it for only CS. But during our discussion with [~cheersyang] we thought its better be generic so that FS can use with ease. And less if checks in FicaSchedulerApp. 6) RegularContainerAllocator#allocate, when the change needed? Earlier for each node heartbeat, we were doing precheck. and then allocate was been invoked. In multi-node mode, we should try to do precheck and allocate in same loop of sorted nodes so that if one check fails,next node could be looked sooner. 7) Instead of adding MultiNodeSortingManager to RMContext, can we limit changes inside Scheduler? Here also We initially had it for only CS. But during this patch review with [~cheersyang], we thought this might be better for a common approach. > Add muti node lookup support for better placement > - > > Key: YARN-7494 > URL: https://issues.apache.org/jira/browse/YARN-7494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7494.001.patch, YARN-7494.002.patch, > YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch, > YARN-7494.006.patch, YARN-7494.007.patch, YARN-7494.008.patch, > YARN-7494.009.patch, YARN-7494.010.patch, YARN-7494.11.patch, > YARN-7494.12.patch, YARN-7494.v0.patch, YARN-7494.v1.patch, > multi-node-designProposal.png > > > Instead of single node, for effectiveness we can consider a multi node lookup > based on partition to start with. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7494) Add muti node lookup support for better placement
[ https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-7494: - Attachment: YARN-7494.12.patch > Add muti node lookup support for better placement > - > > Key: YARN-7494 > URL: https://issues.apache.org/jira/browse/YARN-7494 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-7494.001.patch, YARN-7494.002.patch, > YARN-7494.003.patch, YARN-7494.004.patch, YARN-7494.005.patch, > YARN-7494.006.patch, YARN-7494.007.patch, YARN-7494.008.patch, > YARN-7494.009.patch, YARN-7494.010.patch, YARN-7494.11.patch, > YARN-7494.12.patch, YARN-7494.v0.patch, YARN-7494.v1.patch, > multi-node-designProposal.png > > > Instead of single node, for effectiveness we can consider a multi node lookup > based on partition to start with. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7957) [UI2] Yarn service delete option disappears after stopping application
[
https://issues.apache.org/jira/browse/YARN-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574674#comment-16574674
]
Sunil Govindan commented on YARN-7957:
--
Thanks [~akhilpb]. Patch almost looks good.
Pls check any more states to be added to {{const serviceStates = ['ACCEPTED',
'STARTED', 'STABLE', 'RUNNING'];}}
> [UI2] Yarn service delete option disappears after stopping application
> --
>
> Key: YARN-7957
> URL: https://issues.apache.org/jira/browse/YARN-7957
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Akhil PB
>Priority: Critical
> Attachments: YARN-7957.001.patch
>
>
> Steps:
> 1) Launch yarn service
> 2) Go to service page and click on Setting button->"Stop Service". The
> application will be stopped.
> 3) Refresh page
> Here, setting button disappears. Thus, user can not delete service from UI
> after stopping application
> Expected behavior:
> Setting button should be present on UI page after application is stopped. If
> application is stopped, setting button should only have "Delete Service"
> action available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8639) Sort queue and apps in fair schduler using a separate thread
wan kun created YARN-8639: - Summary: Sort queue and apps in fair schduler using a separate thread Key: YARN-8639 URL: https://issues.apache.org/jira/browse/YARN-8639 Project: Hadoop YARN Issue Type: Improvement Components: yarn Reporter: wan kun If fair scheduler have many queue and each queue have many active applications . For each assignContainer function, we need to sort all the queue, and all the applications in each queue. For a large system,this may be cost too much time. So we can sort the queue and applications using a separate thread asynchronous. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[
https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574418#comment-16574418
]
Konstantinos Karanasos commented on YARN-7953:
--
Thanks [~abmodi] for working on this. A few comments on the latest patch:
* Can we improve a bit the {{propagate}} method of the {{FederationQueue}}?
The name is not very informative and also it does two things, that is,
propagate total capacity top-down, and propagate the rest of the metrics
bottom-up. Let's split that into two methods, one for bottom-up and one for
top-down. Then we can keep the propagate (call it propagateCapacities?) that
will call both.
* Some public methods of FederationQueue do not have comments, including
propagate, rollupIdealAssignedFromChildren, countQueues, etc. Same for the
FedQueueIterator class (it creates an iterator to go over all queues of a
FederationQueue hierarchy).
* [minor] In the first two classes of the patch: This class represent -> This
class represents
* Do we need all three json files for the tests?
> [GQ] Data structures for federation global queues calculations
> --
>
> Key: YARN-7953
> URL: https://issues.apache.org/jira/browse/YARN-7953
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Carlo Curino
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-7953-YARN-7402.v1.patch,
> YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch,
> YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch,
> YARN-7953-YARN-7402.v6.patch, YARN-7953-YARN-7402.v7.patch, YARN-7953.v1.patch
>
>
> This Jira tracks data structures and helper classes used by the core
> algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574383#comment-16574383 ] Hudson commented on YARN-8633: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14735 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14735/]) YARN-8633. Update DataTables version in yarn-common in line with JQuery (sunilg: rev 00013d6ef7fdf65fa8a0f6eb56c0aef2f6e19444) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/css/jui-dt.css * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_asc_disabled.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/css/demo_table.css * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_asc_disabled.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/back_disabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/Sorting icons.psd * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/forward_enabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/favicon.ico * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/back_disabled.jpg * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_both.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/css/demo_page.css * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_desc_disabled.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_asc.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/Sorting icons.psd * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/forward_disabled.jpg * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/back_enabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/css/demo_table.css * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_desc.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/css/demo_page.css * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_desc.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/js/jquery.dataTables.min.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/sort_asc.png * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/back_enabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_desc_disabled.png * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/sort_both.png * (edit) LICENSE.txt * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/favicon.ico * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/js/jquery.dataTables.min.js * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/forward_disabled.jpg * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/css/jui-dt.css * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.10.7/images/forward_enabled.jpg > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Fix
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574382#comment-16574382 ] Sunil Govindan commented on YARN-8633: -- Thanks [~akhilpb] for the patch. And thanks [~msingh] for additional reviews. Committed to trunk/branch-3.1 > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8633.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574380#comment-16574380 ] Sunil Govindan commented on YARN-8633: -- I pulled YARN-8426 to branch-3.1. This will help to cherrypick this patch to branch-3.1 without additional rebase. > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8633.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8426) Upgrade jquery-ui to 1.12.1 in YARN
[ https://issues.apache.org/jira/browse/YARN-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574375#comment-16574375 ] Sunil Govindan commented on YARN-8426: -- I also back ported to branch-3.1. Thanks > Upgrade jquery-ui to 1.12.1 in YARN > --- > > Key: YARN-8426 > URL: https://issues.apache.org/jira/browse/YARN-8426 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8426.001.patch > > > In align to HADOOP-15483, upgrade jquery-ui for YARN common package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8426) Upgrade jquery-ui to 1.12.1 in YARN
[ https://issues.apache.org/jira/browse/YARN-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-8426: - Fix Version/s: 3.1.2 > Upgrade jquery-ui to 1.12.1 in YARN > --- > > Key: YARN-8426 > URL: https://issues.apache.org/jira/browse/YARN-8426 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Reporter: Sunil Govindan >Assignee: Sunil Govindan >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8426.001.patch > > > In align to HADOOP-15483, upgrade jquery-ui for YARN common package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574366#comment-16574366 ] Sunil Govindan commented on YARN-8633: -- Committed to trunk. Cherry-pick to branch-3.1 is failing. [~akhilpb] pls help to share branch-3.1 patch. > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8633.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574353#comment-16574353 ] Sunil Govindan commented on YARN-8633: -- Thanks [~msingh] for confirming. I will commit this patch now. Will take care of the whitespace while committing.Thanks [~akhilpb] > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8633.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8633) Update DataTables version in yarn-common in line with JQuery 3 upgrade
[ https://issues.apache.org/jira/browse/YARN-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574352#comment-16574352 ] Mukul Kumar Singh commented on YARN-8633: - Thanks for working on this [~akhilpb]. The HDFS related test failures are unrelated. +1 for the v1 patch. > Update DataTables version in yarn-common in line with JQuery 3 upgrade > -- > > Key: YARN-8633 > URL: https://issues.apache.org/jira/browse/YARN-8633 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Major > Attachments: YARN-8633.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
