[jira] [Commented] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374399#comment-15374399
]
Varun Saxena commented on YARN-5359:
Patch looks fine to me. Will commit it later today.
I will however go through the timelineservice code once again and see if we
aren't using unix style paths somewhere else.
> FileSystemTimelineReader/Writer uses unix-specific default
> --
>
> Key: YARN-5359
> URL: https://issues.apache.org/jira/browse/YARN-5359
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Attachments: YARN-5359.01.patch, YARN-5359.02.patch,
> YARN-5359.03.patch
>
>
> {{FileSystemTimelineReaderImpl}} and {{FileSystemTimelineWriterImpl}} use a
> unix-specific default. It won't work on Windows.
> Also, {{TestFileSystemTimelineReaderImpl}} uses this default directly, which
> is also brittle against concurrent tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374389#comment-15374389 ] Naganarasimha G R commented on YARN-5287: - We faced issue when umask was set to 077. Well basically issue was identified and fixed by [~surendrasingh]. May be he can add more light on the test failure if any... > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-naga.patch, YARN-5287.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5368) memory leak at timeline server
[
https://issues.apache.org/jira/browse/YARN-5368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374383#comment-15374383
]
Naganarasimha G R commented on YARN-5368:
-
hi [~wyukawa],
5th column of the top output which you are pointing to refers to the {{Virtual
Memory Size}} which includes all code, data and shared libraries plus pages
that have been swapped out and pages that have been mapped but not used. So its
not precisely using 25GB of ram.
Are you facing any particular impact from this and also it would be helpful if
you shares how many Open files are present for the timeline service process.
> memory leak at timeline server
> --
>
> Key: YARN-5368
> URL: https://issues.apache.org/jira/browse/YARN-5368
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.7.1
> Environment: HDP2.4
> CentOS 6.7
> jdk1.8.0_72
>Reporter: Wataru Yukawa
>
> memory usage of timeline server machine increases gradually.
> https://gyazo.com/952dad96c77ae053bae2e4d8c8ab0572
> please check since April.
> According to my investigation, timeline server used about 25GB.
> top command result
> {code}
> 90577 yarn 20 0 28.4g 25g 12m S 0.0 40.1 5162:53
> /usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m
> -Dhdp.version=2.4.0.0-169 -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
> -Dyarn.log.dir=/var/log/hadoop-yarn/yarn ...
> {code}
> ps command result
> {code}
> $ ps ww 90577
> 90577 ?Sl 5162:53 /usr/java/jdk1.8.0_72/bin/java
> -Dproc_timelineserver -Xmx1024m -Dhdp.version=2.4.0.0-169
> -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
> -Dyarn.log.dir=/var/log/hadoop-yarn/yarn
> -Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log
> -Dyarn.log.file=yarn-yarn-timelineserver-myhost.log -Dyarn.home.dir=
> -Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,EWMA,RFA
> -Dyarn.root.logger=INFO,EWMA,RFA
> -Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
> -Dyarn.policy.file=hadoop-policy.xml
> -Djava.io.tmpdir=/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
> -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
> -Dyarn.log.dir=/var/log/hadoop-yarn/yarn
> -Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log
> -Dyarn.log.file=yarn-yarn-timelineserver-myhost.log
> -Dyarn.home.dir=/usr/hdp/current/hadoop-yarn-timelineserver
> -Dhadoop.home.dir=/usr/hdp/2.4.0.0-169/hadoop
> -Dhadoop.root.logger=INFO,EWMA,RFA -Dyarn.root.logger=INFO,EWMA,RFA
> -Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
> -classpath
> /usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/lib/*:/usr/hdp/2.4.0.0-169/hadoop/.//*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/./:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//*:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/*:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//*::/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/current/hadoop-yarn-timelineserver/.//*:/usr/hdp/current/hadoop-yarn-timelineserver/lib/*:/usr/hdp/2.4.0.0-169/hadoop/conf/timelineserver-config/log4j.properties
>
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
> {code}
>
> Alghough I set -Xmx1024m, actual memory usage is 25GB.
> After I restart timeline server, memory usage of timeline server machine
> decreases.
> https://gyazo.com/130600c17a7d41df8606727a859ae7e3
> Now timelineserver uses less than 1GB memory.
> top command result
> {code}
> 6163 yarn 20 0 3959m 783m 46m S 0.3 1.2 3:37.60
> /usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m
> -Dhdp.version=2.4.0.0-169 ...
> {code}
> I suspect memory leak at timeline server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5321) [YARN-3368] Add resource usage for application by node managers
[
https://issues.apache.org/jira/browse/YARN-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374372#comment-15374372
]
Hadoop QA commented on YARN-5321:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m 2s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 37s
{color} | {color:red} The patch generated 14 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 4m 51s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:6d3a5f5 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817592/YARN-5321-YARN-3368.003.patch
|
| JIRA Issue | YARN-5321 |
| Optional Tests | asflicense |
| uname | Linux 113d279db62e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3368 / 8ec70d3 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/12301/artifact/patchprocess/whitespace-eol.txt
|
| asflicense |
https://builds.apache.org/job/PreCommit-YARN-Build/12301/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12301/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> [YARN-3368] Add resource usage for application by node managers
> ---
>
> Key: YARN-5321
> URL: https://issues.apache.org/jira/browse/YARN-5321
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-5321-YARN-3368-0001.patch,
> YARN-5321-YARN-3368.0002.patch, YARN-5321-YARN-3368.003.patch, sample-1.png
>
>
> With this, user can understand distribution of resources allocated to this
> application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5321) [YARN-3368] Add resource usage for application by node managers
[ https://issues.apache.org/jira/browse/YARN-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5321: -- Assignee: Wangda Tan (was: Sunil G) > [YARN-3368] Add resource usage for application by node managers > --- > > Key: YARN-5321 > URL: https://issues.apache.org/jira/browse/YARN-5321 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5321-YARN-3368-0001.patch, > YARN-5321-YARN-3368.0002.patch, YARN-5321-YARN-3368.003.patch, sample-1.png > > > With this, user can understand distribution of resources allocated to this > application. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5321) [YARN-3368] Add resource usage for application by node managers
[ https://issues.apache.org/jira/browse/YARN-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G reassigned YARN-5321: - Assignee: Sunil G (was: Wangda Tan) > [YARN-3368] Add resource usage for application by node managers > --- > > Key: YARN-5321 > URL: https://issues.apache.org/jira/browse/YARN-5321 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Sunil G > Attachments: YARN-5321-YARN-3368-0001.patch, > YARN-5321-YARN-3368.0002.patch, YARN-5321-YARN-3368.003.patch, sample-1.png > > > With this, user can understand distribution of resources allocated to this > application. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5321) [YARN-3368] Add resource usage for application by node managers
[ https://issues.apache.org/jira/browse/YARN-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5321: -- Attachment: YARN-5321-YARN-3368.003.patch Re attaching patch with some minor changes. > [YARN-3368] Add resource usage for application by node managers > --- > > Key: YARN-5321 > URL: https://issues.apache.org/jira/browse/YARN-5321 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Sunil G > Attachments: YARN-5321-YARN-3368-0001.patch, > YARN-5321-YARN-3368.0002.patch, YARN-5321-YARN-3368.003.patch, sample-1.png > > > With this, user can understand distribution of resources allocated to this > application. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5369) Improve Yarn logs command to get container logs based on Node Id
Xuan Gong created YARN-5369:
---
Summary: Improve Yarn logs command to get container logs based on
Node Id
Key: YARN-5369
URL: https://issues.apache.org/jira/browse/YARN-5369
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
It is helpful if we could have yarn logs --applicationId appId --nodeAddress
${nodeId} to get all the container logs which ran on the specific nm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374360#comment-15374360
]
Sangjin Lee commented on YARN-5359:
---
The failed tests are unrelated known issues.
> FileSystemTimelineReader/Writer uses unix-specific default
> --
>
> Key: YARN-5359
> URL: https://issues.apache.org/jira/browse/YARN-5359
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Attachments: YARN-5359.01.patch, YARN-5359.02.patch,
> YARN-5359.03.patch
>
>
> {{FileSystemTimelineReaderImpl}} and {{FileSystemTimelineWriterImpl}} use a
> unix-specific default. It won't work on Windows.
> Also, {{TestFileSystemTimelineReaderImpl}} uses this default directly, which
> is also brittle against concurrent tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5368) memory leak at timeline server
Wataru Yukawa created YARN-5368:
---
Summary: memory leak at timeline server
Key: YARN-5368
URL: https://issues.apache.org/jira/browse/YARN-5368
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 2.7.1
Environment: HDP2.4
CentOS 6.7
jdk1.8.0_72
Reporter: Wataru Yukawa
memory usage of timeline server machine increases gradually.
https://gyazo.com/952dad96c77ae053bae2e4d8c8ab0572
please check since April.
According to my investigation, timeline server used about 25GB.
top command result
{code}
90577 yarn 20 0 28.4g 25g 12m S 0.0 40.1 5162:53
/usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m
-Dhdp.version=2.4.0.0-169 -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
-Dyarn.log.dir=/var/log/hadoop-yarn/yarn ...
{code}
ps command result
{code}
$ ps ww 90577
90577 ?Sl 5162:53 /usr/java/jdk1.8.0_72/bin/java
-Dproc_timelineserver -Xmx1024m -Dhdp.version=2.4.0.0-169
-Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
-Dyarn.log.dir=/var/log/hadoop-yarn/yarn
-Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log
-Dyarn.log.file=yarn-yarn-timelineserver-myhost.log -Dyarn.home.dir=
-Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,EWMA,RFA
-Dyarn.root.logger=INFO,EWMA,RFA
-Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
-Dyarn.policy.file=hadoop-policy.xml
-Djava.io.tmpdir=/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
-Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
-Dyarn.log.dir=/var/log/hadoop-yarn/yarn
-Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log
-Dyarn.log.file=yarn-yarn-timelineserver-myhost.log
-Dyarn.home.dir=/usr/hdp/current/hadoop-yarn-timelineserver
-Dhadoop.home.dir=/usr/hdp/2.4.0.0-169/hadoop
-Dhadoop.root.logger=INFO,EWMA,RFA -Dyarn.root.logger=INFO,EWMA,RFA
-Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
-classpath
/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/lib/*:/usr/hdp/2.4.0.0-169/hadoop/.//*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/./:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//*:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/*:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//*::/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/current/hadoop-yarn-timelineserver/.//*:/usr/hdp/current/hadoop-yarn-timelineserver/lib/*:/usr/hdp/2.4.0.0-169/hadoop/conf/timelineserver-config/log4j.properties
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
{code}
Alghough I set -Xmx1024m, actual memory usage is 25GB.
After I restart timeline server, memory usage of timeline server machine
decreases.
https://gyazo.com/130600c17a7d41df8606727a859ae7e3
Now timelineserver uses less than 1GB memory.
top command result
{code}
6163 yarn 20 0 3959m 783m 46m S 0.3 1.2 3:37.60
/usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m
-Dhdp.version=2.4.0.0-169 ...
{code}
I suspect memory leak at timeline server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374353#comment-15374353
]
Hadoop QA commented on YARN-5359:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 6 unchanged - 1 fixed = 6 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 46s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 28s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 22s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.TestContainerManagerSecurity |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817588/YARN-5359.03.patch |
| JIRA Issue | YARN-5359 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ecbabe9affe0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 06c56ff |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12300/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
|
| unit test logs |
[jira] [Commented] (YARN-4091) Improvement: Introduce more debug/diagnostics information to detail out scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374343#comment-15374343 ] Chen Ge commented on YARN-4091: --- Hi all, Given "YARN-4091.preliminary.1.patch" I uploaded above, here are some brief descriptions about newly added classes and test REST API. Newly Added Classes: ActivityManager: A class to store node or application allocations. It mainly contains operations for allocation start, add, update and finish. NodeAllocation: It contains allocation information for one allocation in a node heartbeat. Detailed allocation activities are first stored in "AllocationActivity" as operations, then transformed to a tree structure. Tree structure starts from root queue and ends in leaf queue, application or container allocation. AllocationActivity: It records an activity operation in allocation, which can be classified as queue, application or container activity. Other information include state, diagnostic, priority. ActivityNode: It represents tree node in "NodeAllocation" tree structure. Each node may represent queue, application or container in allocation activity. Node may have children node if successfully allocated to next level. ActivityDiagnosticConstant: Collection of diagnostics. ActivityState: Collection of activity operation states. AllocationState: Collection of allocation final states. AllocationActivityType: Collection of types for activity operation. AppAllocation: It contains allocation information for one application within a period of time. Each application allocation may have several allocation attempts. ActivitiesInfo: DAO object to display node allocation activity. NodeAllocationInfo: DAO object to display each node allocation in node heartbeat. ActivityNodeInfo: DAO object to display node information in allocation tree. It corresponds to "ActivityNode" class. AppActivitiesInfo: DAO object to display application activity. AppAllocationInfo: DAO object to display application allocation detailed information. Test REST API: look at next node’s activities(by default): http://localhost:18088/ws/v1/cluster/scheduler/activities Only look at specific node: http://localhost:18088/ws/v1/cluster/scheduler/activities?nodeId=node-87:75 OR without port number http://localhost:18088/ws/v1/cluster/scheduler/activities?nodeId=node-87 look at activities for specific application within a period of time(3s in default): http://localhost:18088/ws/v1/cluster/scheduler/app-activities?appId=application_1468198570845_0022 http://localhost:18088/ws/v1/cluster/scheduler/app-activities?appId=application_1468198570845_0022=5.2 Test class: TestRMWebServicesCapacitySched.java org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched#testActivityJSON org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched#testAppActivityJSON Thanks for review. Please feel free to put forward any suggestions for improvements. > Improvement: Introduce more debug/diagnostics information to detail out > scheduler activity > -- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > YARN-4091-design-doc-v1.pdf, YARN-4091.preliminary.1.patch > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4091) Improvement: Introduce more debug/diagnostics information to detail out scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Ge updated YARN-4091: -- Attachment: (was: class description) > Improvement: Introduce more debug/diagnostics information to detail out > scheduler activity > -- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > YARN-4091-design-doc-v1.pdf, YARN-4091.preliminary.1.patch > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-5359:
--
Attachment: YARN-5359.03.patch
Posted patch v.4.
Addressed Joep's comments. Also, I found more places where the default storage
location is used for the filesystem writer/reader. They run the risk of
stepping on each other's foot when multiple unit tests are running possibly
from multiple builds. They have been fixed.
> FileSystemTimelineReader/Writer uses unix-specific default
> --
>
> Key: YARN-5359
> URL: https://issues.apache.org/jira/browse/YARN-5359
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Attachments: YARN-5359.01.patch, YARN-5359.02.patch,
> YARN-5359.03.patch
>
>
> {{FileSystemTimelineReaderImpl}} and {{FileSystemTimelineWriterImpl}} use a
> unix-specific default. It won't work on Windows.
> Also, {{TestFileSystemTimelineReaderImpl}} uses this default directly, which
> is also brittle against concurrent tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5367) HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials
[ https://issues.apache.org/jira/browse/YARN-5367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin updated YARN-5367: -- Attachment: YARN-5367.001.patch > HDFS delegation tokens in ApplicationSubmissionContext should be added to > systemCrednetials > --- > > Key: YARN-5367 > URL: https://issues.apache.org/jira/browse/YARN-5367 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Xianyin Xin >Assignee: Xianyin Xin > Attachments: YARN-5367.001.patch > > > App log aggregation may failed because of the below flow: > 0) suppose the token.max-lifetime is 7 days and renew interval is 1 day; > 1) start a long running job, like sparkJDBC, of which the AM acts as a > service. When submitting the job, HDFS token A in > ApplicationSubmissionContext will be added to DelegationTokenRenewer, but not > added to systemCredentials; > 2) after 1 day, submit a spark query. After received the query, AM will > request containers and start tasks. When start the containers, a new HDFS > token B is used; > 3) after 1 day, kill the job, when doing log aggregation, exception occurs > which show token B is not in the HDFS token cache so the connecting to HDFS > fails; > We should add token A to systemCredentials to make sure token A can be > delivered to NMs in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5353) ResourceManager can leak delegation tokens when they are shared across apps
[ https://issues.apache.org/jira/browse/YARN-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374222#comment-15374222 ] Hudson commented on YARN-5353: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10085 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10085/]) YARN-5353. ResourceManager can leak delegation tokens when they are (varunsaxena: rev 06c56ff79b4cdf82f733498d3edfa0b6e531a2db) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java > ResourceManager can leak delegation tokens when they are shared across apps > --- > > Key: YARN-5353 > URL: https://issues.apache.org/jira/browse/YARN-5353 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.0, 2.6.1 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.8.0, 2.6.5, 2.7.4 > > Attachments: YARN-5353.001.patch > > > Recently saw a ResourceManager go into heavy GC. Heap dump showed that there > were millions of delegation tokens on the heap. It looks like most of them > belonged to the appTokens map in DelegationTokenRenewer. When an app > completes and tokens are removed for it, I noticed that the appTokens entry > for the app is not cleaned up if tokens were shared with other active apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5367) HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials
Xianyin Xin created YARN-5367: - Summary: HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials Key: YARN-5367 URL: https://issues.apache.org/jira/browse/YARN-5367 Project: Hadoop YARN Issue Type: Bug Components: yarn Reporter: Xianyin Xin Assignee: Xianyin Xin App log aggregation may failed because of the below flow: 0) suppose the token.max-lifetime is 7 days and renew interval is 1 day; 1) start a long running job, like sparkJDBC, of which the AM acts as a service. When submitting the job, HDFS token A in ApplicationSubmissionContext will be added to DelegationTokenRenewer, but not added to systemCredentials; 2) after 1 day, submit a spark query. After received the query, AM will request containers and start tasks. When start the containers, a new HDFS token B is used; 3) after 1 day, kill the job, when doing log aggregation, exception occurs which show token B is not in the HDFS token cache so the connecting to HDFS fails; We should add token A to systemCredentials to make sure token A can be delivered to NMs in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5353) ResourceManager can leak delegation tokens when they are shared across apps
[ https://issues.apache.org/jira/browse/YARN-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374202#comment-15374202 ] Varun Saxena commented on YARN-5353: Committed this to trunk, branch-2, branch-2.8, branch-2.7 and branch-2.6 Thanks [~jlowe] for fixing the issue. > ResourceManager can leak delegation tokens when they are shared across apps > --- > > Key: YARN-5353 > URL: https://issues.apache.org/jira/browse/YARN-5353 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.0, 2.6.1 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.8.0, 2.6.5, 2.7.4 > > Attachments: YARN-5353.001.patch > > > Recently saw a ResourceManager go into heavy GC. Heap dump showed that there > were millions of delegation tokens on the heap. It looks like most of them > belonged to the appTokens map in DelegationTokenRenewer. When an app > completes and tokens are removed for it, I noticed that the appTokens entry > for the app is not cleaned up if tokens were shared with other active apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374197#comment-15374197 ] Ying Zhang commented on YARN-5287: -- sure, working on the test case right now. > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-naga.patch, YARN-5287.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374193#comment-15374193 ] Ying Zhang commented on YARN-5287: -- Thanks Naganarasimha. I've tested my change with 3-node cluster, and it works as expected. I've currently working on adding unit test, and I've found the current test-container-executor is not working when running as root (which is required for testing initialize app or run container), for example, under root, run "test-container-executor ". So I've made some minor change there. Have you encountered the same problem? > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-naga.patch, YARN-5287.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5366) Add support for toggling the removal of completed and failed docker containers
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf reassigned YARN-5366: - Assignee: Shane Kumpf > Add support for toggling the removal of completed and failed docker containers > -- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > > Currently, completed and failed docker containers are removed by > container-executor. Add a job level environment variable to > DockerLinuxContainerRuntime to allow the user to toggle whether they want the > container deleted or not and remove the logic from container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5366) Add support for toggling the removal of completed and failed docker containers
Shane Kumpf created YARN-5366: - Summary: Add support for toggling the removal of completed and failed docker containers Key: YARN-5366 URL: https://issues.apache.org/jira/browse/YARN-5366 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Shane Kumpf Currently, completed and failed docker containers are removed by container-executor. Add a job level environment variable to DockerLinuxContainerRuntime to allow the user to toggle whether they want the container deleted or not and remove the logic from container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5216) Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM
[
https://issues.apache.org/jira/browse/YARN-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374174#comment-15374174
]
Hitesh Sharma commented on YARN-5216:
-
We investigated a few approaches over here:
* Have a subclass of {{QueuingContainersManagerImpl}}: this approach has some
pros but the problem here is that subclassing just to override the preemption
behavior isn't the right thing to do.
* Having a pluggable policy in {{QueuingContainersManagerImpl}} requires
extension points to select which containers to run, run the container, preempt
the container, etc. This approach starts to get more complex as we look to add
support for PAUSED containers [YARN-5292].
Based on the feedback here and the discussions we have had, I'm looking into
adding support for PAUSED containers within {{QueuingContainersManagerImpl}}.
That would simplify things quite a bit and allow us to have a more pluggable
and cleaner design.
[~asuresh], [~kkaranasos], thanks for all the feedback!
> Expose configurable preemption policy for OPPORTUNISTIC containers running on
> the NM
>
>
> Key: YARN-5216
> URL: https://issues.apache.org/jira/browse/YARN-5216
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Hitesh Sharma
> Fix For: 2.9.0
>
> Attachments: YARN5216.001.patch, yarn5216.002.patch
>
>
> Currently, the default action taken by the QueuingContainerManager,
> introduced in YARN-2883, when a GUARANTEED Container is scheduled on an NM
> with OPPORTUNISTIC containers using up resources, is to KILL the running
> OPPORTUNISTIC containers.
> This JIRA proposes to expose a configurable hook to allow the NM to take a
> different action.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5339) passing file to -out for YARN log CLI doesnt give warning or error code
[
https://issues.apache.org/jira/browse/YARN-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374171#comment-15374171
]
Xuan Gong commented on YARN-5339:
-
Thanks for the review.
Uploaded a new patch to address all the comments. It depends on YARN-5200
> passing file to -out for YARN log CLI doesnt give warning or error code
> ---
>
> Key: YARN-5339
> URL: https://issues.apache.org/jira/browse/YARN-5339
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Sumana Sathish
>Assignee: Xuan Gong
> Attachments: YARN-5339.1.patch, YARN-5339.2.patch
>
>
> passing file to -out for YARN log CLI doesnt give warning or error code
> {code}
> yarn logs -applicationId application_1467117709224_0003 -out
> /grid/0/hadoopqe/artifacts/file.txt
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5339) passing file to -out for YARN log CLI doesnt give warning or error code
[
https://issues.apache.org/jira/browse/YARN-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated YARN-5339:
Attachment: YARN-5339.2.patch
> passing file to -out for YARN log CLI doesnt give warning or error code
> ---
>
> Key: YARN-5339
> URL: https://issues.apache.org/jira/browse/YARN-5339
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Sumana Sathish
>Assignee: Xuan Gong
> Attachments: YARN-5339.1.patch, YARN-5339.2.patch
>
>
> passing file to -out for YARN log CLI doesnt give warning or error code
> {code}
> yarn logs -applicationId application_1467117709224_0003 -out
> /grid/0/hadoopqe/artifacts/file.txt
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5200) Improve yarn logs to get Container List
[ https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374169#comment-15374169 ] Xuan Gong commented on YARN-5200: - The testcase failures and checkstyle warning are not related > Improve yarn logs to get Container List > --- > > Key: YARN-5200 > URL: https://issues.apache.org/jira/browse/YARN-5200 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5200.1.patch, YARN-5200.10.patch, > YARN-5200.11.patch, YARN-5200.12.patch, YARN-5200.2.patch, YARN-5200.3.patch, > YARN-5200.4.patch, YARN-5200.5.patch, YARN-5200.6.patch, YARN-5200.7.patch, > YARN-5200.8.patch, YARN-5200.9.patch, YARN-5200.9.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5362) TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
[
https://issues.apache.org/jira/browse/YARN-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374162#comment-15374162
]
sandflee commented on YARN-5362:
update a patch to add drainEvents() before asserts, there had a very little
race condition since drainEvents() just grant there are no event in event
queue, not grant event are processed completely.
> TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
> ---
>
> Key: YARN-5362
> URL: https://issues.apache.org/jira/browse/YARN-5362
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: sandflee
> Attachments: YARN-5362.01.patch
>
>
> Saw the following in a precommit build that only changed an unrelated unit
> test:
> {noformat}
> Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 101.265 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testFinishedAppRemovalAfterRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 0.411 sec <<< FAILURE!
> java.lang.AssertionError: expected null, but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1653)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5362) TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
[
https://issues.apache.org/jira/browse/YARN-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sandflee updated YARN-5362:
---
Attachment: YARN-5362.01.patch
> TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
> ---
>
> Key: YARN-5362
> URL: https://issues.apache.org/jira/browse/YARN-5362
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: sandflee
> Attachments: YARN-5362.01.patch
>
>
> Saw the following in a precommit build that only changed an unrelated unit
> test:
> {noformat}
> Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 101.265 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testFinishedAppRemovalAfterRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 0.411 sec <<< FAILURE!
> java.lang.AssertionError: expected null, but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1653)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4091) Improvement: Introduce more debug/diagnostics information to detail out scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Ge updated YARN-4091: -- Attachment: YARN-4091.preliminary.1.patch class description > Improvement: Introduce more debug/diagnostics information to detail out > scheduler activity > -- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > YARN-4091-design-doc-v1.pdf, YARN-4091.preliminary.1.patch, class description > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5364) timelineservice modules have indirect dependencies on mapreduce artifacts
[ https://issues.apache.org/jira/browse/YARN-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374139#comment-15374139 ] Naganarasimha G R commented on YARN-5364: - Thanks for the patch [~sjlee0], patch LGTM will commit it shortly ! > timelineservice modules have indirect dependencies on mapreduce artifacts > - > > Key: YARN-5364 > URL: https://issues.apache.org/jira/browse/YARN-5364 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Attachments: YARN-5364.01.patch > > > The new timelineservice and timelineservice-hbase-tests modules have indirect > dependencies to mapreduce artifacts through HBase and phoenix. Although it's > not causing builds to fail, it's not good hygiene. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4759) Revisit signalContainer() for docker containers
[
https://issues.apache.org/jira/browse/YARN-4759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374138#comment-15374138
]
Shane Kumpf commented on YARN-4759:
---
The two remaining checkstyle errors are because the package names are over 80
characters. Looking at other examples, they also have the same issue, so I
assume this can be ignored?
Also, the changes to container-executor are necessary because the exitcode file
is used in the container reacquisition process. Without these changes, the
exitcode file is not written as the NM user, and cannot be used during
recovery. Since the exitcode file lives in nmPrivate, ensuring the file is
written as the NM user seems appropriate.
Root privileges are also dropped after issuing the "docker" related commands.
Below is the exception without this change.
{code}
2016-07-12 17:32:59,831 ERROR
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch:
Unable to recover container container_1468357024753_0004_01_02
java.io.IOException: File
'/usr/local/src/hadoop_install/hadoop/tmp/yarn/nm-local-dir/nmPrivate/application_1468357024753_0004/container_1468357024753_0004_01_02/container_1468357024753_0004_01_02.pid.exitcode'
cannot be read
at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:296)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1711)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1748)
at
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.reacquireContainer(LinuxContainerExecutor.java:479)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:85)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:48)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)2016-07-12 17:32:59,831 ERROR
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch:
Unable to recover container container_1468357024753_0004_01_02
java.io.IOException: File
'/usr/local/src/hadoop_install/hadoop/tmp/yarn/nm-local-dir/nmPrivate/application_1468357024753_0004/container_1468357024753_0004_01_02/container_1468357024753_0004_01_02.pid.exitcode'
cannot be read
at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:296)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1711)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1748)
at
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.reacquireContainer(LinuxContainerExecutor.java:479)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:85)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:48)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> Revisit signalContainer() for docker containers
> ---
>
> Key: YARN-4759
> URL: https://issues.apache.org/jira/browse/YARN-4759
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Shane Kumpf
> Attachments: YARN-4759.001.patch, YARN-4759.002.patch
>
>
> The current signal handling (in the DockerContainerRuntime) needs to be
> revisited for docker containers. For example, container reacquisition on NM
> restart might not work, depending on which user the process in the container
> runs as.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[
https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374136#comment-15374136
]
Naganarasimha G R commented on YARN-5287:
-
further {{create_directory_for_user}} also uses the same approach...
> LinuxContainerExecutor fails to set proper permission
> -
>
> Key: YARN-5287
> URL: https://issues.apache.org/jira/browse/YARN-5287
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.2
>Reporter: Ying Zhang
>Assignee: Ying Zhang
>Priority: Minor
> Attachments: YARN-5287-naga.patch, YARN-5287.001.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> LinuxContainerExecutor fails to set the proper permissions on the local
> directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster
> has been configured with a restrictive umask, e.g.: umask 077. Job failed due
> to the following reason:
> Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has
> permission 700 but needs permission 750
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-4091) Improvement: Introduce more debug/diagnostics information to detail out scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Ge reassigned YARN-4091: - Assignee: Chen Ge (was: Sunil G) > Improvement: Introduce more debug/diagnostics information to detail out > scheduler activity > -- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > YARN-4091-design-doc-v1.pdf > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4091) Improvement: Introduce more debug/diagnostics information to detail out scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Ge updated YARN-4091: -- Assignee: Sunil G (was: Chen Ge) > Improvement: Introduce more debug/diagnostics information to detail out > scheduler activity > -- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > YARN-4091-design-doc-v1.pdf > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-5287: Attachment: YARN-5287-naga.patch [~Ying Zhang], This was the approach which we had taken to solve the same issue but seems like it would be better to solve the issue based on the approach taken by you as it will get executed only in the failure scenario. Can you check any test case can be added for the same ? > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Minor > Attachments: YARN-5287-naga.patch, YARN-5287.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4759) Revisit signalContainer() for docker containers
[
https://issues.apache.org/jira/browse/YARN-4759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374132#comment-15374132
]
Hadoop QA commented on YARN-4759:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 18 unchanged - 0 fixed = 20 total (was 18) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 1s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 37s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817564/YARN-4759.002.patch |
| JIRA Issue | YARN-4759 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux b264ba90bfaa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a290a98 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12299/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12299/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12299/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Revisit signalContainer() for docker containers
>
[jira] [Assigned] (YARN-4091) Improvement: Introduce more debug/diagnostics information to detail out scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Ge reassigned YARN-4091: - Assignee: Chen Ge (was: Sunil G) > Improvement: Introduce more debug/diagnostics information to detail out > scheduler activity > -- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > YARN-4091-design-doc-v1.pdf > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5364) timelineservice modules have indirect dependencies on mapreduce artifacts
[ https://issues.apache.org/jira/browse/YARN-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374098#comment-15374098 ] Joep Rottinghuis commented on YARN-5364: LGTM. > timelineservice modules have indirect dependencies on mapreduce artifacts > - > > Key: YARN-5364 > URL: https://issues.apache.org/jira/browse/YARN-5364 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Attachments: YARN-5364.01.patch > > > The new timelineservice and timelineservice-hbase-tests modules have indirect > dependencies to mapreduce artifacts through HBase and phoenix. Although it's > not causing builds to fail, it's not good hygiene. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374097#comment-15374097
]
Joep Rottinghuis commented on YARN-5359:
Patch looks good.
There may be one additional unit test using Linux style paths:
TestFileSystemTimelineWriterImpl#testWriteEntityToFile
Around line 87
{code}
String fileName = fsi.getOutputRoot() +
"/entities/cluster_id/user_id/flow_name/flow_version/12345678/" +
"app_id/" + type + "/" + id +
FileSystemTimelineWriterImpl.TIMELINE_SERVICE_STORAGE_EXTENSION;
Path path = Paths.get(fileName);
File f = new File(fileName);
assertTrue(f.exists() && !f.isDirectory());
{code}
Similar around line 102
{code}
String fileName2 = fsi.getOutputRoot() +
"/entities/cluster_id/user_id/flow_name/flow_version/12345678/app_id/"
+ type2 + "/" + id2 +
FileSystemTimelineWriterImpl.TIMELINE_SERVICE_STORAGE_EXTENSION;
{code}
> FileSystemTimelineReader/Writer uses unix-specific default
> --
>
> Key: YARN-5359
> URL: https://issues.apache.org/jira/browse/YARN-5359
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Attachments: YARN-5359.01.patch, YARN-5359.02.patch
>
>
> {{FileSystemTimelineReaderImpl}} and {{FileSystemTimelineWriterImpl}} use a
> unix-specific default. It won't work on Windows.
> Also, {{TestFileSystemTimelineReaderImpl}} uses this default directly, which
> is also brittle against concurrent tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5364) timelineservice modules have indirect dependencies on mapreduce artifacts
[
https://issues.apache.org/jira/browse/YARN-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374093#comment-15374093
]
Hadoop QA commented on YARN-5364:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 55s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 45s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 6s
{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 31s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817557/YARN-5364.01.patch |
| JIRA Issue | YARN-5364 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux d1a088cb154d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a290a98 |
| Default Java | 1.8.0_91 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12298/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12298/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> timelineservice modules have indirect dependencies on mapreduce artifacts
> -
>
> Key: YARN-5364
> URL: https://issues.apache.org/jira/browse/YARN-5364
> Project:
[jira] [Updated] (YARN-4759) Revisit signalContainer() for docker containers
[ https://issues.apache.org/jira/browse/YARN-4759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf updated YARN-4759: -- Attachment: YARN-4759.002.patch > Revisit signalContainer() for docker containers > --- > > Key: YARN-4759 > URL: https://issues.apache.org/jira/browse/YARN-4759 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Shane Kumpf > Attachments: YARN-4759.001.patch, YARN-4759.002.patch > > > The current signal handling (in the DockerContainerRuntime) needs to be > revisited for docker containers. For example, container reacquisition on NM > restart might not work, depending on which user the process in the container > runs as. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5364) timelineservice modules have indirect dependencies on mapreduce artifacts
[ https://issues.apache.org/jira/browse/YARN-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374068#comment-15374068 ] Sangjin Lee commented on YARN-5364: --- Not sure why the unit test failed. It's certainly unrelated. Kicking off the jenkins build one more time. > timelineservice modules have indirect dependencies on mapreduce artifacts > - > > Key: YARN-5364 > URL: https://issues.apache.org/jira/browse/YARN-5364 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Attachments: YARN-5364.01.patch > > > The new timelineservice and timelineservice-hbase-tests modules have indirect > dependencies to mapreduce artifacts through HBase and phoenix. Although it's > not causing builds to fail, it's not good hygiene. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5364) timelineservice modules have indirect dependencies on mapreduce artifacts
[
https://issues.apache.org/jira/browse/YARN-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374041#comment-15374041
]
Hadoop QA commented on YARN-5364:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 44s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 56s {color}
| {color:red} hadoop-yarn-server-timelineservice-hbase-tests in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817557/YARN-5364.01.patch |
| JIRA Issue | YARN-5364 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 2588a1932efe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a290a98 |
| Default Java | 1.8.0_91 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12297/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12297/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12297/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
[jira] [Commented] (YARN-5200) Improve yarn logs to get Container List
[
https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374012#comment-15374012
]
Hadoop QA commented on YARN-5200:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 12
new + 90 unchanged - 14 fixed = 102 total (was 104) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 34s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 37s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 54s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
| | hadoop.yarn.client.cli.TestLogsCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817554/YARN-5200.12.patch |
| JIRA Issue | YARN-5200 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 805e516de0d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d5f5a64 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12296/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12296/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
[jira] [Updated] (YARN-5356) ResourceUtilization should also include resource availability
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated YARN-5356: -- Attachment: YARN-5356.000.patch First proposal for sending physical resources in the node to the RM. > ResourceUtilization should also include resource availability > - > > Key: YARN-5356 > URL: https://issues.apache.org/jira/browse/YARN-5356 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Nathan Roberts >Assignee: Inigo Goiri > Attachments: YARN-5356.000.patch > > > Currently ResourceUtilization contains absolute quantities of resource used > (e.g. 4096MB memory used). It would be good if it also included how much of > that resource is actually available on the node so that the RM can use this > data to schedule more effectively (overcommit, etc) > Currently the only available information is the Resource the node registered > with (or later updated using updateNodeResource). However, these aren't > really sufficient to get a good view of how utilized a resource is. For > example, if a node reports 400% CPU utilization, does that mean it's > completely full, or barely utilized? Today there is no reliable way to figure > this out. > [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you > have thoughts/opinions on this? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5356) ResourceUtilization should also include resource availability
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri reassigned YARN-5356: - Assignee: Inigo Goiri > ResourceUtilization should also include resource availability > - > > Key: YARN-5356 > URL: https://issues.apache.org/jira/browse/YARN-5356 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Nathan Roberts >Assignee: Inigo Goiri > Attachments: YARN-5356.000.patch > > > Currently ResourceUtilization contains absolute quantities of resource used > (e.g. 4096MB memory used). It would be good if it also included how much of > that resource is actually available on the node so that the RM can use this > data to schedule more effectively (overcommit, etc) > Currently the only available information is the Resource the node registered > with (or later updated using updateNodeResource). However, these aren't > really sufficient to get a good view of how utilized a resource is. For > example, if a node reports 400% CPU utilization, does that mean it's > completely full, or barely utilized? Today there is no reliable way to figure > this out. > [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you > have thoughts/opinions on this? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5364) timelineservice modules have indirect dependencies on mapreduce artifacts
[ https://issues.apache.org/jira/browse/YARN-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-5364: -- Attachment: YARN-5364.01.patch Posted patch v.1. Excluded all the undesirable indirect dependencies from these modules. > timelineservice modules have indirect dependencies on mapreduce artifacts > - > > Key: YARN-5364 > URL: https://issues.apache.org/jira/browse/YARN-5364 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Attachments: YARN-5364.01.patch > > > The new timelineservice and timelineservice-hbase-tests modules have indirect > dependencies to mapreduce artifacts through HBase and phoenix. Although it's > not causing builds to fail, it's not good hygiene. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5362) TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
[
https://issues.apache.org/jira/browse/YARN-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373958#comment-15373958
]
sandflee commented on YARN-5362:
seen many test failures related to RMApp/RMAppattempt comes to some state but
some event are not processed in rm event queue or scheduler event queue, cause
test failure, seems we could implicitly invokes drainEvents(should also drain
sheduler event) in some mockRM method like waitForState, thought? cc [~sunilg]
[~rohithsharma]
{code}
void waitForState() {
drainEvents();
}
{code}
> TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
> ---
>
> Key: YARN-5362
> URL: https://issues.apache.org/jira/browse/YARN-5362
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: sandflee
>
> Saw the following in a precommit build that only changed an unrelated unit
> test:
> {noformat}
> Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 101.265 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testFinishedAppRemovalAfterRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 0.411 sec <<< FAILURE!
> java.lang.AssertionError: expected null, but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1653)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5200) Improve yarn logs to get Container List
[ https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373941#comment-15373941 ] Xuan Gong commented on YARN-5200: - Uploaded a v12 patch to fix extra line problem, and do the null check for the node http address. > Improve yarn logs to get Container List > --- > > Key: YARN-5200 > URL: https://issues.apache.org/jira/browse/YARN-5200 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5200.1.patch, YARN-5200.10.patch, > YARN-5200.11.patch, YARN-5200.12.patch, YARN-5200.2.patch, YARN-5200.3.patch, > YARN-5200.4.patch, YARN-5200.5.patch, YARN-5200.6.patch, YARN-5200.7.patch, > YARN-5200.8.patch, YARN-5200.9.patch, YARN-5200.9.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5200) Improve yarn logs to get Container List
[ https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5200: Attachment: YARN-5200.12.patch > Improve yarn logs to get Container List > --- > > Key: YARN-5200 > URL: https://issues.apache.org/jira/browse/YARN-5200 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5200.1.patch, YARN-5200.10.patch, > YARN-5200.11.patch, YARN-5200.12.patch, YARN-5200.2.patch, YARN-5200.3.patch, > YARN-5200.4.patch, YARN-5200.5.patch, YARN-5200.6.patch, YARN-5200.7.patch, > YARN-5200.8.patch, YARN-5200.9.patch, YARN-5200.9.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5362) TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
[
https://issues.apache.org/jira/browse/YARN-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373927#comment-15373927
]
sandflee commented on YARN-5362:
this could simple reproduced by add a sleep to RMAppManager
{code}
case APP_COMPLETED:
{
try {
Thread.sleep(2000);
} catch(InterruptedException e){}
finishApplication(applicationId);
logApplicationSummary(applicationId);
checkAppNumCompletedLimit();
}
{code}
APP_COMPLETED is processed async, and could simple be fixed by add
mockRM#drainEvents() before calling asserts.
> TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
> ---
>
> Key: YARN-5362
> URL: https://issues.apache.org/jira/browse/YARN-5362
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: sandflee
>
> Saw the following in a precommit build that only changed an unrelated unit
> test:
> {noformat}
> Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 101.265 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testFinishedAppRemovalAfterRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 0.411 sec <<< FAILURE!
> java.lang.AssertionError: expected null, but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1653)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5365) Add support for YARN Shared Cache
Ming Ma created YARN-5365: - Summary: Add support for YARN Shared Cache Key: YARN-5365 URL: https://issues.apache.org/jira/browse/YARN-5365 Project: Hadoop YARN Issue Type: Improvement Reporter: Ming Ma YARN provides shared cache in functionality YARN-1492. According to [~ctrezzo] most of the YARN functionality is in hadoop 2.8 and frameworks can start to use it. MR adds the support via MAPREDUCE-5951. Can anyone confirm if Tez supports the upload of application DAG jar and dependent lib jars from client machine to HDFS as part of Tez app submission? From my test, that doesn't seem to happen. Tez expects applications to upload the jars to HDFS beforehand and then set the tez.aux.uris to the HDFS locations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4366) Fix Lint Warnings in YARN Common
[
https://issues.apache.org/jira/browse/YARN-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373843#comment-15373843
]
Hudson commented on YARN-4366:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #10081 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10081/])
YARN-4366. Fix Lint Warnings in YARN Common (templedf via rkanter) (rkanter:
rev bf6f4a3b980a07d0b268eeb984a649a362877734)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java
> Fix Lint Warnings in YARN Common
>
>
> Key: YARN-4366
> URL: https://issues.apache.org/jira/browse/YARN-4366
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Fix For: 2.9.0
>
> Attachments: YARN-4366.001.patch
>
>
> {noformat}
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java:[100,45]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Class for a varargs call
> cast to java.lang.Class[] for a non-varargs call and to suppress this
> warning
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[62,46]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Class for a varargs call
> cast to java.lang.Class[] for a non-varargs call and to suppress this
> warning
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[64,34]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Object for a varargs call
> cast to java.lang.Object[] for a non-varargs call and to suppress this
> warning
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5362) TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
[
https://issues.apache.org/jira/browse/YARN-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sandflee reassigned YARN-5362:
--
Assignee: sandflee
> TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
> ---
>
> Key: YARN-5362
> URL: https://issues.apache.org/jira/browse/YARN-5362
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: sandflee
>
> Saw the following in a precommit build that only changed an unrelated unit
> test:
> {noformat}
> Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 101.265 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testFinishedAppRemovalAfterRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
> Time elapsed: 0.411 sec <<< FAILURE!
> java.lang.AssertionError: expected null, but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1653)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5356) ResourceUtilization should also include resource availability
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373830#comment-15373830 ] Inigo Goiri commented on YARN-5356: --- It looks like there is some work we may want to leverage in YARN-4081 where they have YARN-4081 to add multiple resources. I'd like to have some feedback from [~vvasudev] and [~asuresh] about their opinion on this. > ResourceUtilization should also include resource availability > - > > Key: YARN-5356 > URL: https://issues.apache.org/jira/browse/YARN-5356 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Nathan Roberts > > Currently ResourceUtilization contains absolute quantities of resource used > (e.g. 4096MB memory used). It would be good if it also included how much of > that resource is actually available on the node so that the RM can use this > data to schedule more effectively (overcommit, etc) > Currently the only available information is the Resource the node registered > with (or later updated using updateNodeResource). However, these aren't > really sufficient to get a good view of how utilized a resource is. For > example, if a node reports 400% CPU utilization, does that mean it's > completely full, or barely utilized? Today there is no reliable way to figure > this out. > [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you > have thoughts/opinions on this? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4366) Fix Lint Warnings in YARN Common
[
https://issues.apache.org/jira/browse/YARN-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter resolved YARN-4366.
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
Thanks [~templedf]. Committed to trunk and branch-2!
> Fix Lint Warnings in YARN Common
>
>
> Key: YARN-4366
> URL: https://issues.apache.org/jira/browse/YARN-4366
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Fix For: 2.9.0
>
> Attachments: YARN-4366.001.patch
>
>
> {noformat}
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java:[100,45]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Class for a varargs call
> cast to java.lang.Class[] for a non-varargs call and to suppress this
> warning
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[62,46]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Class for a varargs call
> cast to java.lang.Class[] for a non-varargs call and to suppress this
> warning
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[64,34]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Object for a varargs call
> cast to java.lang.Object[] for a non-varargs call and to suppress this
> warning
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5317) testAMRestartNotLostContainerCompleteMsg may fail
[ https://issues.apache.org/jira/browse/YARN-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373785#comment-15373785 ] sandflee commented on YARN-5317: thanks sunil and jason for review and commit! > testAMRestartNotLostContainerCompleteMsg may fail > - > > Key: YARN-5317 > URL: https://issues.apache.org/jira/browse/YARN-5317 > Project: Hadoop YARN > Issue Type: Test >Reporter: sandflee >Assignee: sandflee >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-5317.01.patch, YARN-5317.02.patch > > > java.lang.Exception: test timed out after 3 milliseconds > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:261) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:225) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:207) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:746) > at > org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testAMRestartNotLostContainerCompleteMsg(TestAMRestart.java:841) > see > https://builds.apache.org/job/PreCommit-YARN-Build/12204/testReport/org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager/TestAMRestart/testAMRestartNotLostContainerCompleteMsg/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5339) passing file to -out for YARN log CLI doesnt give warning or error code
[
https://issues.apache.org/jira/browse/YARN-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373751#comment-15373751
]
Vinod Kumar Vavilapalli commented on YARN-5339:
---
Looks good, couple of nits
- Shall we not print the help message? It is too long and making it hard to
read the actual error message - let's just print the error message and exit.
- Can you split the new test-code into its own test-case? Each of the existing
tests are becoming too long.
> passing file to -out for YARN log CLI doesnt give warning or error code
> ---
>
> Key: YARN-5339
> URL: https://issues.apache.org/jira/browse/YARN-5339
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Sumana Sathish
>Assignee: Xuan Gong
> Attachments: YARN-5339.1.patch
>
>
> passing file to -out for YARN log CLI doesnt give warning or error code
> {code}
> yarn logs -applicationId application_1467117709224_0003 -out
> /grid/0/hadoopqe/artifacts/file.txt
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5363) For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default
[
https://issues.apache.org/jira/browse/YARN-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-5363:
--
Attachment: YARN-5363-2016-07-12.txt
Here's a straightforward patch - based on top of YARN-3200.
- Changes the default to be {{.*}} instead of {{syslog}} for all containers
(including AM containers - YARN-3347.
- Also renamed the inaptly named {{logFiles}} option to be {{log_files}} -
naming CLI options in camelCase is not the norm.
- Added a mock test - we are missing real tests for logs of running
applications.
> For AM containers, or for containers of running-apps, "yarn logs" incorrectly
> only (tries to) shows syslog file-type by default
> ---
>
> Key: YARN-5363
> URL: https://issues.apache.org/jira/browse/YARN-5363
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
> Attachments: YARN-5363-2016-07-12.txt
>
>
> For e.g, for a running application, the following happens:
> {code}
> # yarn logs -applicationId application_1467838922593_0001
> 16/07/06 22:07:05 INFO impl.TimelineClientImpl: Timeline service address:
> http://:8188/ws/v1/timeline/
> 16/07/06 22:07:06 INFO client.RMProxy: Connecting to ResourceManager at
> /:8050
> 16/07/06 22:07:07 INFO impl.TimelineClientImpl: Timeline service address:
> http://l:8188/ws/v1/timeline/
> 16/07/06 22:07:07 INFO client.RMProxy: Connecting to ResourceManager at
> /:8050
> Can not find any log file matching the pattern: [syslog] for the container:
> container_e03_1467838922593_0001_01_01 within the application:
> application_1467838922593_0001
> Can not find any log file matching the pattern: [syslog] for the container:
> container_e03_1467838922593_0001_01_02 within the application:
> application_1467838922593_0001
> Can not find any log file matching the pattern: [syslog] for the container:
> container_e03_1467838922593_0001_01_03 within the application:
> application_1467838922593_0001
> Can not find any log file matching the pattern: [syslog] for the container:
> container_e03_1467838922593_0001_01_04 within the application:
> application_1467838922593_0001
> Can not find any log file matching the pattern: [syslog] for the container:
> container_e03_1467838922593_0001_01_05 within the application:
> application_1467838922593_0001
> Can not find any log file matching the pattern: [syslog] for the container:
> container_e03_1467838922593_0001_01_06 within the application:
> application_1467838922593_0001
> Can not find any log file matching the pattern: [syslog] for the container:
> container_e03_1467838922593_0001_01_07 within the application:
> application_1467838922593_0001
> Can not find the logs for the application: application_1467838922593_0001
> with the appOwner:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5339) passing file to -out for YARN log CLI doesnt give warning or error code
[
https://issues.apache.org/jira/browse/YARN-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumana Sathish updated YARN-5339:
-
Description:
passing file to -out for YARN log CLI doesnt give warning or error code
{code}
yarn logs -applicationId application_1467117709224_0003 -out
/grid/0/hadoopqe/artifacts/file.txt
{code}
was:
passing file to -out for YARN log CLI doesnt give warning or error code
{code}
/usr/hdp/current/hadoop-yarn-client/bin/yarn logs -applicationId
application_1467117709224_0003 -out /grid/0/hadoopqe/artifacts/file.txt
{code}
> passing file to -out for YARN log CLI doesnt give warning or error code
> ---
>
> Key: YARN-5339
> URL: https://issues.apache.org/jira/browse/YARN-5339
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Sumana Sathish
>Assignee: Xuan Gong
> Attachments: YARN-5339.1.patch
>
>
> passing file to -out for YARN log CLI doesnt give warning or error code
> {code}
> yarn logs -applicationId application_1467117709224_0003 -out
> /grid/0/hadoopqe/artifacts/file.txt
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5317) testAMRestartNotLostContainerCompleteMsg may fail
[ https://issues.apache.org/jira/browse/YARN-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373683#comment-15373683 ] Hudson commented on YARN-5317: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10080 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10080/]) YARN-5317. testAMRestartNotLostContainerCompleteMsg may fail. (jlowe: rev 10b704c5946afe7bfd4a6be40192ce7ca745d817) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java > testAMRestartNotLostContainerCompleteMsg may fail > - > > Key: YARN-5317 > URL: https://issues.apache.org/jira/browse/YARN-5317 > Project: Hadoop YARN > Issue Type: Test >Reporter: sandflee >Assignee: sandflee >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-5317.01.patch, YARN-5317.02.patch > > > java.lang.Exception: test timed out after 3 milliseconds > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:261) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:225) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:207) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:746) > at > org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testAMRestartNotLostContainerCompleteMsg(TestAMRestart.java:841) > see > https://builds.apache.org/job/PreCommit-YARN-Build/12204/testReport/org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager/TestAMRestart/testAMRestartNotLostContainerCompleteMsg/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5364) timelineservice modules have indirect dependencies on mapreduce artifacts
Sangjin Lee created YARN-5364: - Summary: timelineservice modules have indirect dependencies on mapreduce artifacts Key: YARN-5364 URL: https://issues.apache.org/jira/browse/YARN-5364 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 3.0.0-alpha1 Reporter: Sangjin Lee Assignee: Sangjin Lee Priority: Minor The new timelineservice and timelineservice-hbase-tests modules have indirect dependencies to mapreduce artifacts through HBase and phoenix. Although it's not causing builds to fail, it's not good hygiene. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4366) Fix Lint Warnings in YARN Common
[
https://issues.apache.org/jira/browse/YARN-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373665#comment-15373665
]
Robert Kanter commented on YARN-4366:
-
+1
> Fix Lint Warnings in YARN Common
>
>
> Key: YARN-4366
> URL: https://issues.apache.org/jira/browse/YARN-4366
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-4366.001.patch
>
>
> {noformat}
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java:[100,45]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Class for a varargs call
> cast to java.lang.Class[] for a non-varargs call and to suppress this
> warning
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[62,46]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Class for a varargs call
> cast to java.lang.Class[] for a non-varargs call and to suppress this
> warning
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[64,34]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Object for a varargs call
> cast to java.lang.Object[] for a non-varargs call and to suppress this
> warning
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-867) Isolation of failures in aux services
[ https://issues.apache.org/jira/browse/YARN-867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373658#comment-15373658 ] Ming Ma commented on YARN-867: -- Will this be simplified if we have YARN-1593? > Isolation of failures in aux services > -- > > Key: YARN-867 > URL: https://issues.apache.org/jira/browse/YARN-867 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Hitesh Shah >Assignee: Xuan Gong > Attachments: YARN-867.1.sampleCode.patch, YARN-867.3.patch, > YARN-867.4.patch, YARN-867.5.patch, YARN-867.6.patch, > YARN-867.sampleCode.2.patch > > > Today, a malicious application can bring down the NM by sending bad data to a > service. For example, sending data to the ShuffleService such that it results > any non-IOException will cause the NM's async dispatcher to exit as the > service's INIT APP event is not handled properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5363) For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default
Vinod Kumar Vavilapalli created YARN-5363:
-
Summary: For AM containers, or for containers of running-apps,
"yarn logs" incorrectly only (tries to) shows syslog file-type by default
Key: YARN-5363
URL: https://issues.apache.org/jira/browse/YARN-5363
Project: Hadoop YARN
Issue Type: Sub-task
Components: log-aggregation
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
For e.g, for a running application, the following happens:
{code}
# yarn logs -applicationId application_1467838922593_0001
16/07/06 22:07:05 INFO impl.TimelineClientImpl: Timeline service address:
http://:8188/ws/v1/timeline/
16/07/06 22:07:06 INFO client.RMProxy: Connecting to ResourceManager at
/:8050
16/07/06 22:07:07 INFO impl.TimelineClientImpl: Timeline service address:
http://l:8188/ws/v1/timeline/
16/07/06 22:07:07 INFO client.RMProxy: Connecting to ResourceManager at
/:8050
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_01 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_02 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_03 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_04 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_05 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_06 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_07 within the application:
application_1467838922593_0001
Can not find the logs for the application: application_1467838922593_0001 with
the appOwner:
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4904) YARN Log tooling enhancements
[ https://issues.apache.org/jira/browse/YARN-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-4904: -- Summary: YARN Log tooling enhancements (was: YARN Log tooling enhancement) > YARN Log tooling enhancements > - > > Key: YARN-4904 > URL: https://issues.apache.org/jira/browse/YARN-4904 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Xuan Gong >Assignee: Xuan Gong > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4759) Revisit signalContainer() for docker containers
[ https://issues.apache.org/jira/browse/YARN-4759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373617#comment-15373617 ] Shane Kumpf commented on YARN-4759: --- working on a new patch to fix cc and checkstyle issues. > Revisit signalContainer() for docker containers > --- > > Key: YARN-4759 > URL: https://issues.apache.org/jira/browse/YARN-4759 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Shane Kumpf > Attachments: YARN-4759.001.patch > > > The current signal handling (in the DockerContainerRuntime) needs to be > revisited for docker containers. For example, container reacquisition on NM > restart might not work, depending on which user the process in the container > runs as. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5200) Improve yarn logs to get Container List
[ https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373606#comment-15373606 ] Xuan Gong commented on YARN-5200: - The testcase failures are not related, and we already have related jira to track them. The checkstyle issues are not related, either. > Improve yarn logs to get Container List > --- > > Key: YARN-5200 > URL: https://issues.apache.org/jira/browse/YARN-5200 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5200.1.patch, YARN-5200.10.patch, > YARN-5200.11.patch, YARN-5200.2.patch, YARN-5200.3.patch, YARN-5200.4.patch, > YARN-5200.5.patch, YARN-5200.6.patch, YARN-5200.7.patch, YARN-5200.8.patch, > YARN-5200.9.patch, YARN-5200.9.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373599#comment-15373599
]
Hadoop QA commented on YARN-5359:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice:
The patch generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 45s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 12s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817503/YARN-5359.02.patch |
| JIRA Issue | YARN-5359 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e82348f398a5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7705812 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12295/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12295/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> FileSystemTimelineReader/Writer uses unix-specific default
> --
>
> Key: YARN-5359
> URL: https://issues.apache.org/jira/browse/YARN-5359
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>
[jira] [Commented] (YARN-5317) testAMRestartNotLostContainerCompleteMsg may fail
[ https://issues.apache.org/jira/browse/YARN-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373598#comment-15373598 ] Jason Lowe commented on YARN-5317: -- +1 lgtm. Filed YARN-5362 for the unrelated TestRMRestart failure. Committing this. > testAMRestartNotLostContainerCompleteMsg may fail > - > > Key: YARN-5317 > URL: https://issues.apache.org/jira/browse/YARN-5317 > Project: Hadoop YARN > Issue Type: Test >Reporter: sandflee >Assignee: sandflee >Priority: Minor > Attachments: YARN-5317.01.patch, YARN-5317.02.patch > > > java.lang.Exception: test timed out after 3 milliseconds > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:261) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:225) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:207) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:746) > at > org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testAMRestartNotLostContainerCompleteMsg(TestAMRestart.java:841) > see > https://builds.apache.org/job/PreCommit-YARN-Build/12204/testReport/org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager/TestAMRestart/testAMRestartNotLostContainerCompleteMsg/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5362) TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
Jason Lowe created YARN-5362:
Summary: TestRMRestart#testFinishedAppRemovalAfterRMRestart can
fail
Key: YARN-5362
URL: https://issues.apache.org/jira/browse/YARN-5362
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jason Lowe
Saw the following in a precommit build that only changed an unrelated unit test:
{noformat}
Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 101.265 sec
<<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
testFinishedAppRemovalAfterRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 0.411 sec <<< FAILURE!
java.lang.AssertionError: expected null, but
was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotNull(Assert.java:664)
at org.junit.Assert.assertNull(Assert.java:646)
at org.junit.Assert.assertNull(Assert.java:656)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1653)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4759) Revisit signalContainer() for docker containers
[
https://issues.apache.org/jira/browse/YARN-4759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373588#comment-15373588
]
Hadoop QA commented on YARN-4759:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 25s {color} |
{color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 10 new + 18 unchanged - 0 fixed = 28 total (was 18)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 1s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 40s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817502/YARN-4759.001.patch |
| JIRA Issue | YARN-4759 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux b3a6f98f0b6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7705812 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/12294/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12294/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12294/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
[jira] [Commented] (YARN-5200) Improve yarn logs to get Container List
[
https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373584#comment-15373584
]
Hadoop QA commented on YARN-5200:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 13
new + 91 unchanged - 13 fixed = 104 total (was 104) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 14s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 31s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
| | hadoop.yarn.client.cli.TestLogsCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817497/YARN-5200.11.patch |
| JIRA Issue | YARN-5200 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 52b5f7c0c250 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7705812 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12293/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12293/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
[jira] [Updated] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-5359:
--
Attachment: YARN-5359.02.patch
Posted patch v.2 to address the checkstyle complaint.
> FileSystemTimelineReader/Writer uses unix-specific default
> --
>
> Key: YARN-5359
> URL: https://issues.apache.org/jira/browse/YARN-5359
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Attachments: YARN-5359.01.patch, YARN-5359.02.patch
>
>
> {{FileSystemTimelineReaderImpl}} and {{FileSystemTimelineWriterImpl}} use a
> unix-specific default. It won't work on Windows.
> Also, {{TestFileSystemTimelineReaderImpl}} uses this default directly, which
> is also brittle against concurrent tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4393) TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently
[
https://issues.apache.org/jira/browse/YARN-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-4393:
-
Fix Version/s: (was: 2.9.0)
2.7.4
2.6.5
2.8.0
Thanks [~varun_saxena]! I committed this also to branch-2.8, branch-2.7 and
branch-2.6 since this was originally broken when YARN-90 went into branch-2.6.
> TestResourceLocalizationService#testFailedDirsResourceRelease fails
> intermittently
> --
>
> Key: YARN-4393
> URL: https://issues.apache.org/jira/browse/YARN-4393
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Affects Versions: 2.7.1
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Labels: test
> Fix For: 2.8.0, 2.6.5, 2.7.4
>
> Attachments: YARN-4393.01.patch
>
>
> [~ozawa] pointed out this failure on YARN-4380.
> Check
> https://issues.apache.org/jira/browse/YARN-4380?focusedCommentId=15023773=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15023773
> {noformat}
> sts run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.518 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
> testFailedDirsResourceRelease(org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService)
> Time elapsed: 0.093 sec <<< FAILURE!
> org.mockito.exceptions.verification.junit.ArgumentsAreDifferent:
> Argument(s) are different! Wanted:
> eventHandler.handle(
>
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService.testFailedDirsResourceRelease(TestResourceLocalizationService.java:2632)
> Actual invocation has different arguments:
> eventHandler.handle(
> EventType: APPLICATION_INITED
> );
> -> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService.testFailedDirsResourceRelease(TestResourceLocalizationService.java:2632)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5200) Improve yarn logs to get Container List
[
https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373544#comment-15373544
]
Hadoop QA commented on YARN-5200:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 36s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 13
new + 91 unchanged - 13 fixed = 104 total (was 104) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 28s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 41s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 4s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.cli.TestLogsCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817486/YARN-5200.10.patch |
| JIRA Issue | YARN-5200 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a8aabe72c699 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7705812 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12292/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12292/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
[jira] [Updated] (YARN-4759) Revisit signalContainer() for docker containers
[ https://issues.apache.org/jira/browse/YARN-4759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf updated YARN-4759: -- Attachment: YARN-4759.001.patch > Revisit signalContainer() for docker containers > --- > > Key: YARN-4759 > URL: https://issues.apache.org/jira/browse/YARN-4759 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Shane Kumpf > Attachments: YARN-4759.001.patch > > > The current signal handling (in the DockerContainerRuntime) needs to be > revisited for docker containers. For example, container reacquisition on NM > restart might not work, depending on which user the process in the container > runs as. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5200) Improve yarn logs to get Container List
[ https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373489#comment-15373489 ] Xuan Gong commented on YARN-5200: - sigh.. v10 is the wrong patch.. Uploaded v11 which is the correct one. > Improve yarn logs to get Container List > --- > > Key: YARN-5200 > URL: https://issues.apache.org/jira/browse/YARN-5200 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5200.1.patch, YARN-5200.10.patch, > YARN-5200.11.patch, YARN-5200.2.patch, YARN-5200.3.patch, YARN-5200.4.patch, > YARN-5200.5.patch, YARN-5200.6.patch, YARN-5200.7.patch, YARN-5200.8.patch, > YARN-5200.9.patch, YARN-5200.9.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5200) Improve yarn logs to get Container List
[ https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5200: Attachment: YARN-5200.11.patch > Improve yarn logs to get Container List > --- > > Key: YARN-5200 > URL: https://issues.apache.org/jira/browse/YARN-5200 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5200.1.patch, YARN-5200.10.patch, > YARN-5200.11.patch, YARN-5200.2.patch, YARN-5200.3.patch, YARN-5200.4.patch, > YARN-5200.5.patch, YARN-5200.6.patch, YARN-5200.7.patch, YARN-5200.8.patch, > YARN-5200.9.patch, YARN-5200.9.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5361) Obtaining logs for completed container says 'file belongs to a running container ' at the end
[
https://issues.apache.org/jira/browse/YARN-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated YARN-5361:
Issue Type: Sub-task (was: Improvement)
Parent: YARN-4904
> Obtaining logs for completed container says 'file belongs to a running
> container ' at the end
> -
>
> Key: YARN-5361
> URL: https://issues.apache.org/jira/browse/YARN-5361
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Sumana Sathish
>Assignee: Xuan Gong
>Priority: Critical
>
> Obtaining logs via yarn CLI for completed container but running application
> says "This log file belongs to a running container
> (container_e32_1468319707096_0001_01_04) and so may not be complete"
> which is not correct.
> {code}
> LogType:stdout
> Log Upload Time:Tue Jul 12 10:38:14 + 2016
> Log Contents:
> End of LogType:stdout. This log file belongs to a running container
> (container_e32_1468319707096_0001_01_04) and so may not be complete.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5200) Improve yarn logs to get Container List
[ https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373467#comment-15373467 ] Xuan Gong commented on YARN-5200: - Uploaded a new patch v10 to rename the global variable mYarnClient to yarnClient. > Improve yarn logs to get Container List > --- > > Key: YARN-5200 > URL: https://issues.apache.org/jira/browse/YARN-5200 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5200.1.patch, YARN-5200.10.patch, > YARN-5200.2.patch, YARN-5200.3.patch, YARN-5200.4.patch, YARN-5200.5.patch, > YARN-5200.6.patch, YARN-5200.7.patch, YARN-5200.8.patch, YARN-5200.9.patch, > YARN-5200.9.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5200) Improve yarn logs to get Container List
[ https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5200: Attachment: YARN-5200.10.patch > Improve yarn logs to get Container List > --- > > Key: YARN-5200 > URL: https://issues.apache.org/jira/browse/YARN-5200 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5200.1.patch, YARN-5200.10.patch, > YARN-5200.2.patch, YARN-5200.3.patch, YARN-5200.4.patch, YARN-5200.5.patch, > YARN-5200.6.patch, YARN-5200.7.patch, YARN-5200.8.patch, YARN-5200.9.patch, > YARN-5200.9.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5361) Obtaining logs for completed container says 'file belongs to a running container ' at the end
Sumana Sathish created YARN-5361:
Summary: Obtaining logs for completed container says 'file belongs
to a running container ' at the end
Key: YARN-5361
URL: https://issues.apache.org/jira/browse/YARN-5361
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Sumana Sathish
Assignee: Xuan Gong
Priority: Critical
Obtaining logs via yarn CLI for completed container but running application
says "This log file belongs to a running container
(container_e32_1468319707096_0001_01_04) and so may not be complete" which
is not correct.
{code}
LogType:stdout
Log Upload Time:Tue Jul 12 10:38:14 + 2016
Log Contents:
End of LogType:stdout. This log file belongs to a running container
(container_e32_1468319707096_0001_01_04) and so may not be complete.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5356) ResourceUtilization should also include resource availability
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373394#comment-15373394 ] Nathan Roberts commented on YARN-5356: -- bq. I can post a patch with these changes if you want. That would be great. If not I can work on it later this week. > ResourceUtilization should also include resource availability > - > > Key: YARN-5356 > URL: https://issues.apache.org/jira/browse/YARN-5356 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Nathan Roberts > > Currently ResourceUtilization contains absolute quantities of resource used > (e.g. 4096MB memory used). It would be good if it also included how much of > that resource is actually available on the node so that the RM can use this > data to schedule more effectively (overcommit, etc) > Currently the only available information is the Resource the node registered > with (or later updated using updateNodeResource). However, these aren't > really sufficient to get a good view of how utilized a resource is. For > example, if a node reports 400% CPU utilization, does that mean it's > completely full, or barely utilized? Today there is no reliable way to figure > this out. > [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you > have thoughts/opinions on this? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373357#comment-15373357
]
Hadoop QA commented on YARN-5359:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice:
The patch generated 1 new + 3 unchanged - 1 fixed = 4 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 45s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 44s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817472/YARN-5359.01.patch |
| JIRA Issue | YARN-5359 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fbe076c6d142 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7705812 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12291/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12291/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12291/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> FileSystemTimelineReader/Writer uses unix-specific default
> --
>
> Key: YARN-5359
> URL:
[jira] [Commented] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373342#comment-15373342 ] Ming Ma commented on YARN-4676: --- Sorry for joining the discussion late. For the timeout configuration, we had similar discussion for HDFS features such as upgrade domain and datanode maintenance. To support more general per-node attribute, we ended up replace existing host file format with json file in https://issues.apache.org/jira/browse/HDFS-9005. We plan to leverage the new configuration file to support datanode maintenance timeout in https://issues.apache.org/jira/browse/HDFS-7877. It is much easier for admin if the same json format can be used bertween hdfs and yarn. Granted, if we decide to use united json file format, that will delay the commit of this patch. However once the timeout is added to existing host file format, we have to support it for backward compatibility. Maybe the support for decomnission timeout configuration can be a separate jira from the timeout state handling functionality? > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Attachments: GracefulDecommissionYarnNode.pdf, > GracefulDecommissionYarnNode.pdf, YARN-4676.004.patch, YARN-4676.005.patch, > YARN-4676.006.patch, YARN-4676.007.patch, YARN-4676.008.patch, > YARN-4676.009.patch, YARN-4676.010.patch, YARN-4676.011.patch, > YARN-4676.012.patch, YARN-4676.013.patch, YARN-4676.014.patch, > YARN-4676.015.patch, YARN-4676.016.patch > > > YARN-4676 implements an automatic, asynchronous and flexible mechanism to > graceful decommission > YARN nodes. After user issues the refreshNodes request, ResourceManager > automatically evaluates > status of all affected nodes to kicks out decommission or recommission > actions. RM asynchronously > tracks container and application status related to DECOMMISSIONING nodes to > decommission the > nodes immediately after there are ready to be decommissioned. Decommissioning > timeout at individual > nodes granularity is supported and could be dynamically updated. The > mechanism naturally supports multiple > independent graceful decommissioning “sessions” where each one involves > different sets of nodes with > different timeout settings. Such support is ideal and necessary for graceful > decommission request issued > by external cluster management software instead of human. > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5200) Improve yarn logs to get Container List
[
https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373327#comment-15373327
]
Hadoop QA commented on YARN-5200:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 6s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 28s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 13
new + 91 unchanged - 13 fixed = 104 total (was 104) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 25s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 48s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 0s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
| | hadoop.yarn.client.cli.TestLogsCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817469/YARN-5200.9.rebase.patch
|
| JIRA Issue | YARN-5200 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9484a8609819 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7705812 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12289/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12289/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
[jira] [Commented] (YARN-5340) Race condition in RollingLevelDBTimelineStore#getAndSetStartTime()
[
https://issues.apache.org/jira/browse/YARN-5340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373314#comment-15373314
]
Vinod Kumar Vavilapalli commented on YARN-5340:
---
[~gtCarrera9], taking this coarse lock looks bad - we are making levelDB
inserts with the lock held after the patch. Is that okay? /cc [~jeagles],
[~hitesh].
> Race condition in RollingLevelDBTimelineStore#getAndSetStartTime()
> --
>
> Key: YARN-5340
> URL: https://issues.apache.org/jira/browse/YARN-5340
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Assignee: Li Lu
>Priority: Critical
> Attachments: YARN-5340-trunk.001.patch
>
>
> App Name/User/RPC Port/AM Host info is missing from ATS web service or YARN
> CLI's app info
> {code}
> RUNNING: /usr/hdp/current/hadoop-yarn-client/bin/yarn --config
> /tmp/hadoopConf application -status application_1467931619679_0001
> Application Report :
> Application-Id : application_1467931619679_0001
> Application-Name : null
> Application-Type : null
> User : null
> Queue : null
> Application Priority : null
> Start-Time : 0
> Finish-Time : 1467931672057
> Progress : 100%
> State : FINISHED
> Final-State : SUCCEEDED
> Tracking-URL : N/A
> RPC Port : -1
> AM Host : N/A
> Aggregate Resource Allocation : 290014 MB-seconds, 74 vcore-seconds
> Log Aggregation Status : N/A
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-5359:
--
Attachment: YARN-5359.01.patch
Posted patch v.1. Moved to use {{hadoop.tmp.dir}} as the parent for the storage
directory.
> FileSystemTimelineReader/Writer uses unix-specific default
> --
>
> Key: YARN-5359
> URL: https://issues.apache.org/jira/browse/YARN-5359
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
> Attachments: YARN-5359.01.patch
>
>
> {{FileSystemTimelineReaderImpl}} and {{FileSystemTimelineWriterImpl}} use a
> unix-specific default. It won't work on Windows.
> Also, {{TestFileSystemTimelineReaderImpl}} uses this default directly, which
> is also brittle against concurrent tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5359) FileSystemTimelineReader/Writer uses unix-specific default
[
https://issues.apache.org/jira/browse/YARN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-5359:
--
Priority: Minor (was: Major)
> FileSystemTimelineReader/Writer uses unix-specific default
> --
>
> Key: YARN-5359
> URL: https://issues.apache.org/jira/browse/YARN-5359
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Minor
>
> {{FileSystemTimelineReaderImpl}} and {{FileSystemTimelineWriterImpl}} use a
> unix-specific default. It won't work on Windows.
> Also, {{TestFileSystemTimelineReaderImpl}} uses this default directly, which
> is also brittle against concurrent tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5317) testAMRestartNotLostContainerCompleteMsg may fail
[
https://issues.apache.org/jira/browse/YARN-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373290#comment-15373290
]
Hadoop QA commented on YARN-5317:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 77 unchanged - 1 fixed = 79 total (was 78) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 36s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 7s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817463/YARN-5317.02.patch |
| JIRA Issue | YARN-5317 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c812955b271a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7705812 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12288/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12288/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12288/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12288/testReport/ |
| modules | C:
[jira] [Updated] (YARN-5200) Improve yarn logs to get Container List
[ https://issues.apache.org/jira/browse/YARN-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5200: Attachment: YARN-5200.9.rebase.patch > Improve yarn logs to get Container List > --- > > Key: YARN-5200 > URL: https://issues.apache.org/jira/browse/YARN-5200 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5200.1.patch, YARN-5200.2.patch, YARN-5200.3.patch, > YARN-5200.4.patch, YARN-5200.5.patch, YARN-5200.6.patch, YARN-5200.7.patch, > YARN-5200.8.patch, YARN-5200.9.patch, YARN-5200.9.rebase.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5265) Make HBase configuration for the timeline service configurable
[ https://issues.apache.org/jira/browse/YARN-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373261#comment-15373261 ] Joep Rottinghuis commented on YARN-5265: Unit tests are failing. Very plausible that it is caused by this patch. Locally tests run fine. Will do some more debugging and/or add debug statements / patch shortly. > Make HBase configuration for the timeline service configurable > -- > > Key: YARN-5265 > URL: https://issues.apache.org/jira/browse/YARN-5265 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: YARN-5355 > Attachments: ATS v2 cluster deployment v1.png, > YARN-5265-YARN-2928.01.patch, YARN-5265-YARN-2928.02.patch, > YARN-5265-YARN-2928.03.patch, YARN-5265-YARN-2928.04.patch, > YARN-5265-YARN-2928.05.patch, YARN-5265-YARN-5355.06.patch > > > Currently we create "default" HBase configurations, this works as long as the > user places the appropriate configuration on the classpath. > This works fine for a standalone Hadoop cluster. > However, if a user wants to monitor an HBase cluster and has a separate ATS > HBase cluster, then it can become tricky to create the right classpath for > the nodemanagers and still have tasks have their separate configs. > It will be much easier to add a yarn configuration to let cluster admins > configure which HBase to connect to to write ATS metrics to. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373227#comment-15373227
]
Hadoop QA commented on YARN-5333:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 37s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 22s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817453/YARN-5333.01.patch |
| JIRA Issue | YARN-5333 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8fcc99471cec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7705812 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12287/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12287/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12287/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
[jira] [Commented] (YARN-5317) testAMRestartNotLostContainerCompleteMsg may fail
[
https://issues.apache.org/jira/browse/YARN-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373216#comment-15373216
]
Sunil G commented on YARN-5317:
---
Thanks [~sandflee] for the updated patch.
Earlier we had few failures in similar lines. So we used {{launchAM}} and
solved those problems. At present, I am not seeing this same issue in any other
places. Approach in new patch looks fine and thanks for giving a detailed
comment as well.
> testAMRestartNotLostContainerCompleteMsg may fail
> -
>
> Key: YARN-5317
> URL: https://issues.apache.org/jira/browse/YARN-5317
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: sandflee
>Assignee: sandflee
>Priority: Minor
> Attachments: YARN-5317.01.patch, YARN-5317.02.patch
>
>
> java.lang.Exception: test timed out after 3 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:261)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:225)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:207)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:746)
> at
> org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testAMRestartNotLostContainerCompleteMsg(TestAMRestart.java:841)
> see
> https://builds.apache.org/job/PreCommit-YARN-Build/12204/testReport/org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager/TestAMRestart/testAMRestartNotLostContainerCompleteMsg/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5317) testAMRestartNotLostContainerCompleteMsg may fail
[ https://issues.apache.org/jira/browse/YARN-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandflee updated YARN-5317: --- Attachment: YARN-5317.02.patch > testAMRestartNotLostContainerCompleteMsg may fail > - > > Key: YARN-5317 > URL: https://issues.apache.org/jira/browse/YARN-5317 > Project: Hadoop YARN > Issue Type: Test >Reporter: sandflee >Assignee: sandflee >Priority: Minor > Attachments: YARN-5317.01.patch, YARN-5317.02.patch > > > java.lang.Exception: test timed out after 3 milliseconds > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:261) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:225) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:207) > at > org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:746) > at > org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testAMRestartNotLostContainerCompleteMsg(TestAMRestart.java:841) > see > https://builds.apache.org/job/PreCommit-YARN-Build/12204/testReport/org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager/TestAMRestart/testAMRestartNotLostContainerCompleteMsg/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5356) ResourceUtilization should also include resource availability
[
https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373186#comment-15373186
]
Inigo Goiri commented on YARN-5356:
---
[~nroberts], I think that changing machine resources it's not that common and
admins could always restart the Node Manager. in that case, I would just extend
{{RegisterNodeManagerRequestProto}} in
{{yarn_server_common_service_protos.proto}} and populate it in
{{NodeStatusUpdaterImpl#registerWithRM()}}. If we don't care about reporting
network, then adding {{Resource}} is fine. However, if we go into network and
disk, {{ResourceUtilization}} has those fields but I don't think that the
semantics we want to provide matches with resource utilization.
I can post a patch with these changes if you want.
> ResourceUtilization should also include resource availability
> -
>
> Key: YARN-5356
> URL: https://issues.apache.org/jira/browse/YARN-5356
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Nathan Roberts
>
> Currently ResourceUtilization contains absolute quantities of resource used
> (e.g. 4096MB memory used). It would be good if it also included how much of
> that resource is actually available on the node so that the RM can use this
> data to schedule more effectively (overcommit, etc)
> Currently the only available information is the Resource the node registered
> with (or later updated using updateNodeResource). However, these aren't
> really sufficient to get a good view of how utilized a resource is. For
> example, if a node reports 400% CPU utilization, does that mean it's
> completely full, or barely utilized? Today there is no reliable way to figure
> this out.
> [~elgoiri] - Lots of good work is happening in YARN-2965 so curious if you
> have thoughts/opinions on this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Gong updated YARN-5333:
---
Summary: Some recovered apps are put into default queue when RM HA (was:
Recovered apps are rejected when RM HA)
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will reject recovered apps because it might have
> not loaded the new {{fair-scheduler.xml}}. We need call {{initScheduler}}
> before start active services or bring {{refreshAll()}} in front of
> {{rm.transitionToActive()}}. *It seems it is aslo important for other
> scheduler*.
> Related logs are as following:
> {quote}
> 2016-07-07 16:55:34,756 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Recover ended
> ...
> 2016-07-07 16:55:34,824 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService:
> Loading allocation file /gaia/hadoop/etc/hadoop/fair-scheduler.xml
> 2016-07-07 16:55:34,826 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application rejected by queue placement policy
> 2016-07-07 16:55:34,828 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application appattempt_1467803586002_0006_01 is done. finalState=FAILED
> 2016-07-07 16:55:34,828 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Unknown application appattempt_1467803586002_0006_01 has completed!
> 2016-07-07 16:55:34,828 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application rejected by queue placement policy
> 2016-07-07 16:55:34,828 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application appattempt_1467803586002_0004_01 is done. finalState=FAILED
> 2016-07-07 16:55:34,828 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Unknown application appattempt_1467803586002_0004_01 has completed!
> 2016-07-07 16:55:34,828 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Can't handle
> this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> APP_REJECTED at ACCEPTED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:697)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:718)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:702)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:191)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:124)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5333) Some recovered apps are put into default queue when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Gong updated YARN-5333:
---
Description:
Enable RM HA and use FairScheduler,
{{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
{{yarn.scheduler.fair.user-as-default-queue}} is set to false.
Reproduce steps:
1. Start two RMs.
2. After RMs are running, change both RM's file
{{etc/hadoop/fair-scheduler.xml}}, then add some queues.
3. Submit some apps to the new added queues.
4. Stop the active RM, then the standby RM will transit to active and recover
apps.
However the new active RM will put recovered apps into default queue because it
might have not loaded the new {{fair-scheduler.xml}}. We need call
{{initScheduler}} before start active services or bring {{refreshAll()}} in
front of {{rm.transitionToActive()}}. *It seems it is also important for other
scheduler*.
was:
Enable RM HA and use FairScheduler,
{{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
{{yarn.scheduler.fair.user-as-default-queue}} is set to false.
Reproduce steps:
1. Start two RMs.
2. After RMs are running, change both RM's file
{{etc/hadoop/fair-scheduler.xml}}, then add some queues.
3. Submit some apps to the new added queues.
4. Stop the active RM, then the standby RM will transit to active and recover
apps.
However the new active RM will reject recovered apps because it might have not
loaded the new {{fair-scheduler.xml}}. We need call {{initScheduler}} before
start active services or bring {{refreshAll()}} in front of
{{rm.transitionToActive()}}. *It seems it is aslo important for other
scheduler*.
Related logs are as following:
{quote}
2016-07-07 16:55:34,756 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Recover ended
...
2016-07-07 16:55:34,824 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService:
Loading allocation file /gaia/hadoop/etc/hadoop/fair-scheduler.xml
2016-07-07 16:55:34,826 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Application rejected by queue placement policy
2016-07-07 16:55:34,828 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Application appattempt_1467803586002_0006_01 is done. finalState=FAILED
2016-07-07 16:55:34,828 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Unknown application appattempt_1467803586002_0006_01 has completed!
2016-07-07 16:55:34,828 ERROR
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Application rejected by queue placement policy
2016-07-07 16:55:34,828 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Application appattempt_1467803586002_0004_01 is done. finalState=FAILED
2016-07-07 16:55:34,828 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
Unknown application appattempt_1467803586002_0004_01 has completed!
2016-07-07 16:55:34,828 ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Can't handle
this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
APP_REJECTED at ACCEPTED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:697)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:88)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:718)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:702)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:191)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:124)
at java.lang.Thread.run(Thread.java:745)
{quote}
> Some recovered apps are put into default queue when RM HA
> -
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change
[jira] [Commented] (YARN-5333) Recovered apps are rejected when RM HA
[
https://issues.apache.org/jira/browse/YARN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373126#comment-15373126
]
Jun Gong commented on YARN-5333:
Sorry for my mistakes:
1. We changed some code in our code, so that apps will be rejected if the queue
does not exist. For the trunk code, recovered apps will be put into 'default'
queue if their queue does not exist. I think we still need fix it.
2. It will only refresh active RM's queues when running {{rmadmin
-refreshQueues}} on both active and standby RM.
Attach a patch to fix the problem that mentioned above. I verify it on my local
cluster and it works.
> Recovered apps are rejected when RM HA
> --
>
> Key: YARN-5333
> URL: https://issues.apache.org/jira/browse/YARN-5333
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: YARN-5333.01.patch
>
>
> Enable RM HA and use FairScheduler,
> {{yarn.scheduler.fair.allow-undeclared-pools}} is set to false,
> {{yarn.scheduler.fair.user-as-default-queue}} is set to false.
> Reproduce steps:
> 1. Start two RMs.
> 2. After RMs are running, change both RM's file
> {{etc/hadoop/fair-scheduler.xml}}, then add some queues.
> 3. Submit some apps to the new added queues.
> 4. Stop the active RM, then the standby RM will transit to active and recover
> apps.
> However the new active RM will reject recovered apps because it might have
> not loaded the new {{fair-scheduler.xml}}. We need call {{initScheduler}}
> before start active services or bring {{refreshAll()}} in front of
> {{rm.transitionToActive()}}. *It seems it is aslo important for other
> scheduler*.
> Related logs are as following:
> {quote}
> 2016-07-07 16:55:34,756 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Recover ended
> ...
> 2016-07-07 16:55:34,824 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService:
> Loading allocation file /gaia/hadoop/etc/hadoop/fair-scheduler.xml
> 2016-07-07 16:55:34,826 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application rejected by queue placement policy
> 2016-07-07 16:55:34,828 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application appattempt_1467803586002_0006_01 is done. finalState=FAILED
> 2016-07-07 16:55:34,828 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Unknown application appattempt_1467803586002_0006_01 has completed!
> 2016-07-07 16:55:34,828 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application rejected by queue placement policy
> 2016-07-07 16:55:34,828 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application appattempt_1467803586002_0004_01 is done. finalState=FAILED
> 2016-07-07 16:55:34,828 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Unknown application appattempt_1467803586002_0004_01 has completed!
> 2016-07-07 16:55:34,828 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Can't handle
> this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> APP_REJECTED at ACCEPTED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:697)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:88)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:718)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:702)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:191)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:124)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
