[jira] [Created] (YARN-5369) Improve Yarn logs command to get container logs based on Node Id
Xuan Gong created YARN-5369:
---
Summary: Improve Yarn logs command to get container logs based on
Node Id
Key: YARN-5369
URL: https://issues.apache.org/jira/browse/YARN-5369
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
It is helpful if we could have yarn logs --applicationId appId --nodeAddress
${nodeId} to get all the container logs which ran on the specific nm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5368) memory leak at timeline server
Wataru Yukawa created YARN-5368:
---
Summary: memory leak at timeline server
Key: YARN-5368
URL: https://issues.apache.org/jira/browse/YARN-5368
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 2.7.1
Environment: HDP2.4
CentOS 6.7
jdk1.8.0_72
Reporter: Wataru Yukawa
memory usage of timeline server machine increases gradually.
https://gyazo.com/952dad96c77ae053bae2e4d8c8ab0572
please check since April.
According to my investigation, timeline server used about 25GB.
top command result
{code}
90577 yarn 20 0 28.4g 25g 12m S 0.0 40.1 5162:53
/usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m
-Dhdp.version=2.4.0.0-169 -Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
-Dyarn.log.dir=/var/log/hadoop-yarn/yarn ...
{code}
ps command result
{code}
$ ps ww 90577
90577 ?Sl 5162:53 /usr/java/jdk1.8.0_72/bin/java
-Dproc_timelineserver -Xmx1024m -Dhdp.version=2.4.0.0-169
-Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
-Dyarn.log.dir=/var/log/hadoop-yarn/yarn
-Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log
-Dyarn.log.file=yarn-yarn-timelineserver-myhost.log -Dyarn.home.dir=
-Dyarn.id.str=yarn -Dhadoop.root.logger=INFO,EWMA,RFA
-Dyarn.root.logger=INFO,EWMA,RFA
-Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
-Dyarn.policy.file=hadoop-policy.xml
-Djava.io.tmpdir=/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
-Dhadoop.log.dir=/var/log/hadoop-yarn/yarn
-Dyarn.log.dir=/var/log/hadoop-yarn/yarn
-Dhadoop.log.file=yarn-yarn-timelineserver-myhost.log
-Dyarn.log.file=yarn-yarn-timelineserver-myhost.log
-Dyarn.home.dir=/usr/hdp/current/hadoop-yarn-timelineserver
-Dhadoop.home.dir=/usr/hdp/2.4.0.0-169/hadoop
-Dhadoop.root.logger=INFO,EWMA,RFA -Dyarn.root.logger=INFO,EWMA,RFA
-Djava.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native:/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir
-classpath
/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/lib/*:/usr/hdp/2.4.0.0-169/hadoop/.//*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/./:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/*:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//*:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/*:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/*:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//*::/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/2.4.0.0-169/tez/*:/usr/hdp/2.4.0.0-169/tez/lib/*:/usr/hdp/2.4.0.0-169/tez/conf:/usr/hdp/current/hadoop-yarn-timelineserver/.//*:/usr/hdp/current/hadoop-yarn-timelineserver/lib/*:/usr/hdp/2.4.0.0-169/hadoop/conf/timelineserver-config/log4j.properties
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer
{code}
Alghough I set -Xmx1024m, actual memory usage is 25GB.
After I restart timeline server, memory usage of timeline server machine
decreases.
https://gyazo.com/130600c17a7d41df8606727a859ae7e3
Now timelineserver uses less than 1GB memory.
top command result
{code}
6163 yarn 20 0 3959m 783m 46m S 0.3 1.2 3:37.60
/usr/java/jdk1.8.0_72/bin/java -Dproc_timelineserver -Xmx1024m
-Dhdp.version=2.4.0.0-169 ...
{code}
I suspect memory leak at timeline server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5367) HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials
Xianyin Xin created YARN-5367: - Summary: HDFS delegation tokens in ApplicationSubmissionContext should be added to systemCrednetials Key: YARN-5367 URL: https://issues.apache.org/jira/browse/YARN-5367 Project: Hadoop YARN Issue Type: Bug Components: yarn Reporter: Xianyin Xin Assignee: Xianyin Xin App log aggregation may failed because of the below flow: 0) suppose the token.max-lifetime is 7 days and renew interval is 1 day; 1) start a long running job, like sparkJDBC, of which the AM acts as a service. When submitting the job, HDFS token A in ApplicationSubmissionContext will be added to DelegationTokenRenewer, but not added to systemCredentials; 2) after 1 day, submit a spark query. After received the query, AM will request containers and start tasks. When start the containers, a new HDFS token B is used; 3) after 1 day, kill the job, when doing log aggregation, exception occurs which show token B is not in the HDFS token cache so the connecting to HDFS fails; We should add token A to systemCredentials to make sure token A can be delivered to NMs in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5366) Add support for toggling the removal of completed and failed docker containers
Shane Kumpf created YARN-5366: - Summary: Add support for toggling the removal of completed and failed docker containers Key: YARN-5366 URL: https://issues.apache.org/jira/browse/YARN-5366 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Shane Kumpf Currently, completed and failed docker containers are removed by container-executor. Add a job level environment variable to DockerLinuxContainerRuntime to allow the user to toggle whether they want the container deleted or not and remove the logic from container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5365) Add support for YARN Shared Cache
Ming Ma created YARN-5365: - Summary: Add support for YARN Shared Cache Key: YARN-5365 URL: https://issues.apache.org/jira/browse/YARN-5365 Project: Hadoop YARN Issue Type: Improvement Reporter: Ming Ma YARN provides shared cache in functionality YARN-1492. According to [~ctrezzo] most of the YARN functionality is in hadoop 2.8 and frameworks can start to use it. MR adds the support via MAPREDUCE-5951. Can anyone confirm if Tez supports the upload of application DAG jar and dependent lib jars from client machine to HDFS as part of Tez app submission? From my test, that doesn't seem to happen. Tez expects applications to upload the jars to HDFS beforehand and then set the tez.aux.uris to the HDFS locations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4366) Fix Lint Warnings in YARN Common
[
https://issues.apache.org/jira/browse/YARN-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter resolved YARN-4366.
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
Thanks [~templedf]. Committed to trunk and branch-2!
> Fix Lint Warnings in YARN Common
>
>
> Key: YARN-4366
> URL: https://issues.apache.org/jira/browse/YARN-4366
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Fix For: 2.9.0
>
> Attachments: YARN-4366.001.patch
>
>
> {noformat}
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java:[100,45]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Class for a varargs call
> cast to java.lang.Class[] for a non-varargs call and to suppress this
> warning
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[62,46]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Class for a varargs call
> cast to java.lang.Class[] for a non-varargs call and to suppress this
> warning
> [WARNING]
> /Users/daniel/NetBeansProjects/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/RpcFactoryProvider.java:[64,34]
> non-varargs call of varargs method with inexact argument type for last
> parameter;
> cast to java.lang.Object for a varargs call
> cast to java.lang.Object[] for a non-varargs call and to suppress this
> warning
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5364) timelineservice modules have indirect dependencies on mapreduce artifacts
Sangjin Lee created YARN-5364: - Summary: timelineservice modules have indirect dependencies on mapreduce artifacts Key: YARN-5364 URL: https://issues.apache.org/jira/browse/YARN-5364 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 3.0.0-alpha1 Reporter: Sangjin Lee Assignee: Sangjin Lee Priority: Minor The new timelineservice and timelineservice-hbase-tests modules have indirect dependencies to mapreduce artifacts through HBase and phoenix. Although it's not causing builds to fail, it's not good hygiene. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5363) For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default
Vinod Kumar Vavilapalli created YARN-5363:
-
Summary: For AM containers, or for containers of running-apps,
"yarn logs" incorrectly only (tries to) shows syslog file-type by default
Key: YARN-5363
URL: https://issues.apache.org/jira/browse/YARN-5363
Project: Hadoop YARN
Issue Type: Sub-task
Components: log-aggregation
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
For e.g, for a running application, the following happens:
{code}
# yarn logs -applicationId application_1467838922593_0001
16/07/06 22:07:05 INFO impl.TimelineClientImpl: Timeline service address:
http://:8188/ws/v1/timeline/
16/07/06 22:07:06 INFO client.RMProxy: Connecting to ResourceManager at
/:8050
16/07/06 22:07:07 INFO impl.TimelineClientImpl: Timeline service address:
http://l:8188/ws/v1/timeline/
16/07/06 22:07:07 INFO client.RMProxy: Connecting to ResourceManager at
/:8050
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_01 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_02 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_03 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_04 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_05 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_06 within the application:
application_1467838922593_0001
Can not find any log file matching the pattern: [syslog] for the container:
container_e03_1467838922593_0001_01_07 within the application:
application_1467838922593_0001
Can not find the logs for the application: application_1467838922593_0001 with
the appOwner:
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5362) TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail
Jason Lowe created YARN-5362:
Summary: TestRMRestart#testFinishedAppRemovalAfterRMRestart can
fail
Key: YARN-5362
URL: https://issues.apache.org/jira/browse/YARN-5362
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jason Lowe
Saw the following in a precommit build that only changed an unrelated unit test:
{noformat}
Tests run: 29, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 101.265 sec
<<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
testFinishedAppRemovalAfterRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 0.411 sec <<< FAILURE!
java.lang.AssertionError: expected null, but
was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotNull(Assert.java:664)
at org.junit.Assert.assertNull(Assert.java:646)
at org.junit.Assert.assertNull(Assert.java:656)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1653)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5361) Obtaining logs for completed container says 'file belongs to a running container ' at the end
Sumana Sathish created YARN-5361:
Summary: Obtaining logs for completed container says 'file belongs
to a running container ' at the end
Key: YARN-5361
URL: https://issues.apache.org/jira/browse/YARN-5361
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Sumana Sathish
Assignee: Xuan Gong
Priority: Critical
Obtaining logs via yarn CLI for completed container but running application
says "This log file belongs to a running container
(container_e32_1468319707096_0001_01_04) and so may not be complete" which
is not correct.
{code}
LogType:stdout
Log Upload Time:Tue Jul 12 10:38:14 + 2016
Log Contents:
End of LogType:stdout. This log file belongs to a running container
(container_e32_1468319707096_0001_01_04) and so may not be complete.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/ [Jul 11, 2016 9:55:33 PM] (jitendra) HDFS-10579. HDFS web interfaces lack configs for X-FRAME-OPTIONS [Jul 12, 2016 12:21:56 AM] (aajisaka) HADOOP-13297. Add missing dependency in setting [Jul 12, 2016 1:06:28 AM] (wang) HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by [Jul 12, 2016 5:36:20 AM] (jianhe) YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.cli.TestLogsCLI hadoop.yarn.client.api.impl.TestYarnClient cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-compile-javac-root.txt [172K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/diff-javadoc-javadoc-root.txt [2.3M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/92/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5360) Use UID instead of user name to build the Docker run command
Zhankun Tang created YARN-5360: -- Summary: Use UID instead of user name to build the Docker run command Key: YARN-5360 URL: https://issues.apache.org/jira/browse/YARN-5360 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Zhankun Tang Assignee: Zhankun Tang There is *a dependency between job submitting user and the user in the Docker image* in LCE currently. For instance, in order to run the Docker container as yarn user, we can choose set the "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" to yarn and leave "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" default (true). Then LCE will choose yarn ( UID maybe 1001) as the user running jobs. But because LCE will mount the generated launch_container.sh (owned by the running job user) into the Docker container and utilizes "docker run --user=" option to get it done internally, we also need to create a *same user name* in the Docker image with the *same UID* as the running job user. Otherwise LCE will fail to launch container or report unable to find user. This burdens the Docker image creator with YARN dependency. Luckily this can be solved through Docker. As far as I know, since Docker v1.8 (or maybe earlier), the Docker run command "--user=" option accepts UID and *when passing UID, the user does not have to exist in the container*. So we should use UID instead of user name to construct the Docker run command to eliminate the dependency that create the same user in the Docker image. This enables LCE the ability to launch any Docker container safely regardless what users in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
