[
https://issues.apache.org/jira/browse/MAPREDUCE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874776#comment-13874776
]
Hudson commented on MAPREDUCE-5672:
-----------------------------------
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1672 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1672/])
MAPREDUCE-5672. Provide optional RollingFileAppender for container log4j
(syslog). Contributed by Gera Shegalov (jlowe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1558948)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties
> Provide optional RollingFileAppender for container log4j (syslog)
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-5672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5672
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 2.2.0
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Fix For: 2.4.0
>
> Attachments: MAPREDUCE-5672.v01.patch, MAPREDUCE-5672.v02.patch,
> MAPREDUCE-5672.v03.patch, MAPREDUCE-5672.v04.patch, MAPREDUCE-5672.v05.patch,
> MAPREDUCE-5672.v06.patch, MAPREDUCE-5672.v07.patch, MAPREDUCE-5672.v07.patch,
> Screen Shot 2013-12-05 at 3.21.02 PM.png, Screen Shot 2013-12-05 at 3.23.33
> PM.png
>
>
> This JIRA is an alternative take on YARN-1130
> We propose providing an option of using a RollingFileAppender(RFA)-based
> implementation of container log appender as means of log size control via
> mapreduce.task.userlog.limit.kb.
> The idea is to use mapreduce.task.userlog.limit.kb as maximumFileSize of RFA.
> In addition yarn.app.mapreduce.container.log.backups (task attempt
> containers) and yarn.app.mapreduce.am.log.backups (MR-AM) are passed as
> maxBackupIndex.
> Both current ContainerLogAppender (CLA) and new ContainerRollingLogAppender
> (CRLA) co-exist. CLA is the default. CRLA is chosen when
> mapreduce.task.userlog.limit.kb > 0 && *.backups > 0.
> Pros:
> 1) CRLA output is visible in UI right away. CLA output with
> mapreduce.task.userlog.limit.kb > 0 is not visible until the task attempt
> finishes that prevents timely diagnostics.
> 2) Even with excessive logging and a large mapreduce.task.userlog.limit.kb,
> no space is taken from the JVM heap.
> 3) No UI impact, since YARN is already designed to deal with any log name
> beyond stderr/out, syslog, debug.out, profile.out
> Cons:
> 1) if the logging is excessive there will be more local filesystem metadata
> I/O due to roll. That should be negligible in the grand scheme.
> Furthermore, to improve log consistency and completeness in the case of JVM
> crashes and SIGTERMing by NM, we propose to restore the MRv1 behavior of
> periodic log syncing (every 5s) and having log sync as part of a shutdown
> hook.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
