[
https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765870#comment-13765870
]
Paul Han commented on YARN-1130:
--------------------------------
Patch is submitted.
A few notes here:
# Added a LogUtils class to facilitate triggering log flush since Log4J's
interface such as LogManager doesn't provide a flush() interface
# Modified Task class to trigger the flushing of logs before it sends TASK_DONE
event to MRMaster. In some cases, TASK_DONE may trigger the container to be
killed before all log being written to disk
# ContainerLogAppender supports to flush the log when a special message is
received. The log flush is done in a "synchronous" manner with a timeout. This
will ensure the invoker of flush waits until logs are written to disk or
timeout happens.
> Improve the log flushing for tasks when mapred.userlog.limit.kb is set
> ----------------------------------------------------------------------
>
> Key: YARN-1130
> URL: https://issues.apache.org/jira/browse/YARN-1130
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.5-alpha
> Reporter: Paul Han
> Fix For: 2.0.5-alpha
>
> Attachments: YARN-1130.patch
>
>
> When userlog limit is set with something like this:
> {code}
> <property>
> <name>mapred.userlog.limit.kb</name>
> <value>2048</value>
> <description>The maximum size of user-logs of each task in KB. 0 disables the
> cap.
> </description>
> </property>
> {code}
> the log entry will be truncated randomly for the jobs.
> The log size is left between 1.2MB to 1.6MB.
> Since the log is already limited, avoid the log truncation is crucial for
> user.
> The other issue with the current
> impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will
> not flush to file until the container shutdown and logmanager close all
> appenders. If user likes to see the log during task execution, it doesn't
> support it.
> Will propose a patch to add a flush mechanism and also flush the log when
> task is done.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
