[ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13787370#comment-13787370 ]
Arun C Murthy commented on YARN-1130: ------------------------------------- I'm reviewing this, meanwhile, [~paulhan] can you pls look at the unit test failures? Tx. > Improve the log flushing for tasks when mapred.userlog.limit.kb is set > ---------------------------------------------------------------------- > > Key: YARN-1130 > URL: https://issues.apache.org/jira/browse/YARN-1130 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Affects Versions: 2.0.5-alpha > Reporter: Paul Han > Assignee: Paul Han > Fix For: 2.0.5-alpha > > Attachments: YARN-1130.patch > > > When userlog limit is set with something like this: > {code} > <property> > <name>mapred.userlog.limit.kb</name> > <value>2048</value> > <description>The maximum size of user-logs of each task in KB. 0 disables the > cap. > </description> > </property> > {code} > the log entry will be truncated randomly for the jobs. > The log size is left between 1.2MB to 1.6MB. > Since the log is already limited, avoid the log truncation is crucial for > user. > The other issue with the current > impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will > not flush to file until the container shutdown and logmanager close all > appenders. If user likes to see the log during task execution, it doesn't > support it. > Will propose a patch to add a flush mechanism and also flush the log when > task is done. -- This message was sent by Atlassian JIRA (v6.1#6144)