Hi Anthony, On Wed, Jul 18, 2007 at 07:42:58PM -0700, Anthony D. Urso wrote: >I have started to use the following log4j xml to send logs to both the >mapreduce tasklog and to the syslog daemon. Unfortunately, it creates >a new log split in the tasklog for each log entry. > >Is this a problem with the TaskLogAppender? If not, does anyone know >method of tapping into the mapreduce logging that won't cause zillions >of log splits to be created? >
Tweak the following in your hadoop-site.xml (please see hadoop-defaults.xml for their descriptions): mapred.userlog.purgesplits (set this to *false*) mapred.userlog.num.splits (set this to 1) hth, Arun >XML properties and an exception below: > ><?xml version="1.0" encoding="UTF-8" ?> ><!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> ><log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/";> > <!-- Log base level and above to syslog, for posterity's sake. > <appender name="syslog" class="org.apache.log4j.net.SyslogAppender"> > <param name="SyslogHost" value="localhost" /> > <param name="Facility" value="LOCAL5" /> > <layout class="org.apache.log4j.PatternLayout"> > <param name="ConversionPattern" value="%p %c: %m%n"/> > </layout> > </appender> --> > > <!-- Log everything to the task log. --> > <appender name="TLA" class="org.apache.hadoop.mapred.TaskLogAppender"> > <param name="taskId" value="${hadoop.tasklog.taskid}"/> > <!-- keep alot, they are one line each --> > <param name="noKeepSplits" value="999999"/> > <param name="totalLogFileSize" value="100"/> > <param name="purgeLogSplits" value="true"/> > <param name="logsRetainHours" value="4"/> > <layout class="org.apache.log4j.PatternLayout"> > <param name="ConversionPattern" value="%d %p %c: %m%n"/> > </layout> > </appender> > > <!-- A console logger for hadoop. --> > <appender name="console" class="org.apache.log4j.ConsoleAppender"> > <layout class="org.apache.log4j.PatternLayout"> > <param name="ConversionPattern" value="%d %p %c: %m%n"/> > </layout> > </appender> > > <!-- Log task stuff tasklog and syslog --> > <logger name="com.example.package" additivity="false"> > <level value="ALL" /> > <!-- <appender-ref ref="syslog" /> --> > <appender-ref ref="TLA" /> > </logger> > > <!-- Log hadoop stuff as per the original log4j.properties. --> > <root> > <level value="INFO" /> > <appender-ref ref="console" /> > </root> ></log4j:configuration >> >An exception, perhaps related... > >ERROR Failed to close the task's log with the exception: java.io.IOException: >Bad file descriptor > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > at > org.apache.hadoop.mapred.TaskLog$Writer.writeIndexRecord(TaskLog.java:252) > at org.apache.hadoop.mapred.TaskLog$Writer.close(TaskLog.java:236) > >-- > Au > > PGP Key ID: 0x385B44CB > Fingerprint: 9E9E B116 DB2C D734 C090 E72F 43A0 95C4 385B 44CB > "Maximus vero fugiens a quodam Urso, milite Romano, interemptus est" > - Getica 235
