You might consider looking at Log4j 2. It has a Flume Appender that records the whole formatted log message in the body. In addition, it will record the MDC fields as well.
Sent from my iPad On Jul 27, 2012, at 5:49 AM, khadar basha <[email protected]> wrote: > Hi > > I am using Flume1.2. Using avro source and hdfs sink. Sending message using > the Logger channel from application server. For that i am using the > org.apache.flume.clients.log4jappender.Log4jAppender in MyApp. > > But i am getting only body of the message (description). loosing the time, > thread, Level information. > > flume-conf.properties file > ================== > agent2Test1.sources = seqGenSrc > agent2Test1.channels = memoryChannel > agent2Test1.sinks = loggerSink > > # For each one of the sources, the type is defined > agent2Test1.sources.seqGenSrc.type = avro > agent2Test1.sources.seqGenSrc.bind=localhost > agent2Test1.sources.seqGenSrc.port=41414 > > # interceptors for host and date > agent2Test1.sources.seqGenSrc.interceptors = time hostInterceptor > agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.type = > org.apache.flume.interceptor.HostInterceptor$Builder > agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader = host > agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.useIP = false > agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader.preserveExisting > = false > agent2Test1.sources.seqGenSrc.interceptors.time.type = > org.apache.flume.interceptor.TimestampInterceptor$Builder > > # The channel can be defined as follows. > agent2Test1.sources.seqGenSrc.channels = memoryChannel > > # Each sink's type must be defined > agent2Test1.sinks.loggerSink.type = hdfs > agent2Test1.sinks.loggerSink.hdfs.path = > hdfs://hadoopHost:8020/data/%Y/%m/%d/%{host}/Logs > > agent2Test1.sinks.loggerSink.hdfs.fileType = DataStream > > #Specify the channel the sink should use > agent2Test1.sinks.loggerSink.channel = memoryChannel > > # Each channel's type is defined. > agent2Test1.channels.memoryChannel.type = memory > > # Other config values specific to each type of channel(sink or source) > # can be defined as well > # In this case, it specifies the capacity of the memory channel > agent2Test1.channels.memoryChannel.capacity = 1000 > > > > Sample java program to generate the log message: > ===================================== > > > package com.test; > > > import org.apache.flume.clients.log4jappender.Log4jAppender; > import org.apache.log4j.Logger; > import org.apache.log4j.MDC; > import org.apache.log4j.PatternLayout; > > import java.util.UUID; > > > public class Main { > static Logger log = Logger.getLogger(Main.class); > > public static void main(String[] args) { > try { > Log4jAppender appender = new Log4jAppender(); > appender.setHostname("localhost"); > appender.setPort(41414); > appender.setLayout(new PatternLayout("%d [%c] (%t) <%X{user} > %X{field}> %m")); > // appender.setReconnectAttempts(100); > > appender.activateOptions(); > > log.addAppender(appender); > > MDC.put("user", "chris"); > // while (true) { > MDC.put("field", UUID.randomUUID().toString()); > log.info("=====> Hello World"); > try { > throw new Exception("Testing"); > } catch (Exception e) { > log.error("Gone wrong ===>", e); > } > //} > System.in.read(); > System.in.read(); > } > catch (Exception e) { > e.printStackTrace(); > } > } > } > > > > I am missing any config here ? > > > > -- > Thanks, > Khadar >
