Hi
I am using Flume1.2. Using avro source and hdfs sink. Sending message using
the Logger channel from application server. For that i am using the
*org.apache.flume.clients.log4jappender.Log4jAppender
*in MyApp.
But i am getting only body of the message (description). loosing the time,
thread, Level information.
flume-conf.properties file
==================
agent2Test1.sources = seqGenSrc
agent2Test1.channels = memoryChannel
agent2Test1.sinks = loggerSink
# For each one of the sources, the type is defined
agent2Test1.sources.seqGenSrc.type = avro
agent2Test1.sources.seqGenSrc.bind=localhost
agent2Test1.sources.seqGenSrc.port=41414
# interceptors for host and date
agent2Test1.sources.seqGenSrc.interceptors = time hostInterceptor
agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.type =
org.apache.flume.interceptor.HostInterceptor$Builder
agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader = host
agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.useIP = false
agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader.preserveExisting
= false
agent2Test1.sources.seqGenSrc.interceptors.time.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
# The channel can be defined as follows.
agent2Test1.sources.seqGenSrc.channels = memoryChannel
# Each sink's type must be defined
agent2Test1.sinks.loggerSink.type = hdfs
agent2Test1.sinks.loggerSink.hdfs.path =
hdfs://hadoopHost:8020/data/%Y/%m/%d/%{host}/Logs
agent2Test1.sinks.loggerSink.hdfs.fileType = DataStream
#Specify the channel the sink should use
agent2Test1.sinks.loggerSink.channel = memoryChannel
# Each channel's type is defined.
agent2Test1.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent2Test1.channels.memoryChannel.capacity = 1000
Sample java program to generate the log message:
=====================================
package com.test;
import org.apache.flume.clients.log4jappender.Log4jAppender;
import org.apache.log4j.Logger;
import org.apache.log4j.MDC;
import org.apache.log4j.PatternLayout;
import java.util.UUID;
public class Main {
static Logger log = Logger.getLogger(Main.class);
public static void main(String[] args) {
try {
Log4jAppender appender = new Log4jAppender();
appender.setHostname("localhost");
appender.setPort(41414);
appender.setLayout(new PatternLayout("%d [%c] (%t) <%X{user}
%X{field}> %m"));
// appender.setReconnectAttempts(100);
appender.activateOptions();
log.addAppender(appender);
MDC.put("user", "chris");
// while (true) {
MDC.put("field", UUID.randomUUID().toString());
log.info("=====> Hello World");
try {
throw new Exception("Testing");
} catch (Exception e) {
log.error("Gone wrong ===>", e);
}
//}
System.in.read();
System.in.read();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
I am missing any config here ?
--
Thanks,
Khadar