hi,
I am using Flume log4j appender to write log events to HDFS but it contains
some junk value and I am not able to see anything other than log message no
timestamp.
Here is my configuration
Log4j.properties
log4j.logger.log4jExample= DEBUG,out2
log4j.appender.out2 = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.out2.Port = 41414
log4j.appender.out2.Hostname = 172.20.104.223
here is agent configuration
a1.sources = r1
a1.sinks = k1
a1.channels = c1
#sources
a1.sources.r1.type = avro
a1.sources.r1.bind =172.20.104.226
a1.sources.r1.port= 41414
a1.sources.r1.restart =true
a1.sources.r1.batchsize=10000
# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path=hdfs://172.20.104.226:8020/flumeinput/%{host}
a1.sinks.k1.hdfs.file.Type=DataStream
a1.sinks.k1.hdfs.writeFormat=Writable
a1.sinks.k1.hdfs.rollCount=10000
a1.sinks.k1.serializer=TEXT
# Use a channel which buffers events in memory
a1.channels.c1.type = file
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
Expected output
[2013-01-09 15:15:45,457] - [main] DEBUG log4jExample Current data unavailalbe,
using cached values
[2013-01-09 15:15:45,458] - [main] INFO log4jExample Hello this is an info
message
[2013-01-09 15:15:45,460] - [main] ERROR log4jExample Dabase unavaliable,
connetion lost
[2013-01-09 15:15:45,461] - [main] WARN log4jExample Attention!! Application
running in debugmode
[2013-01-09 15:15:45,463] - [main] DEBUG log4jExample Current data unavailalbe,
using cached values
[2013-01-09 15:15:45,465] - [main] INFO log4jExample Hello this is an info
message
[2013-01-09 15:15:45,467] - [main] ERROR log4jExample Dabase unavaliable,
connetion lost
[2013-01-09 15:15:45,468] - [main] WARN log4jExample Attention!! Application
running in debugmode
[2013-01-09 15:15:45,470] - [main] DEBUG log4jExample Current data unavailalbe,
using cached values
But getting this
Output on HDFS
SEQ!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable������+�AE����9<‑��-Current
data unavailalbe, using cached values)<‑��Hello this is an info
message.<‑��"Dabase unavaliable, connetion lost8<‑��,Attention!! Application
running in debugmode9<‑��-Current data unavailalbe, using cached
values)<‑��Hello this is an info message.<‑��"Dabase unavaliable, connetion
lost8<‑�� ,Attention!! Application running in debugmode9<‑��-Current data
unavailalbe, using cached values)<‑��‑Hello this is an info
message.<‑��‑"Dabase unavaliable, connetion lost8<‑��,Attention!!
Application running in debugmode9<‑�� -Current data unavailalbe, using cached
values)<‑�� Hello this is an info message.<‑��!"Dabase unavaliable, connetion
lost8<‑��",Attention!! Application running in debugmode9<‑��"-Current data
unavailalbe, using cached values)<‑��#Hello this is an info
message.<‑��#"Dabase unavaliable, connetion lost8<‑��$,Attention!!
Application running in debugmode9<‑��$-Current data unavailalbe, using cached
values)<‑��%Hello this is an info message.<‑��%"Dabase unavaliable, connetion
lost8<‑��%,Attention!! Application running in debugmode9<‑��&-Current data
unavailalbe, using cached values)<‑��&Hello this is an info
message.<‑��'"Dabase unavaliable, connetion lost8<‑��(,Attention!!
Application running in debugmode
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or
privileged information for the intended recipient(s). Unintended recipients are
prohibited from taking action on the basis of information in this e-mail and
using or disseminating the information, and must notify the sender and delete
it from their system. L&T Infotech will not accept responsibility or liability
for the accuracy or completeness of, or the presence of any virus or disabling
code in this e-mail"
