Hi, The log4J appender doesn't pay any attention to the PatternLayout, at least up through 1.3.0-SNAPSHOT. That's why you only see %m and nothing else.
As for the garbled text, your property of this: agent.sinks.hdfsSink.hdfs.file.Type should be this: agent.sinks.hdfsSink.hdfs.fileType Since you had file.Type, Flume was using the default of SequenceFile. Hope that helps! Chris On Wed, Sep 12, 2012 at 1:58 AM, Hao Jian <haoj...@neusoft.com> wrote: > ** > Hi, > I'm using flume ng(1.2.0) to collect logs from log4j and save logs to hdfs > . There are two problems: > (1) Flume only collect %m in log4j, but not %d, %p, %t ... > (2) The log saved in hdfs is garbled, not plain text. > My log4j configuration is as fowllows: > <appender name="Console" class="org.apache.log4j.ConsoleAppender"> > <layout class="org.apache.log4j.PatternLayout"> > <param name="ConversionPattern" > value="%d >> %-5p >> %t >> %l >> %m%n"/> > </layout> > </appender> > <appender name="flume" > class="org.apache.flume.clients.log4jappender.Log4jAppender"> > <param name="Hostname" value="10.4.46.125" /> > <param name="Port" value="44444" /> > <layout class="org.apache.log4j.PatternLayout"> > <param name="ConversionPattern" > value="%d >> %-5p >> %t >> %l >> %m%n"/> > </layout> > </appender> > <root> > <level value="info"/> > <appender-ref ref="Console"/> > <appender-ref ref="flume"/> > </root> > The console print is: > 2012-09-12 11:43:37,391 >> INFO >> main >> Main.main(Main.java:22) >> > this is test1 > 2012-09-12 11:43:37,454 >> INFO >> main >> Main.main(Main.java:22) >> this > is test2 > 2012-09-12 11:43:37,460 >> INFO >> main >> Main.main(Main.java:22) >> this > is test3 > 2012-09-12 11:43:37,465 >> INFO >> main >> Main.main(Main.java:22) >> this > is test4 > 2012-09-12 11:43:37,470 >> INFO >> main >> Main.main(Main.java:22) >> this > is test5 > 2012-09-12 11:43:37,475 >> INFO >> main >> Main.main(Main.java:22) >> this > is test6 > 2012-09-12 11:43:37,480 >> INFO >> main >> Main.main(Main.java:22) >> this > is test7 > 2012-09-12 11:43:37,485 >> INFO >> main >> Main.main(Main.java:22) >> this > is test8 > 2012-09-12 11:43:37,492 >> INFO >> main >> Main.main(Main.java:22) >> this > is test9 > 2012-09-12 11:43:37,497 >> INFO >> main >> Main.main(Main.java:22) >> this > is test10 > The flume configuration is: > agent.channels = jdbcChannel memChannel > agent.sinks = hdfsSink fileSink > agent.channels.jdbcChannel.type = jdbc > agent.channels.memChannel.type = memory > agent.sources.avroSrc.type = avro > agent.sources.avroSrc.bind = 0.0.0.0 > agent.sources.avroSrc.port = 40000 > agent.sources.avroSrc.channels = jdbcChannel memChannel > agent.sinks.hdfsSink.channel = jdbcChannel > agent.sinks.hdfsSink.type = hdfs > agent.sinks.hdfsSink.hdfs.path = hdfs://10.4.44.134/flume/events/ > agent.sinks.hdfsSink.hdfs.rollCount = 0 > agent.sinks.hdfsSink.hdfs.writeFormat = Writable > agent.sinks.hdfsSink.hdfs.file.Type = DataStream > agent.sinks.fileSink.channel = memChannel > agent.sinks.fileSink.type = FILE_ROLL > agent.sinks.fileSink.sink.directory = /var/log/flume/ > agent.sinks.fileSink.sink.rollInterval = 60 > The log saved in /var/log/flume is: > this is test1 > this is test2 > this is test3 > this is test4 > this is test5 > this is test6 > this is test7 > this is test8 > this is test9 > this is test10 > The log save in hdfs is: > SEQ > !org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable������� > � > �D� ���# �cQ��� ��� �� 9���K��� > this is test1��� ��� �� 9���m��� > this is test2��� ��� �� 9������ > this is test3��� ��� �� 9������ > this is test4��� ��� �� 9������ > this is test5��� ��� �� 9������� > this is test6��� ��� �� 9������� > this is test7��� ��� �� 9������� > this is test8��� ��� �� 9������� > this is test9��� ��� �� 9������� this is test10 > or: > > SEQ!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd > > \ufffdD\ufffd\ufffd\ufffd\ufffd#\ufffdcQ\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffdK\ufffd\ufffd\ufffd > this is > test1\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffdm\ufffd\ufffd\ufffd > this is > test2\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd > this is > test3\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd > this is > test4\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd > this is > test5\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd > this is > test6\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd > this is > test7\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd > this is > test8\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd > this is > test9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd9\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdthis > is test10 > I've tried to set "agent.sinks.hdfsSink.hdfs.writeFormat" and > "agent.sinks.hdfsSink.hdfs.file.Type" to other value, but the data in hdfs > is still garbled. > > 2012-09-12 > ------------------------------ > > > > --------------------------------------------------------------------------------------------------- > Confidentiality Notice: The information contained in this e-mail and any > accompanying attachment(s) > is intended only for the use of the intended recipient and may be > confidential and/or privileged of > Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader > of this communication is > not the intended recipient, unauthorized use, forwarding, printing, > storing, disclosure or copying > is strictly prohibited, and may be unlawful.If you have received this > communication in error,please > immediately notify the sender by return e-mail, and delete the original > message and all copies from > your system. Thank you. > > --------------------------------------------------------------------------------------------------- >
