Hi I am using spooldir source, memory channel, hdfs sink to collect log files and store into HDFS.
When I run the flume agent, it is creating very very small files with size 766 bytes. Input file: test.log [11.4 KB] Output files: sales_web_log.1443871052640.log, etc.[all are very very small files with size 766 bytes] *How to increase the output file size?* *Thanks & Regards,* *Shiva Ram* *Website: http://datamaking.com <http://datamaking.com>Facebook Page: www.facebook.com/datamaking <http://www.facebook.com/datamaking>*
2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412121 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412121 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412121 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412121 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412121 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412122 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412122 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412122 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412122 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412122 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412123 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412123 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412123 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412123 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412123 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412124 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412124 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412124 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412124 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412124 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412125 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412125 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412125 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412125 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412125 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412126 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412126 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412126 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412126 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412126 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412127 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412127 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412127 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412127 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412127 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412128 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412128 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412128 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412128 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412128 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412129 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412129 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412129 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412129 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412129 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412130 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412130 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412130 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412130 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412130 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412131 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412131 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412131 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412131 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412131 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412132 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412132 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412132 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412132 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412132 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412133 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412133 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412133 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412133 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412133 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412134 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412134 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412134 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412134 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412134 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412135 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412135 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412135 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412135 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412135 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412136 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412136 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412136 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412136 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412136 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412137 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412137 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412137 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412137 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412137 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412138 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412138 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412138 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412138 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412138 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412139 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412139 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412139 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412139 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412139 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412140 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412140 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412140 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412140 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412140 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412141 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412141 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412141 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412141 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412141 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412142 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412142 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412142 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412142 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412142 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412143 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412143 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412143 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412143 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412143 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412144 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412144 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412144 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412144 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412144 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412145 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412145 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412145 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412145 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412145 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412146 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412146 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412146 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412146 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412146 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412147 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412147 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412147 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412147 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412147 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412148 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412148 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412148 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412148 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412148 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412149 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412149 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412149 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412149 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412149 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412150 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412150 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412150 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412150
2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412121 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412121 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412121 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412121 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412121 2015-10-02 18:06:26 DEBUG HelloWorldDemo:20 - This is debug : mkyong||412122 2015-10-02 18:06:26 INFO HelloWorldDemo:24 - This is info : mkyong||412122 2015-10-02 18:06:26 WARN HelloWorldDemo:27 - This is warn : mkyong||412122 2015-10-02 18:06:26 ERROR HelloWorldDemo:28 - This is error : mkyong||412122 2015-10-02 18:06:26 FATAL HelloWorldDemo:29 - This is fatal : mkyong||412122
