Try bumping your memory channel capacities up, they are the same as the batch size. I would go to at least 1000 on each mem channel.
Also, what to the logs and metrics show? From: Siddharth Tiwari [mailto:[email protected]] Sent: Thursday, October 31, 2013 11:53 AM To: [email protected] Subject: Flume not moving data to HDFS or local Hi team I created flume source and sink as following in hadoop yarn and I am not getting data transferred from source to sink in HDFS it doesnt create any file and on local everytime I start agent it creates one empty file. Below are my configs in source and sink Source :- agent.sources = logger1 agent.sources.logger1.type = exec agent.sources.logger1.command = tail -f /var/log/messages agent.sources.logger1.batchsSize = 0 agent.sources.logger1.channels = memoryChannel agent.channels = memoryChannel agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 100 agent.sinks = AvroSink agent.sinks.AvroSink.type = avro agent.sinks.AvroSink.channel = memoryChannel agent.sinks.AvroSink.hostname = 192.168.147.101 agent.sinks.AvroSink.port = 4545 agent.sources.logger1.interceptors = itime ihost agent.sources.logger1.interceptors.itime.type = TimestampInterceptor agent.sources.logger1.interceptors.ihost.type = host agent.sources.logger1.interceptors.ihost.useIP = false agent.sources.logger1.interceptors.ihost.hostHeader = host Sink at one of the slave ( datanodes on my Yarn cluster ) : collector.sources = AvroIn collector.sources.AvroIn.type = avro collector.sources.AvroIn.bind = 0.0.0.0 collector.sources.AvroIn.port = 4545 collector.sources.AvroIn.channels = mc1 mc2 collector.channels = mc1 mc2 collector.channels.mc1.type = memory collector.channels.mc1.capacity = 100 collector.channels.mc2.type = memory collector.channels.mc2.capacity = 100 collector.sinks = LocalOut HadoopOut collector.sinks.LocalOut.type = file_roll collector.sinks.LocalOut.sink.directory = /home/hadoop/flume collector.sinks.LocalOut.sink.rollInterval = 0 collector.sinks.LocalOut.channel = mc1 collector.sinks.HadoopOut.type = hdfs collector.sinks.HadoopOut.channel = mc2 collector.sinks.HadoopOut.hdfs.path = /flume collector.sinks.HadoopOut.hdfs.fileType = DataStream collector.sinks.HadoopOut.hdfs.writeFormat = Text collector.sinks.HadoopOut.hdfs.rollSize = 0 collector.sinks.HadoopOut.hdfs.rollCount = 10000 collector.sinks.HadoopOut.hdfs.rollInterval = 600 can somebody point me to what am I doing wrong ? This is what I get in my local directory [hadoop@node1 flume]$ ls -lrt total 0 -rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:25 1383243942803-1 -rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:28 1383244097923-1 -rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:31 1383244302225-1 -rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:33 1383244404929-1 when I restart the collector it creates one 0 bytes file. Please help *------------------------* Cheers !!! Siddharth Tiwari Have a refreshing day !!! "Every duty is holy, and devotion to duty is the highest form of worship of God." "Maybe other people will try to limit me but I don't limit myself"
