First, sorry about not good at English. I am trying to use Flume-ng to grab 128MB of log information and put it into a file in HDFS. But HDFS rolling options not working. Flume-ng send log file per seconds. How can I fix flume.conf file?
my flume.conf looks below: agent01.sources = avroGenSrc agent01.channels = memoryChannel hdfsChannel agent01.sinks = fileSink hadoopSink # For each one of the sources, the type is defined agent01.sources.avroGenSrc.type = avro agent01.sources.avroGenSrc.bind = dev-hadoop03.ncl agent01.sources.avroGenSrc.port = 3333 # The channel can be defined as follows. agent01.sources.avroGenSrc.channels = memoryChannel hdfsChannel # Each sink's type must be defined agent01.sinks.fileSink.type = file_roll agent01.sinks.fileSink.sink.directory = /home1/irteam/flume/data agent01.sinks.fileSink.sink.rollInterval = 3600 agent01.sinks.fileSink.sink.batchSize = 100 #Specify the channel the sink should use agent01.sinks.fileSink.channel = memoryChannel agent01.sinks.hadoopSink.type = hdfs agent01.sinks.hadoopSink.hdfs.useLocalTimeStamp = true agent01.sinks.hadoopSink.hdfs.path = hdfs://dev-hadoop04.ncl:9000/user/hive/warehouse/raw_logs/year=%Y/month=%m/day=%d agent01.sinks.hadoopSink.hdfs.filePrefix = AccessLog.%Y-%m-%d.%Hh agent01.sinks.hadoopSink.hdfs.fileType = DataStream agent01.sinks.hadoopSink.hdfs.writeFormat = Text agent01.sinks.hadoopSink.hdfs.rollInterval = 0 agent01.sinks.hadoopSink.hdfs.rollSize = 134217728 agent01.sinks.hadoopSink.hdfs.rollCount = 0 #Specify the channel the sink should use agent01.sinks.hadoopSink.channel = hdfsChannel # Each channel's type is defined. agent01.channels.memoryChannel.type = memory agent01.channels.hdfsChannel.type = memory # Other config values specific to each type of channel(sink or source) # can be defined as well # In this case, it specifies the capacity of the memory channel agent01.channels.memoryChannel.capacity = 100000 agent01.channels.memoryChannel.transactionCapacity = 10000 agent01.channels.hdfsChannel.capacity = 100000 agent01.channels.hdfsChannel.transactionCapacity = 10000
