Hi, I have 10 flume agents configured at a single machine. A single log file has frequency of 500 log events/sec. Hence in 10 log files the logs are getting generated as 5000 log events per second (5000/sec).
If my channel capacity is 1 million, more than 70% of log events is lost! If I increase the channel capacity to 50 millions, then flume agent takes more than 24 hours to transfer the log events from source to sink. The size of dataDir (agent.channels.fileChannel.dataDirs = /var/log/flume-ng/file-channel/data) is almost 2G all the time. Could you please guide me the optimum configuration so that I don't miss any of log events & the transfer is also good enough. My flume-conf.properties has following contents: agent.channels = fileChannel agent.sinks = avroSink # Each sink's type must be defined agent.sinks.avroSink.type = avro agent.sinks.avroSink.hostname = spnnq01.nam.nsroot.net agent.sinks.avroSink.port = 1442 agent.sinks.avroSink.batchSize = 1000 #Specify the channel the sink should use agent.sinks.avroSink.channel = fileChannel # Each channel's type is defined. agent.channels.fileChannel.type = file agent.channels.fileChannel.checkpointDir = /var/log/flume-ng/file-channel/checkpoint agent.channels.fileChannel.dataDirs = /var/log/flume-ng/file-channel/data agent.channels.fileChannel.transactionCapacity = 1000 agent.channels.fileChannel.checkpointInterval = 30000 agent.channels.fileChannel.maxFileSize = 2146435071 agent.channels.fileChannel.minimumRequiredSpace = 524288000 agent.channels.fileChannel.keep-alive = 5 agent.channels.fileChannel.write-timeout = 10 agent.channels.fileChannel.checkpoint-timeout = 600 agent.channels.fileChannel.capacity = 50000000 agent.sources.s2.batchSize = 1 agent.sources.s2.channels = fileChannel agent.sources.s2.command = tail -F /var/log/creditcard/AggKeyListener.2.2013-01-19 agent.sources.s2.interceptors = logIntercept agent.sources.s2.interceptors.logIntercept.appId = 153299 agent.sources.s2.interceptors.logIntercept.env = SP agent.sources.s2.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s2.interceptors.logIntercept.logFileName = AggKeyListener.2.2013-01-19 agent.sources.s2.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s2.interceptors.logIntercept.logType = creditcard log agent.sources.s2.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s2.type = exec agent.sources.s0.batchSize = 1 agent.sources.s0.channels = fileChannel agent.sources.s0.command = tail -F /var/log/creditcard/AggKeyListener.0.2013-01-19 agent.sources.s0.interceptors = logIntercept agent.sources.s0.interceptors.logIntercept.appId = 153299 agent.sources.s0.interceptors.logIntercept.env = SP agent.sources.s0.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s0.interceptors.logIntercept.logFileName = AggKeyListener.0.2013-01-19 agent.sources.s0.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s0.interceptors.logIntercept.logType = creditcard log agent.sources.s0.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s0.type = exec agent.sources.s1.batchSize = 1 agent.sources.s1.channels = fileChannel agent.sources.s1.command = tail -F /var/log/creditcard/AggKeyListener.1.2013-01-19 agent.sources.s1.interceptors = logIntercept agent.sources.s1.interceptors.logIntercept.appId = 153299 agent.sources.s1.interceptors.logIntercept.env = SP agent.sources.s1.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s1.interceptors.logIntercept.logFileName = AggKeyListener.1.2013-01-19 agent.sources.s1.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s1.interceptors.logIntercept.logType = creditcard log agent.sources.s1.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s1.type = exec agent.sources.s3.batchSize = 1 agent.sources.s3.channels = fileChannel agent.sources.s3.command = tail -F /var/log/creditcard/AggKeyListener.3.2013-01-19 agent.sources.s3.interceptors = logIntercept agent.sources.s3.interceptors.logIntercept.appId = 153299 agent.sources.s3.interceptors.logIntercept.env = SP agent.sources.s3.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s3.interceptors.logIntercept.logFileName = AggKeyListener.3.2013-01-19 agent.sources.s3.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s3.interceptors.logIntercept.logType = creditcard log agent.sources.s3.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s3.type = exec agent.sources.s4.batchSize = 1 agent.sources.s4.channels = fileChannel agent.sources.s4.command = tail -F /var/log/creditcard/AggKeyListener.4.2013-01-19 agent.sources.s4.interceptors = logIntercept agent.sources.s4.interceptors.logIntercept.appId = 153299 agent.sources.s4.interceptors.logIntercept.env = SP agent.sources.s4.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s4.interceptors.logIntercept.logFileName = AggKeyListener.4.2013-01-19 agent.sources.s4.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s4.interceptors.logIntercept.logType = creditcard log agent.sources.s4.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s4.type = exec agent.sources.s5.batchSize = 1 agent.sources.s5.channels = fileChannel agent.sources.s5.command = tail -F /var/log/creditcard/AggKeyListener.5.2013-01-19 agent.sources.s5.interceptors = logIntercept agent.sources.s5.interceptors.logIntercept.appId = 153299 agent.sources.s5.interceptors.logIntercept.env = SP agent.sources.s5.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s5.interceptors.logIntercept.logFileName = AggKeyListener.5.2013-01-19 agent.sources.s5.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s5.interceptors.logIntercept.logType = creditcard log agent.sources.s5.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s5.type = exec agent.sources.s6.batchSize = 1 agent.sources.s6.channels = fileChannel agent.sources.s6.command = tail -F /var/log/creditcard/AggKeyListener.6.2013-01-19 agent.sources.s6.interceptors = logIntercept agent.sources.s6.interceptors.logIntercept.appId = 153299 agent.sources.s6.interceptors.logIntercept.env = SP agent.sources.s6.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s6.interceptors.logIntercept.logFileName = AggKeyListener.6.2013-01-19 agent.sources.s6.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s6.interceptors.logIntercept.logType = creditcard log agent.sources.s6.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s6.type = exec agent.sources.s7.batchSize = 1 agent.sources.s7.channels = fileChannel agent.sources.s7.command = tail -F /var/log/creditcard/AggKeyListener.7.2013-01-19 agent.sources.s7.interceptors = logIntercept agent.sources.s7.interceptors.logIntercept.appId = 153299 agent.sources.s7.interceptors.logIntercept.env = SP agent.sources.s7.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s7.interceptors.logIntercept.logFileName = AggKeyListener.7.2013-01-19 agent.sources.s7.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s7.interceptors.logIntercept.logType = creditcard log agent.sources.s7.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s7.type = exec agent.sources.s8.batchSize = 1 agent.sources.s8.channels = fileChannel agent.sources.s8.command = tail -F /var/log/creditcard/AggKeyListener.8.2013-01-19 agent.sources.s8.interceptors = logIntercept agent.sources.s8.interceptors.logIntercept.appId = 153299 agent.sources.s8.interceptors.logIntercept.env = SP agent.sources.s8.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s8.interceptors.logIntercept.logFileName = AggKeyListener.8.2013-01-19 agent.sources.s8.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s8.interceptors.logIntercept.logType = creditcard log agent.sources.s8.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s8.type = exec agent.sources.s9.batchSize = 1 agent.sources.s9.channels = fileChannel agent.sources.s9.command = tail -F /var/log/creditcard/AggKeyListener.9.2013-01-19 agent.sources.s9.interceptors = logIntercept agent.sources.s9.interceptors.logIntercept.appId = 153299 agent.sources.s9.interceptors.logIntercept.env = SP agent.sources.s9.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net agent.sources.s9.interceptors.logIntercept.logFileName = AggKeyListener.9.2013-01-19 agent.sources.s9.interceptors.logIntercept.logFilePath = /var/log/creditcard/ agent.sources.s9.interceptors.logIntercept.logType = creditcard log agent.sources.s9.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder agent.sources.s9.type = exec Regards, Deepak
