Hi all,
I'm currently doing benchmarks on flume.
We're planning to use flume with syslogtcp as source and filechannel in
order to have avoid data loss.
The performances are quiet good when a memorychannel is used :
~*100 000events/sec* (event size = 600bytes)
But as soon as i switch to filechannel the performances drop drammatically:
~*300events/sec*
Despite this poor result, the behaviour is really strange because i have
a heavy disk usage (all the disks), near 100%.
I use a tool provided by syslog-ng in order to generate syslog logs :
loggen
<http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/loggen.1.html>
ex : loggen -i -I 3000000 --size 600 --active-connections 200
myflumehost 20515
Flume version : 1.5.2
Operating System : Centos 6
Please find my flume configuration enclosed. The filechannel is spread
over 5 disks in order to improve performance.
Could you please help me to configure properly syslogtcp source with
filechannel ?
Regards,
--
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196
27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
arkena.com
agent.sources = src1
agent.sources.src1.type = syslogtcp
agent.sources.src1.channels = chan1
agent.sources.src1.host = 0.0.0.0
agent.sources.src1.port = 20515
agent.sources.src1.eventSize = 200000
agent.channels = chan1
agent.channels.chan1.type = file
agent.channels.chan1.capacity = 100000000
agent.channels.chan1.transactionCapacity = 2000000
agent.channels.chan1.checkpointDir=/mnt/flumeCheckpoint
agent.channels.chan1.checkpointInterval=300000
agent.channels.chan1.useDualCheckpoints=true
agent.channels.chan1.backupCheckpointDir=/mnt/diske/backupCheckpoint
agent.channels.chan1.dataDirs =
/mnt/diske,/mnt/diskf,/mnt/diskg,/mnt/diskh,/mnt/diski
#agent.channels.chan1.type = memory
#agent.channels.chan1.capacity = 5000000
#agent.channels.chan1.transactionCapacity = 100000
agent.sinks = sin1 sin2 sin3 sin4 sin5
agent.sinks.sin1.channel = chan1
agent.sinks.sin1.type = file_roll
agent.sinks.sin1.sink.directory = /mnt/diske
agent.sinks.sin1.batchSize = 20000
agent.sinks.sin2.channel = chan1
agent.sinks.sin2.type = file_roll
agent.sinks.sin2.sink.directory = /mnt/diskf
agent.sinks.sin1.batchSize = 20000
agent.sinks.sin3.channel = chan1
agent.sinks.sin3.type = file_roll
agent.sinks.sin3.sink.directory = /mnt/diskg
agent.sinks.sin1.batchSize = 20000
agent.sinks.sin4.channel = chan1
agent.sinks.sin4.type = file_roll
agent.sinks.sin4.sink.directory = /mnt/diskh
agent.sinks.sin1.batchSize = 20000
agent.sinks.sin5.channel = chan1
agent.sinks.sin5.type = file_roll
agent.sinks.sin5.sink.directory = /mnt/diski
agent.sinks.sin1.batchSize = 20000
