---------- Forwarded message ----------
From: ajeet singh <[email protected]>
Date: Mon, Apr 6, 2015 at 12:03 AM
Subject: Issue : Flume performance
To: [email protected]
I am using flume for logging events from aws servers to KAFKA. I am using
logback as flume source, and kafka-sink as sink. In servers we are getting
3000 events/ per seconds, and there are almost 30 servers running and one
central kafka. At each server flume process is running which will be
writing to central kafka.
I am getting some strange situations like at some servers
flume is able for process the event with rate in which it's getting the
event but on some servers some time it get slow ( like it's getting event
with rate 3k / sec but transferring events to kafka with rate 1Kv per sec).
This way 2K /per sec events are getting added to flume channel, and after
3-4 hours channel overflow starts. I compared both type of servers where
flume is working fine, and where flume is slow, all parameters ( load, cpu
utilization, memory usages and disk writes) are almost same. What can be
reason for this strange behavior ??
Thanks
Ajeet
--
AJEET PRATAP SINGH
