Missed this thread. No direct solution comes to my mind for this Couple of things, since the degradation is occurring only with few instances.
1. Find out what is different on the machines seeing degradation from the others (including OS configurations) 2. Is it the degradation random or occurring on specific hosts Put me in this situation and I would do following 1. Try to look out for a Pattern of degradation, similar hosts, some event storm or could be anything 2. If you have monitoring system, see what's happening around during the degradation, heavy GC activity. Once you get a similar pattern, perhaps debugging and solving it would be easier. On Mon, Apr 6, 2015 at 12:20 AM, ajeet singh <ajeetpr.si...@gmail.com> wrote: > ---------- Forwarded message ---------- > From: ajeet singh <ajeetpr.si...@gmail.com> > Date: Mon, Apr 6, 2015 at 12:03 AM > Subject: Issue : Flume performance > To: u...@flume.apache.org > > > I am using flume for logging events from aws servers to KAFKA. I am using > logback as flume source, and kafka-sink as sink. In servers we are getting > 3000 events/ per seconds, and there are almost 30 servers running and one > central kafka. At each server flume process is running which will be > writing to central kafka. > I am getting some strange situations like at some servers > flume is able for process the event with rate in which it's getting the > event but on some servers some time it get slow ( like it's getting event > with rate 3k / sec but transferring events to kafka with rate 1Kv per sec). > This way 2K /per sec events are getting added to flume channel, and after > 3-4 hours channel overflow starts. I compared both type of servers where > flume is working fine, and where flume is slow, all parameters ( load, cpu > utilization, memory usages and disk writes) are almost same. What can be > reason for this strange behavior ?? > > Thanks > Ajeet > > > > -- > AJEET PRATAP SINGH -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
