Re: kafka 0.8.2 broker behaviour

2016-05-24 Thread Anishek Agarwal
yeh mostly they are up always, sometimes when we do new deployment they might be down for about 15 mins or so. seems to have become slightly better now, i changed the replica socket buffer size and also the number of replica fetcher threads. On Mon, May 23, 2016 at 6:24 PM, Gerard Klijs wrote:

Re: kafka 0.8.2 broker behaviour

2016-05-23 Thread Gerard Klijs
Are you sure consumers are always up, when they are behind they could generate a lot of traffic in a small amount of time? On Mon, May 23, 2016 at 9:11 AM Anishek Agarwal wrote: > additionally all the read / writes are happening via storm topologies. > > On Mon, May 23, 2016 at 12:17 PM, Anishek

Re: kafka 0.8.2 broker behaviour

2016-05-23 Thread Anishek Agarwal
additionally all the read / writes are happening via storm topologies. On Mon, May 23, 2016 at 12:17 PM, Anishek Agarwal wrote: > Hello, > > we are using 4 kafka machines in production with 4 topics and each topic > either 16/32 partitions and replication factor of 2. each machine has 3 > disks

kafka 0.8.2 broker behaviour

2016-05-22 Thread Anishek Agarwal
Hello, we are using 4 kafka machines in production with 4 topics and each topic either 16/32 partitions and replication factor of 2. each machine has 3 disks for kafka logs. we see a strange behaviour where we see high disk usage spikes on one of the disks on all machines. it varies over time, wi