HI All -- has any one faced the problem i have posted in my previous email ?
Regards, Nitin Kumar Sharma. On Wed, Dec 17, 2014 at 11:08 AM, nitin sharma <kumarsharma.ni...@gmail.com> wrote: > > Hi All, > > I am trying to figure out best configuration for my Kafka brokers so that > in case of restarted, the new node catch up with Leader at quick pace. > > My test environment has 2 kafka brokers and 1 Topic with one Partition. > > I first ran the test (Test#1) with default setting, i.e. > num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB). > it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec ( > 37.9*1024/700) > > Later I increased the num.replica.fetchers = 5 and > replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the > replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec. > > i ran two more test and results got much worse: > Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB. > replication rate = 92.7 MB/sec > > Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB. > replication rate = 93.54 MB/sec > > Any reason why increasing the replica fetcher thread or increase in fetch > max bytes not increasing my replication rate linearly. > > note: in all the test CPU utilization was not more than 45% > > > Regards, > Nitin Kumar Sharma. > >
