Hi All,
I am trying to figure out best configuration for my Kafka brokers so that
in case of restarted, the new node catch up with Leader at quick pace.
My test environment has 2 kafka brokers and 1 Topic with one Partition.
I first ran the test (Test#1) with default setting, i.e.
num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB).
it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec (
37.9*1024/700)
Later I increased the num.replica.fetchers = 5 and
replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the
replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec.
i ran two more test and results got much worse:
Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB.
replication rate = 92.7 MB/sec
Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB.
replication rate = 93.54 MB/sec
Any reason why increasing the replica fetcher thread or increase in fetch
max bytes not increasing my replication rate linearly.
note: in all the test CPU utilization was not more than 45%
Regards,
Nitin Kumar Sharma.
