Hello there, I was wondering if anyone here could help me with some insight into a conundrum that I am facing.
Basically, the story is that I am running three Kafka brokers via docker on a single vm with log.flush.interval.messages = 1 and min.insync.replicas = 2. Then I create two topics: both with replication factor = 3, but one with one partition and the other with 64. Then I try to run a benchmark using these topics and what I find is as follows: 1 partition, 1381.02 records/sec, 685.87 ms average latency 64 partitions, 601.00 records/sec, 1298.18 ms average latency This is the opposite of what I expected. In neither case am I even close to the IOPS of what the disk can handle. So what I would like to know is if there is any obvious reason that I am missing for the slow down with more partitions? If it is helpful the docker-compose file and the code to do the benchmarking can be found at https://github.com/siyopao/kafka-benchmark. (Any comments or advice on how to make the code better are greatly appreciated!) The benchmarking code is inspired by and very similar to what the bin/kafka-producer-perf-test.sh script does. Thank you! Best wishes, Craig
