Hi, I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along the bundle
I have setup a 5 machine cluster and on the same 5 machines, I'm also running zookeeper as well I am trying to see what is the maximum produce throughput I can get on this 5 node cluster I have created only 1 topic - tops1 in the cluster:- root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper localhost:2182 Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 This is the output of the kafka-producer-perf-test.sh for request-num-acks 1 and request-num-acks -1:- root@x.x.x.x:~# date;time kafka-producer-perf-test.sh --broker-list x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages 1000000 --message-size 500 --topics tops1 --show-detailed-stats --threads 30 --request-num-acks 1 --batch-size 1000 --request-timeout-ms 10000 start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec 2015-08-21 09:36:01:021, 2015-08-21 09:36:49:351, 0, 500, 1000, 476.83, 9.8662, 999990, 20690.8752 real 0m49.375s user 0m38.777s sys 0m5.951s root@x.x.x.x:~# date;time kafka-producer-perf-test.sh --broker-list x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages 1000000 --message-size 500 --topics tops1 --show-detailed-stats --threads 30 --request-num-acks -1 --batch-size 1000 --request-timeout-ms 10000 start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec 2015-08-21 09:31:19:329, 2015-08-21 09:32:08:627, 0, 500, 1000, 476.83, 9.6724, 999990, 20284.5957 real 0m50.341s user 0m39.029s sys 0m6.051s I'll continue to test with more number of messages, to see if I get any difference. But, I just want to be sure whatever I'm doing is right. Please share your thoughts on the results. Regards, prabcs
