Hello Folks,

I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along
the bundle

I have setup a 5 machine cluster and on the same 5 machines, I'm also
running zookeeper as well

I am trying to see what is the maximum produce throughput I can get on this
5 node cluster

I have created only 1 topic - tops1 in the cluster:-

[email protected]:~# kafka-topics.sh --describe --topic tops1 --zookeeper
localhost:2182

Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs:

Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2

Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2

Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3

Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5


When I run this command in parallel from 2 different machines, I get
SocketTimeoutException:-

time kafka-producer-perf-test.sh --broker-list
x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages
1000000 --message-size 500 --topics tops1 --show-detailed-stats  --threads
5 --request-num-acks -1 --batch-size 1000


My server.properties has these details:-


broker.id=0

port=9092

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=1048576

socket.receive.buffer.bytes=1048576

socket.request.max.bytes=104857600

log.dirs=/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

log.cleaner.enable=false

zookeeper.connect=localhost:2181

zookeeper.connection.timeout.ms=6000

num.replica.fetchers=4


2 major changes in the server.properties, which gave a performance boost on
this test were:-

num.replica.fetchers=4

socket.send.buffer.bytes=1048576

socket.receive.buffer.bytes=1048576


I think I'm giving enough requests on the network, but not sure how I
should correct it

I am particularly interested in knowing how much max throughput per topic
can I get, with the highest level of durability (i.e. request-num-acks = -1)


Request you to share your thoughts on this.

Thanks,

Prabhjot

Reply via email to