Re: SocketTimeoutException with kafka-producer-perf-test.sh
Hi, Never mind. I have solved this problem by referring to an earlier question, where Neha had suggested to use a higher value for request-timeout-ms Regards, prabcs On Fri, Aug 21, 2015 at 12:22 PM, Prabhjot Bharaj prabhbha...@gmail.com wrote: Hello Folks, I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along the bundle I have setup a 5 machine cluster and on the same 5 machines, I'm also running zookeeper as well I am trying to see what is the maximum produce throughput I can get on this 5 node cluster I have created only 1 topic - tops1 in the cluster:- root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper localhost:2182 Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 When I run this command in parallel from 2 different machines, I get SocketTimeoutException:- time kafka-producer-perf-test.sh --broker-list x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages 100 --message-size 500 --topics tops1 --show-detailed-stats --threads 5 --request-num-acks -1 --batch-size 1000 My server.properties has these details:- broker.id=0 port=9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 log.dirs=/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=30 log.cleaner.enable=false zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 num.replica.fetchers=4 2 major changes in the server.properties, which gave a performance boost on this test were:- num.replica.fetchers=4 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 I think I'm giving enough requests on the network, but not sure how I should correct it I am particularly interested in knowing how much max throughput per topic can I get, with the highest level of durability (i.e. request-num-acks = -1) Request you to share your thoughts on this. Thanks, Prabhjot -- - There are only 10 types of people in the world: Those who understand binary, and those who don't
Re: SocketTimeoutException with kafka-producer-perf-test.sh
This link already had the answer I was searching for: http://grokbase.com/t/kafka/users/145vmz70cb/java-net-sockettimeoutexception-in-broker On Fri, Aug 21, 2015 at 3:03 PM, Prabhjot Bharaj prabhbha...@gmail.com wrote: Hi, Never mind. I have solved this problem by referring to an earlier question, where Neha had suggested to use a higher value for request-timeout-ms Regards, prabcs On Fri, Aug 21, 2015 at 12:22 PM, Prabhjot Bharaj prabhbha...@gmail.com wrote: Hello Folks, I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along the bundle I have setup a 5 machine cluster and on the same 5 machines, I'm also running zookeeper as well I am trying to see what is the maximum produce throughput I can get on this 5 node cluster I have created only 1 topic - tops1 in the cluster:- root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper localhost:2182 Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 When I run this command in parallel from 2 different machines, I get SocketTimeoutException:- time kafka-producer-perf-test.sh --broker-list x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages 100 --message-size 500 --topics tops1 --show-detailed-stats --threads 5 --request-num-acks -1 --batch-size 1000 My server.properties has these details:- broker.id=0 port=9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 log.dirs=/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=30 log.cleaner.enable=false zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 num.replica.fetchers=4 2 major changes in the server.properties, which gave a performance boost on this test were:- num.replica.fetchers=4 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 I think I'm giving enough requests on the network, but not sure how I should correct it I am particularly interested in knowing how much max throughput per topic can I get, with the highest level of durability (i.e. request-num-acks = -1) Request you to share your thoughts on this. Thanks, Prabhjot -- - There are only 10 types of people in the world: Those who understand binary, and those who don't -- - There are only 10 types of people in the world: Those who understand binary, and those who don't
SocketTimeoutException with kafka-producer-perf-test.sh
Hello Folks, I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along the bundle I have setup a 5 machine cluster and on the same 5 machines, I'm also running zookeeper as well I am trying to see what is the maximum produce throughput I can get on this 5 node cluster I have created only 1 topic - tops1 in the cluster:- root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper localhost:2182 Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 When I run this command in parallel from 2 different machines, I get SocketTimeoutException:- time kafka-producer-perf-test.sh --broker-list x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages 100 --message-size 500 --topics tops1 --show-detailed-stats --threads 5 --request-num-acks -1 --batch-size 1000 My server.properties has these details:- broker.id=0 port=9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 log.dirs=/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=30 log.cleaner.enable=false zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 num.replica.fetchers=4 2 major changes in the server.properties, which gave a performance boost on this test were:- num.replica.fetchers=4 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 I think I'm giving enough requests on the network, but not sure how I should correct it I am particularly interested in knowing how much max throughput per topic can I get, with the highest level of durability (i.e. request-num-acks = -1) Request you to share your thoughts on this. Thanks, Prabhjot
