Re: Poor performance running performance test
That error indicates the broker closed the connection for some reason. Any useful logs from the broker? It looks like you're using ELB, which could also be the culprit. A connection timeout seems doubtful, but ELB can also close connections for other reasons, like failed health checks. -Ewen On Tue, Jan 27, 2015 at 11:21 PM, Dillian Murphey crackshotm...@gmail.com wrote: I was running the performance command from a virtual box server, so that seems like it was part of the problem. I'm getting better results running this on a server on aws, but that's kind of expected. Can you look at these results, and comment on the occasional warning I see? I appreciate it! 1220375 records sent, 243928.6 records/sec (23.26 MB/sec), 2111.5 ms avg latency, 4435.0 max latency. 1195090 records sent, 239018.0 records/sec (22.79 MB/sec), 2203.1 ms avg latency, 4595.0 max latency. 1257165 records sent, 251433.0 records/sec (23.98 MB/sec), 2172.6 ms avg latency, 4525.0 max latency. 1230981 records sent, 246196.2 records/sec (23.48 MB/sec), 2173.5 ms avg latency, 4465.0 max latency. [2015-01-28 07:19:07,274] WARN Error in I/O with myawsloadbalancer(org.apache.kafka.common.network.Selector) java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 1090689 records sent, 218137.8 records/sec (20.80 MB/sec), 2413.6 ms avg latency, 4829.0 max latency. On Tue, Jan 27, 2015 at 7:37 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Where are you running ProducerPerformance in relation to ZK and the Kafka brokers? You should definitely see much higher performance than this. A couple of other things I can think of that might be going wrong: Are all your VMs in the same AZ? Are you storing Kafka data in EBS or local ephemeral storage? If EBS, have you provisioned enough IOPS. On Tue, Jan 27, 2015 at 4:29 PM, Dillian Murphey crackshotm...@gmail.com wrote: I'm a new user/admin to kafka. I'm running a 3 node ZK and a 6 brokers on aws. The performance I'm seeing is shockingly bad. I need some advice! bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test2 5000 100 -1 acks=1 bootstrap.servers=5myloadbalancer:9092 buffer.memory=67108864 batch.size=8196 6097 records sent, 13198.3 records/sec (1.26 MB/sec), 2098.0 ms avg latency, 4306.0 max latency. 71695 records sent, 14339.0 records/sec (1.37 MB/sec), 6658.1 ms avg latency, 9053.0 max latency. 65195 records sent, 13028.6 records/sec (1.24 MB/sec), 11504.0 ms avg latency, 13809.0 max latency. 71955 records sent, 14391.0 records/sec (1.37 MB/sec), 16137.4 ms avg latency, 18541.0 max latency. Thanks for any help! -- Thanks, Ewen -- Thanks, Ewen
Re: Poor performance running performance test
You could be right Ewen. I was starting to wonder about the load balancer too. Is using a load balancer a bad idea? How else do users know which kafka broker to connect to? I'm using one of the IPs directly and I don't see that error. I am seeing an occasional connection refused. What the heck. Maybe this is another aws specific thing. OR, I am running kafka brokers in a docker container. I think I will remove the docker component and see if that makes a difference. Thanks for the reply. bow
Re: Poor performance running performance test
I was running the performance command from a virtual box server, so that seems like it was part of the problem. I'm getting better results running this on a server on aws, but that's kind of expected. Can you look at these results, and comment on the occasional warning I see? I appreciate it! 1220375 records sent, 243928.6 records/sec (23.26 MB/sec), 2111.5 ms avg latency, 4435.0 max latency. 1195090 records sent, 239018.0 records/sec (22.79 MB/sec), 2203.1 ms avg latency, 4595.0 max latency. 1257165 records sent, 251433.0 records/sec (23.98 MB/sec), 2172.6 ms avg latency, 4525.0 max latency. 1230981 records sent, 246196.2 records/sec (23.48 MB/sec), 2173.5 ms avg latency, 4465.0 max latency. [2015-01-28 07:19:07,274] WARN Error in I/O with myawsloadbalancer(org.apache.kafka.common.network.Selector) java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 1090689 records sent, 218137.8 records/sec (20.80 MB/sec), 2413.6 ms avg latency, 4829.0 max latency. On Tue, Jan 27, 2015 at 7:37 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Where are you running ProducerPerformance in relation to ZK and the Kafka brokers? You should definitely see much higher performance than this. A couple of other things I can think of that might be going wrong: Are all your VMs in the same AZ? Are you storing Kafka data in EBS or local ephemeral storage? If EBS, have you provisioned enough IOPS. On Tue, Jan 27, 2015 at 4:29 PM, Dillian Murphey crackshotm...@gmail.com wrote: I'm a new user/admin to kafka. I'm running a 3 node ZK and a 6 brokers on aws. The performance I'm seeing is shockingly bad. I need some advice! bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test2 5000 100 -1 acks=1 bootstrap.servers=5myloadbalancer:9092 buffer.memory=67108864 batch.size=8196 6097 records sent, 13198.3 records/sec (1.26 MB/sec), 2098.0 ms avg latency, 4306.0 max latency. 71695 records sent, 14339.0 records/sec (1.37 MB/sec), 6658.1 ms avg latency, 9053.0 max latency. 65195 records sent, 13028.6 records/sec (1.24 MB/sec), 11504.0 ms avg latency, 13809.0 max latency. 71955 records sent, 14391.0 records/sec (1.37 MB/sec), 16137.4 ms avg latency, 18541.0 max latency. Thanks for any help! -- Thanks, Ewen
Re: Poor performance running performance test
Where are you running ProducerPerformance in relation to ZK and the Kafka brokers? You should definitely see much higher performance than this. A couple of other things I can think of that might be going wrong: Are all your VMs in the same AZ? Are you storing Kafka data in EBS or local ephemeral storage? If EBS, have you provisioned enough IOPS. On Tue, Jan 27, 2015 at 4:29 PM, Dillian Murphey crackshotm...@gmail.com wrote: I'm a new user/admin to kafka. I'm running a 3 node ZK and a 6 brokers on aws. The performance I'm seeing is shockingly bad. I need some advice! bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test2 5000 100 -1 acks=1 bootstrap.servers=5myloadbalancer:9092 buffer.memory=67108864 batch.size=8196 6097 records sent, 13198.3 records/sec (1.26 MB/sec), 2098.0 ms avg latency, 4306.0 max latency. 71695 records sent, 14339.0 records/sec (1.37 MB/sec), 6658.1 ms avg latency, 9053.0 max latency. 65195 records sent, 13028.6 records/sec (1.24 MB/sec), 11504.0 ms avg latency, 13809.0 max latency. 71955 records sent, 14391.0 records/sec (1.37 MB/sec), 16137.4 ms avg latency, 18541.0 max latency. Thanks for any help! -- Thanks, Ewen
