I was running the performance command from a virtual box server, so that seems like it was part of the problem. I'm getting better results running this on a server on aws, but that's kind of expected. Can you look at these results, and comment on the occasional warning I see? I appreciate it!
1220375 records sent, 243928.6 records/sec (23.26 MB/sec), 2111.5 ms avg latency, 4435.0 max latency. 1195090 records sent, 239018.0 records/sec (22.79 MB/sec), 2203.1 ms avg latency, 4595.0 max latency. 1257165 records sent, 251433.0 records/sec (23.98 MB/sec), 2172.6 ms avg latency, 4525.0 max latency. 1230981 records sent, 246196.2 records/sec (23.48 MB/sec), 2173.5 ms avg latency, 4465.0 max latency. [2015-01-28 07:19:07,274] WARN Error in I/O with <myawsloadbalancer>(org.apache.kafka.common.network.Selector) java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 1090689 records sent, 218137.8 records/sec (20.80 MB/sec), 2413.6 ms avg latency, 4829.0 max latency. On Tue, Jan 27, 2015 at 7:37 PM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > Where are you running ProducerPerformance in relation to ZK and the Kafka > brokers? You should definitely see much higher performance than this. > > A couple of other things I can think of that might be going wrong: Are all > your VMs in the same AZ? Are you storing Kafka data in EBS or local > ephemeral storage? If EBS, have you provisioned enough IOPS. > > > On Tue, Jan 27, 2015 at 4:29 PM, Dillian Murphey <crackshotm...@gmail.com> > wrote: > > > I'm a new user/admin to kafka. I'm running a 3 node ZK and a 6 brokers on > > aws. > > > > The performance I'm seeing is shockingly bad. I need some advice! > > > > bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance > > test2 50000000 100 -1 acks=1 bootstrap.servers=5<myloadbalancer>:9092 > > buffer.memory=67108864 batch.size=8196 > > > > > > > > > > 6097 records sent, 13198.3 records/sec (1.26 MB/sec), 2098.0 ms avg > > latency, 4306.0 max latency. > > 71695 records sent, 14339.0 records/sec (1.37 MB/sec), 6658.1 ms avg > > latency, 9053.0 max latency. > > 65195 records sent, 13028.6 records/sec (1.24 MB/sec), 11504.0 ms avg > > latency, 13809.0 max latency. > > 71955 records sent, 14391.0 records/sec (1.37 MB/sec), 16137.4 ms avg > > latency, 18541.0 max latency. > > > > Thanks for any help! > > > > > > -- > Thanks, > Ewen >
