Thanks Guaozhang. I have a follow-up question.
Say if I can push 1M/s messages (100 size) into 2 node cluster. Is it safe to say I should be able to produce more or less 2M/s messages into 4 node cluster? On Tue, Jul 8, 2014 at 6:00 PM, Guozhang Wang <wangg...@gmail.com> wrote: > The second script is using the new producer which allows sending multiple > in-flight requests whereas the first script use the current (old) producer > which sends only one request at a time to a certain broker. > > The new producer will be officially released in 0.8.2. > > Guozhang > > > On Tue, Jul 8, 2014 at 2:22 PM, Chen Song <chen.song...@gmail.com> wrote: > > > While testing kafka producer performance, I found 2 testing scripts. > > > > 1) performance testing script in kafka distribution > > > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages > > 10000000 --topic test --threads 10 --message-size 100 --batch-size 10000 > > --compression-codec 1 > > > > 2) performance testing script mentioned in > > > > > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > > > bin/kafka-run-class.sh > > org.apache.kafka.clients.tools.ProducerPerformance test6 50000000 100 > > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > > buffer.memory=67108864 batch.size=8196 > > > > based on org.apache.kafka.clients.producer.Producer. > > > > > > On my local testing environment, > > > > For 1), the loading is ~15M/s and 150,000 msgs/s for mesage size = 100, > no > > matter how I adjust batch size and number of loading threads. > > For 2), it can load up to 90M/s and 1,000,000 records/s. > > > > I am wondering what makes these 2 implementations perform so differently? > > > > -- > > Chen Song > > > > > > -- > -- Guozhang > -- Chen Song