Actually I was thinking, since my payloads sizes are larger, it would make sense to flush to disk more often right? Since I only have 4GB ram on the server.
What effect does the poll time to check if a flush is required have? Is it an expensive operation, i.e. does it walk the entire collection and determine the size? On Sat, May 5, 2012 at 3:22 PM, S Ahmed <sahmed1...@gmail.com> wrote: > These are the performance #'s I got on a dedicated server, everthing > (zookeeper, kafka server) is running on the same server: > > bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092 > --messages 1000000 --topic test --threads 10 --message-size=30720 > [2012-05-05 14:03:31,344] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,345] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,348] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,352] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,353] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,354] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,354] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,354] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,354] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,354] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,354] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,354] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,354] INFO Creating sync producer for broker id = 0 at > localhost:9092 (kafka.producer.ProducerPool) > [2012-05-05 14:03:31,357] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,358] INFO Messages per thread = 500 > (kafka.tools.ProducerPerformance$SyncProducerThread) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing > (kafka.producer.SyncProducer) > [2012-05-05 14:07:48,431] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:07:48,431] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:07:48,885] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:07:48,885] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:07:50,600] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:07:50,600] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:07:53,427] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:07:53,427] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:07:53,429] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:07:53,429] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:07:55,643] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:07:55,643] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:08:21,173] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:08:21,173] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:08:23,758] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:08:23,758] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:08:23,922] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:08:23,922] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:08:23,952] INFO Closing all sync producers > (kafka.producer.ProducerPool) > [2012-05-05 14:08:23,952] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:08:23,953] INFO Total Num Messages: 1000000 bytes: > 30720000000 in 292.713 secs (kafka.tools.ProducerPerformance$) > [2012-05-05 14:08:23,981] INFO Messages/sec: 3416.3156 > (kafka.tools.ProducerPerformance$) > [2012-05-05 14:08:23,981] INFO MB/sec: 100.0874 > (kafka.tools.ProducerPerformance$) > > I also noticed when running top, that the java process was pretty much at > 100% and my 4GB of ram were used up. > When I tried with a message size of 100KB it crashed with a OOM error. > > Any comments on how I could tweak the configuration to get better results? > My use case is with 2-30KB sizes, but some could be a bit higher but no > larger than 100KB. > > I changed the default flush to 10seconds, and check for flush time to > 5seconds, and I got these #'s: > > [2012-05-05 14:21:52,782] INFO Messages/sec: 3412.6199 > (kafka.tools.ProducerPerformance$) > [2012-05-05 14:21:52,782] INFO MB/sec: 99.9791 > (kafka.tools.ProducerPerformance$) > > i.e. nothing really changed. > > When I lowered the message size, I got: > > bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092 > --messages 1000000 --topic test3 --threads 10 --message-size=100 > > [2012-05-05 14:13:28,553] INFO Disconnecting from localhost:9092 > (kafka.producer.SyncProducer) > [2012-05-05 14:13:28,553] INFO Total Num Messages: 1000000 bytes: > 100000000 in 5.512 secs (kafka.tools.ProducerPerformance$) > [2012-05-05 14:13:28,560] INFO Messages/sec: 181422.3512 > (kafka.tools.ProducerPerformance$) > [2012-05-05 14:13:28,561] INFO MB/sec: 17.3018 > (kafka.tools.ProducerPerformance$) > > >
