Thanks for sharing the results. Was the topic created with replication factor of 2? Could you test acks=-1 as well?
Thanks, Jun On Mon, Nov 25, 2013 at 4:30 AM, Magnus Edenhill <mag...@edenhill.se> wrote: > The following tests were using a single producer application > (examples/rdkafka_performance): > > * Test1: 2 brokers, 2 partitions, required.acks=2, 100 byte messages: > 850000 messages/second, 85 MB/second > > * Test2: 1 broker, 1 partition, required.acks=0, 100 byte messages: 710000 > messages/second, 71 MB/second > > * Test3: 2 broker2, 2 partitions, required.acks=2, 100 byte messages, > snappy compression: 300000 messages/second, 30 MB/second > > * Test4: 2 broker2, 2 partitions, required.acks=2, 100 byte messages, gzip > compression: 230000 messages/second, 23 MB/second > > > log.flush broker configuration was increased to avoid the disk being the > bottleneck. > > > /Magnus > > > > 2013/11/24 Neha Narkhede <neha.narkh...@gmail.com> > > > So, a single producer'a throughput is 80 MB/s? That seems pretty high. > What > > was the number of acks setting? Thanks for sharing these numbers. > > > > On Sunday, November 24, 2013, Magnus Edenhill wrote: > > > > > Hi Neha, > > > > > > these tests were done using 100 byte messages. More information about > the > > > producer performance tests can be found here: > > > > > > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers > > > > > > The tests are indicative at best and in no way scientific, but I must > say > > > that the Kafka broker performance is impressive. > > > > > > Regards, > > > Magnus > > > > > > > > > > > > 2013/11/22 Neha Narkhede <neha.narkh...@gmail.com <javascript:;>> > > > > > > > Thanks for sharing this! What is the message size for the throughput > > > > numbers stated below? > > > > > > > > Thanks, > > > > Neha > > > > On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <mag...@edenhill.se > > <javascript:;>> > > > wrote: > > > > > > > > > This announces the 0.8.0 release of librdkafka - The Apache Kafka > > > client > > > > C > > > > > library - now with 0.8 protocol support. > > > > > > > > > > Features: > > > > > * Producer (~800K msgs/s) > > > > > * Consumer (~3M msgs/s) > > > > > * Compression (Snappy, gzip) > > > > > * Proper failover and leader re-election support - no message is > ever > > > > lost. > > > > > * Configuration properties compatible with official Apache Kafka. > > > > > * Stabilized ABI-safe API > > > > > * Mainline Debian package submitted > > > > > * Production quality > > > > > > > > > > > > > > > Home: > > > > > https://github.com/edenhill/librdkafka > > > > > > > > > > Introduction and performance numbers: > > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md > > > > > > > > > > Have fun. > > > > > > > > > > Regards, > > > > > Magnus > > > > > > > > > > P.S. > > > > > Check out Wikimedia Foundation's varnishkafka daemon for a use > case - > > > > > varnish log forwarding over Kafka: > > > > > https://github.com/wikimedia/varnishkafka > > > > > > > > > > > > > > >
