Out of curiosity - what is "Golang's Kafka interface"? Are you referring to Sarama client?
On Sun, Oct 9, 2016 at 9:28 AM, Christopher Stelly <cdste...@gmail.com> wrote: > Hello, > > The last thread available regarding 10GBe is about 2 years old, with no > obvious recommendations on tuning. > > Is there a more complex tuning guide than the example production config > available on Kafka's main site? Anything other than the list of possible > configs? > > I currently have access to a rather substantial academic cluster to test > on, including multiple machines with the following hardware: > > 10GBe NICs > 250GB RAM each > SSDs on each > (also, optional access to single NVMe) > > Using Golang's Kafka interface, I can only seem to get about 80MB/s on the > producer pushing to logs on the localhost, using no replication and reading > from/logging to SSD. If it helps, I can post my configs. I've tried > fiddling with a bunch of broker configs as well as producer configs, > raising the memory limits, max message size, io&network threads etc. > > Since the last post from 2014 indicates that there is no public > benchmarking for 10GBe, I'd be happy to run benchmarks /publish results on > this hardware if we can get it tuned up properly. > > What kind of broker/producer/consumer settings would you recommend? > > Thanks! > - chris -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
