Roger, Consider using rsyslog with omkafka. rsyslog rocks! And it's pretty popular, too - http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/ Oh, and it's FAST - some numbers and charts with an older version from 1 year ago: http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-output-performance/
Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Mar 5, 2015 at 3:53 AM, Roger Hoover <roger.hoo...@gmail.com> wrote: > I think my test include some grok filters and file input so it's not > necessarily bottlenecked on Kafka producer. > > On Thu, Mar 5, 2015 at 12:37 AM, Vineet Mishra <clearmido...@gmail.com> > wrote: > > > Hey Roger, > > > > As per your stats you have around 5k msg/s of size 42 bytes > > > > 5000msgs * 42 byte = 210000 = ~ 205kbps > > > > while I am getting around 500 msgs of around 350 bytes. > > > > 500msgs * 350 = 175000 = ~ 170kbps > > > > Which is even collectively very degrading write throughput. > > > > It seems this rate of publishing of data is quiet slow with Logstash, as > > the Kafka claims for write throughput to be around 50Mbps(please refer > this > > link http://kafka.apache.org/07/performance.html ) which obviously be > the > > best performance with the config. tuning and infact I have myself have > > achieved around 6-10Mbps of data transfer while publishing to the kafka > > using Kafka client. > > > > I guess this latency is their at Logstash end and perhaps we need to look > > for an alternative to the same. > > > > Do let me know your observation and understanding as well. > > > > Thanks! > > > > > > > > On Thu, Mar 5, 2015 at 1:13 PM, Roger Hoover <roger.hoo...@gmail.com> > > wrote: > > > > > Seeing around 5k msgs/s. The messages are small (average 42 bytes > after > > > snappy compression) > > > > > > On Wed, Mar 4, 2015 at 11:34 PM, Vineet Mishra <clearmido...@gmail.com > > > > > wrote: > > > > > > > Hi Roger, > > > > > > > > I have already enabled the snappy, the throughput which I have > > mentioned > > > is > > > > after only. > > > > > > > > Could you mention what's the throughput you have reaching. > > > > > > > > Thanks! > > > > > > > > On Thu, Mar 5, 2015 at 12:56 PM, Roger Hoover < > roger.hoo...@gmail.com> > > > > wrote: > > > > > > > > > Hi Vineet, > > > > > > > > > > Try enabling compression. That improves throughput 3-4x usually > for > > > me. > > > > > Also, you can use async mode if you're willing to trade some chance > > of > > > > > dropping messages for more throughput. > > > > > > > > > > kafka { > > > > > > > > > > codec => 'json' > > > > > > > > > > broker_list => "localhost:9092" > > > > > > > > > > topic_id => "blah" > > > > > > > > > > compression_codec => "snappy" > > > > > > > > > > request_required_acks => 1 > > > > > > > > > > producer_type => "async" > > > > > > > > > > } > > > > > > > > > > On Wed, Mar 4, 2015 at 11:03 PM, Vineet Mishra < > > clearmido...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I am having a Logstash Forwarder which is publishing events to > > Kafka, > > > > but > > > > > > as I can see the rate at which the events is published to Kafka > is > > > > really > > > > > > very slow. > > > > > > > > > > > > With the reference to some links I could get the Kafka Publish > > > > throughput > > > > > > reaching in 50-60Mbs per second but in my case I am hardly > getting > > > few > > > > > Kbs' > > > > > > of throughput. > > > > > > > > > > > > Looking for Expert intervention to the same. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > >
