On Fri, 1 Mar 2013 10:20:28 +0000 Ben Bradley <[email protected]> wrote:
> Hi everyone > > So I've got Rsyslog happily transmitting log messages over the network to > Logstash. I have disk assisted queueing on the rsyslog log "clients". > Sometimes I don't think Logstash can keep up on the other end, it blocks > because it can't get data into ElasticSearch fast enough. I've not got > logstash using elasticsearch_http which bulks the messages into ElasticSearch > so we'll see how that goes. > But this is for relatively low logging volume (15/sec to 40/sec). > > I probably need some sort of queueing system on the receiving end, in front > of Logstash. We will probably be increasing our log volume gradually to about > 5x the current level. Though I probably also need some queueing between > Logstash and ElasticSearch. > > I could ditch Logstash altogether and have Rsyslog pumping directly into > ElasticSearch with omelasticsearch. But I'm not sure about the reliability of > this (mention of ElasticSearch crashes) and how I need to structure the > schema to maintain compatibility with Kibana. Also Logstash's filtering and > pattern matching is (unfortunately) much easier to get working at this point > (though I have got Rsyslog doing this in testing). > > Or I could install Rsyslog in front of Logstash and use Rsyslog's queue > options (and zip compression). > But what would be the best output module to use with Rsyslog to then send to > the local Logstash? > Using omfwd to send via TCP to Logstash on the same machine seems a bit > wasteful so I looked at omuxsock but then I'm not sure Logstash can receive > through a socket. > > Or is this the point where I need some sort of proper queueing system. What > would be great is if there was something that could just receive the TCP > packets from Rsyslog and queue them up. Then Logstash just reads from the > queue when it can. Is that what systems like 0mq and AMQP do? > > Are there any queueing systems in particular that are best to use for > queueing messages from Rsyslog? > > Cheers, Ben > Ah... just noticed rsyslog-zmq3 and zeromq3 RPMs in the repo! Do they do what I think? Only CentOS 5 though. I guess it needs to be installed as an output module in order to send to a zeromq socket. Cheers, Ben _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
