I've never actually tried this, but I think the best way for performance is to send over TCP, but make the template a JSON with everything rsyslog can parse (by default, stuff like severity, date, etc). On the Logstash side, you'll use the JSON that should parse much faster than grok can parse syslog. After that, you'd set the rest of the Logstash filters you want to use for fancy processing.
Also, sending over TCP allows you to use rsyslog for buffering, and if you're using in-memory queues (or disk-assisted, assuming those rarely spill out to disk), this means you'll avoid the I/O penalty of writing to disks and having Logstash poll from disk periodically. If you need help with any of those, please write here (or on the Logstash ML for the Logstash part, people are really helpful there). Best regards, Radu On Thu, Apr 10, 2014 at 6:13 PM, masoom alam <[email protected]> wrote: > Is it necessary to fill the templates inside rsyslog so that rsyslog should > write each log source to a separate file for logtash - will be easy for it > for parsing? - also due to the reason logrtash has to catch rsyslog? . What > is the alternative if we are doing extensive parsing in logtash? - simply > directing log on to a port and ask logtash to pick it up - match it against > 200 plugins? > > from phone thus brief. > On Apr 10, 2014 5:06 PM, "Radu Gheorghe" <[email protected]> > wrote: > > > Here's an article that explains how to configure squeeze performance > from a > > rsyslog>ES>Kibana setup, and the numbers I got (20-30K EPS on my good-old > > laptop): http://www.rsyslog.com/performance-tuning-elasticsearch/ > > > > You also have links there about other articles in this are (that also > have > > config snippets and explanations). > > > > On Tue, Apr 8, 2014 at 11:34 PM, Josh Bitto <[email protected]> > > wrote: > > > > > If I'm reading this right your saying that you did > > > Rsyslog->Elasticsearch->gui? > > > > > > I've tried installing the rpm on centos and it installs but apparently > it > > > doesn't come with a config file and so the daemon starts it errors out > in > > > the logs and just shuts down after that. > > > > > > > > > > > > > > > -----Original Message----- > > > From: [email protected] [mailto: > > > [email protected]] On Behalf Of Rick Brown > > > Sent: Tuesday, April 08, 2014 11:31 AM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server > > > > > > Today I've setup my central rsyslog server to replay the logs via > > > omudpspoof to a logstash server -> ES. It's already indexing about > twice > > > as much as just rsyslog -> ES was using the recipe in the first link > > below, > > > and I haven't even begun to dig into the scads of plugins available for > > > logstash. > > > > > > > > > http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/isagood > place to start, although you can replace the omelasticsearch OM > > > with omudpspoof if you want to do logstash. > > > > > > http://cookbook.logstash.net/recipes/rsyslog-agent/ is a good place to > > > start with rsyslog -> logstash, although I did UDP instead of TCP, and > > used > > > the elasticsearch output module instead of stdout, which is documented > > > here: http://cookbook.logstash.net/recipes/central-syslog/ > > > > > > Good luck to you! Those three links is basically all I needed, and > > should > > > set you down the right path, regardless of how your path differs from > > mine > > > ;) > > > > > > ----- Original Message ----- > > > > From: "Orangepeel Beef" <[email protected]> > > > > To: "rsyslog-users" <[email protected]> > > > > Sent: Tuesday, April 8, 2014 2:17:42 PM > > > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > server > > > > > > > > it works, but I find it overly complex for my environment. read: I > > > > don't need it ;) On Apr 8, 2014 11:13 AM, "Josh Bitto" > > > > <[email protected]> wrote: > > > > > > > > > I have read about Redis as being the "broker" thoughts? > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: [email protected] [mailto: > > > > > [email protected]] On Behalf Of Orangepeel Beef > > > > > Sent: Tuesday, April 08, 2014 11:11 AM > > > > > To: rsyslog-users > > > > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > > > > server > > > > > > > > > > I use rsyslog to pipe into sec, and then use logstash file input to > > > > > index. > > > > > could be done without SEC as well. I don't like delivering syslog > > > > > right into logstash. > > > > > On Apr 8, 2014 11:09 AM, "Sphonic" <[email protected]> wrote: > > > > > > > > > > > I use rsyslog to send all items to logstash which has a syslog > > > > > > listener enabled. > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > On 8 Apr 2014, at 18:05, Josh Bitto <[email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > > > I'm wanting to setup a syslog server that combines the three > > > > > > > programs > > > > > > listed above with rsyslog. Has anyone had any success using this? > > > > > > I'm > > > > > > running on a CentOS 6.5 and finding adequate instructions on how > > > > > > to not only setup all three PLUS rsyslog has been somewhat of a > > > > > > challenge. > > > > > > > > > > > > > > This issue that I run into is on how to get > > > > > > > logstash/elasticsearch and > > > > > > kibana to talk with rsyslog. Halp meh! Please! > > > > > > > > > > > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of > > > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > > DON'T LIKE THAT. > > > > > > > > > > -- > > > Rick Brown > > > Office of Information Technology > > > Georgia Institute of Technology > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: > > > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > > > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE > > > THAT. > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > > > > > > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
