Very detailed answer. Thanks!!! Since it is related with both Rsyslog and Logtash, thats why I am asking here. After your kind guidance, Its now clear that we should use JASON template in Rsyslog, and then use JASON in logtash. I did not find any significant difference at the logtash end regarding Grok and JASON, except the word Jason in the filter instead of Grok, am I right? -- I mean as for as the syntax is concerned. For the execution it will definitely have performance gains, as you suggested.
Another thing which I think I did not explain well in my email is that we are thinking to place some regex at the Rsyslog end too. Suppose we have >200 filters defined in Logtash, so will happen that when a log entry will arrive at the Logtash, it will have to match it against all the 200 filters -- worst case, and/or some thing matches earlier and we compose the configuration file of Logtash in a way that it escapes. Any ideas how to optimize the log deep/fancy parsing at this end? Once Again thanks Radu. You are very helpful. On Fri, Apr 11, 2014 at 12:36 AM, Radu Gheorghe <[email protected]>wrote: > I've never actually tried this, but I think the best way for performance is > to send over TCP, but make the template a JSON with everything rsyslog can > parse (by default, stuff like severity, date, etc). On the Logstash side, > you'll use the JSON that should parse much faster than grok can parse > syslog. After that, you'd set the rest of the Logstash filters you want to > use for fancy processing. > > Also, sending over TCP allows you to use rsyslog for buffering, and if > you're using in-memory queues (or disk-assisted, assuming those rarely > spill out to disk), this means you'll avoid the I/O penalty of writing to > disks and having Logstash poll from disk periodically. > > If you need help with any of those, please write here (or on the Logstash > ML for the Logstash part, people are really helpful there). > > Best regards, > Radu > > > On Thu, Apr 10, 2014 at 6:13 PM, masoom alam <[email protected]> > wrote: > > > Is it necessary to fill the templates inside rsyslog so that rsyslog > should > > write each log source to a separate file for logtash - will be easy for > it > > for parsing? - also due to the reason logrtash has to catch rsyslog? . > What > > is the alternative if we are doing extensive parsing in logtash? - simply > > directing log on to a port and ask logtash to pick it up - match it > against > > 200 plugins? > > > > from phone thus brief. > > On Apr 10, 2014 5:06 PM, "Radu Gheorghe" <[email protected]> > > wrote: > > > > > Here's an article that explains how to configure squeeze performance > > from a > > > rsyslog>ES>Kibana setup, and the numbers I got (20-30K EPS on my > good-old > > > laptop): http://www.rsyslog.com/performance-tuning-elasticsearch/ > > > > > > You also have links there about other articles in this are (that also > > have > > > config snippets and explanations). > > > > > > On Tue, Apr 8, 2014 at 11:34 PM, Josh Bitto <[email protected]> > > > wrote: > > > > > > > If I'm reading this right your saying that you did > > > > Rsyslog->Elasticsearch->gui? > > > > > > > > I've tried installing the rpm on centos and it installs but > apparently > > it > > > > doesn't come with a config file and so the daemon starts it errors > out > > in > > > > the logs and just shuts down after that. > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: [email protected] [mailto: > > > > [email protected]] On Behalf Of Rick Brown > > > > Sent: Tuesday, April 08, 2014 11:31 AM > > > > To: rsyslog-users > > > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > server > > > > > > > > Today I've setup my central rsyslog server to replay the logs via > > > > omudpspoof to a logstash server -> ES. It's already indexing about > > twice > > > > as much as just rsyslog -> ES was using the recipe in the first link > > > below, > > > > and I haven't even begun to dig into the scads of plugins available > for > > > > logstash. > > > > > > > > > > > > > > http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/isagoodplace > to start, although you can replace the omelasticsearch OM > > > > with omudpspoof if you want to do logstash. > > > > > > > > http://cookbook.logstash.net/recipes/rsyslog-agent/ is a good place > to > > > > start with rsyslog -> logstash, although I did UDP instead of TCP, > and > > > used > > > > the elasticsearch output module instead of stdout, which is > documented > > > > here: http://cookbook.logstash.net/recipes/central-syslog/ > > > > > > > > Good luck to you! Those three links is basically all I needed, and > > > should > > > > set you down the right path, regardless of how your path differs from > > > mine > > > > ;) > > > > > > > > ----- Original Message ----- > > > > > From: "Orangepeel Beef" <[email protected]> > > > > > To: "rsyslog-users" <[email protected]> > > > > > Sent: Tuesday, April 8, 2014 2:17:42 PM > > > > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > server > > > > > > > > > > it works, but I find it overly complex for my environment. read: > I > > > > > don't need it ;) On Apr 8, 2014 11:13 AM, "Josh Bitto" > > > > > <[email protected]> wrote: > > > > > > > > > > > I have read about Redis as being the "broker" thoughts? > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: [email protected] [mailto: > > > > > > [email protected]] On Behalf Of Orangepeel Beef > > > > > > Sent: Tuesday, April 08, 2014 11:11 AM > > > > > > To: rsyslog-users > > > > > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > > > > > server > > > > > > > > > > > > I use rsyslog to pipe into sec, and then use logstash file input > to > > > > > > index. > > > > > > could be done without SEC as well. I don't like delivering > syslog > > > > > > right into logstash. > > > > > > On Apr 8, 2014 11:09 AM, "Sphonic" <[email protected]> > wrote: > > > > > > > > > > > > > I use rsyslog to send all items to logstash which has a syslog > > > > > > > listener enabled. > > > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > > > On 8 Apr 2014, at 18:05, Josh Bitto <[email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > > > > > I'm wanting to setup a syslog server that combines the three > > > > > > > > programs > > > > > > > listed above with rsyslog. Has anyone had any success using > this? > > > > > > > I'm > > > > > > > running on a CentOS 6.5 and finding adequate instructions on > how > > > > > > > to not only setup all three PLUS rsyslog has been somewhat of a > > > > > > > challenge. > > > > > > > > > > > > > > > > This issue that I run into is on how to get > > > > > > > > logstash/elasticsearch and > > > > > > > kibana to talk with rsyslog. Halp meh! Please! > > > > > > > > > > > > > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com/professional-services/ > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of > > > > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > > > DON'T LIKE THAT. > > > > > > > > > > > > > -- > > > > Rick Brown > > > > Office of Information Technology > > > > Georgia Institute of Technology > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > WELL: > > > > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites > > > > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE > > > > THAT. > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > > DON'T LIKE THAT. > > > > > > > > > > > > > > > > -- > > > Performance Monitoring * Log Analytics * Search Analytics > > > Solr & Elasticsearch Support * http://sematext.com/ > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > > > > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
