Hi, Rainer - here's where Grok seems to live: https://github.com/jordansissel/grok
It is indeed a library, and there are also other implementations (I know of one in Ruby and one in Java). Masoom - I think Richard and David already answered your questions about what you can use in rsyslog. If that's not enough, please give some examples of what kind of filtering you need to do. Best regards, Radu On Fri, Apr 11, 2014 at 8:03 AM, Rainer Gerhards <[email protected]>wrote: > I have yet to look into it, but maybe someone knows if grok is a > stand-alone component. If so, we could probably very easily make it > available in rsyslog via the new external message modification plugin > capability. > > Anyone in the know (else I'll try to find out)? > > Rainer > > > On Fri, Apr 11, 2014 at 5:51 AM, David Lang <[email protected]> wrote: > > > On Fri, 11 Apr 2014, masoom alam wrote: > > > > Very detailed answer. Thanks!!! > >> > >> Since it is related with both Rsyslog and Logtash, thats why I am asking > >> here. After your kind guidance, Its now clear that we should use JASON > >> template in Rsyslog, and then use JASON in logtash. I did not find any > >> significant difference at the logtash end regarding Grok and JASON, > except > >> the word Jason in the filter instead of Grok, am I right? -- I mean as > for > >> as the syntax is concerned. For the execution it will definitely have > >> performance gains, as you suggested. > >> > >> Another thing which I think I did not explain well in my email is that > we > >> are thinking to place some regex at the Rsyslog end too. Suppose we have > >> > >>> 200 filters defined in Logtash, so will happen that when a log entry > will > >>> > >> arrive at the Logtash, it will have to match it against all the 200 > >> filters > >> -- worst case, and/or some thing matches earlier and we compose the > >> configuration file of Logtash in a way that it escapes. Any ideas how to > >> optimize the log deep/fancy parsing at this end? > >> > > > > I don't know about the logstash side, but I suspect that you are correct. > > On the rsyslog side, the equivalent would be mmlognorm, and with it the > > number of rules doesn't matter because they get compiled into a parse > tree, > > you go through the log message once. > > > > David Lang > > > > > > Once Again thanks Radu. You are very helpful. > >> > >> > >> > >> > >> > >> > >> On Fri, Apr 11, 2014 at 12:36 AM, Radu Gheorghe > >> <[email protected]>wrote: > >> > >> I've never actually tried this, but I think the best way for > performance > >>> is > >>> to send over TCP, but make the template a JSON with everything rsyslog > >>> can > >>> parse (by default, stuff like severity, date, etc). On the Logstash > side, > >>> you'll use the JSON that should parse much faster than grok can parse > >>> syslog. After that, you'd set the rest of the Logstash filters you want > >>> to > >>> use for fancy processing. > >>> > >>> Also, sending over TCP allows you to use rsyslog for buffering, and if > >>> you're using in-memory queues (or disk-assisted, assuming those rarely > >>> spill out to disk), this means you'll avoid the I/O penalty of writing > to > >>> disks and having Logstash poll from disk periodically. > >>> > >>> If you need help with any of those, please write here (or on the > Logstash > >>> ML for the Logstash part, people are really helpful there). > >>> > >>> Best regards, > >>> Radu > >>> > >>> > >>> On Thu, Apr 10, 2014 at 6:13 PM, masoom alam <[email protected]> > >>> wrote: > >>> > >>> Is it necessary to fill the templates inside rsyslog so that rsyslog > >>>> > >>> should > >>> > >>>> write each log source to a separate file for logtash - will be easy > for > >>>> > >>> it > >>> > >>>> for parsing? - also due to the reason logrtash has to catch rsyslog? . > >>>> > >>> What > >>> > >>>> is the alternative if we are doing extensive parsing in logtash? - > >>>> simply > >>>> directing log on to a port and ask logtash to pick it up - match it > >>>> > >>> against > >>> > >>>> 200 plugins? > >>>> > >>>> from phone thus brief. > >>>> On Apr 10, 2014 5:06 PM, "Radu Gheorghe" <[email protected]> > >>>> wrote: > >>>> > >>>> Here's an article that explains how to configure squeeze performance > >>>>> > >>>> from a > >>>> > >>>>> rsyslog>ES>Kibana setup, and the numbers I got (20-30K EPS on my > >>>>> > >>>> good-old > >>> > >>>> laptop): http://www.rsyslog.com/performance-tuning-elasticsearch/ > >>>>> > >>>>> You also have links there about other articles in this are (that also > >>>>> > >>>> have > >>>> > >>>>> config snippets and explanations). > >>>>> > >>>>> On Tue, Apr 8, 2014 at 11:34 PM, Josh Bitto <[email protected]> > >>>>> wrote: > >>>>> > >>>>> If I'm reading this right your saying that you did > >>>>>> Rsyslog->Elasticsearch->gui? > >>>>>> > >>>>>> I've tried installing the rpm on centos and it installs but > >>>>>> > >>>>> apparently > >>> > >>>> it > >>>> > >>>>> doesn't come with a config file and so the daemon starts it errors > >>>>>> > >>>>> out > >>> > >>>> in > >>>> > >>>>> the logs and just shuts down after that. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -----Original Message----- > >>>>>> From: [email protected] [mailto: > >>>>>> [email protected]] On Behalf Of Rick Brown > >>>>>> Sent: Tuesday, April 08, 2014 11:31 AM > >>>>>> To: rsyslog-users > >>>>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > >>>>>> > >>>>> server > >>> > >>>> > >>>>>> Today I've setup my central rsyslog server to replay the logs via > >>>>>> omudpspoof to a logstash server -> ES. It's already indexing about > >>>>>> > >>>>> twice > >>>> > >>>>> as much as just rsyslog -> ES was using the recipe in the first link > >>>>>> > >>>>> below, > >>>>> > >>>>>> and I haven't even begun to dig into the scads of plugins available > >>>>>> > >>>>> for > >>> > >>>> logstash. > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> http://blog.sematext.com/2013/07/01/recipe-rsyslog- > >>> elasticsearch-kibana/isagoodplace to start, although you can replace > >>> the omelasticsearch OM > >>> > >>>> with omudpspoof if you want to do logstash. > >>>>>> > >>>>>> http://cookbook.logstash.net/recipes/rsyslog-agent/ is a good place > >>>>>> > >>>>> to > >>> > >>>> start with rsyslog -> logstash, although I did UDP instead of TCP, > >>>>>> > >>>>> and > >>> > >>>> used > >>>>> > >>>>>> the elasticsearch output module instead of stdout, which is > >>>>>> > >>>>> documented > >>> > >>>> here: http://cookbook.logstash.net/recipes/central-syslog/ > >>>>>> > >>>>>> Good luck to you! Those three links is basically all I needed, and > >>>>>> > >>>>> should > >>>>> > >>>>>> set you down the right path, regardless of how your path differs > from > >>>>>> > >>>>> mine > >>>>> > >>>>>> ;) > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>> > >>>>>>> From: "Orangepeel Beef" <[email protected]> > >>>>>>> To: "rsyslog-users" <[email protected]> > >>>>>>> Sent: Tuesday, April 8, 2014 2:17:42 PM > >>>>>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > >>>>>>> > >>>>>> server > >>>> > >>>>> > >>>>>>> it works, but I find it overly complex for my environment. read: > >>>>>>> > >>>>>> I > >>> > >>>> don't need it ;) On Apr 8, 2014 11:13 AM, "Josh Bitto" > >>>>>>> <[email protected]> wrote: > >>>>>>> > >>>>>>> I have read about Redis as being the "broker" thoughts? > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: [email protected] [mailto: > >>>>>>>> [email protected]] On Behalf Of Orangepeel Beef > >>>>>>>> Sent: Tuesday, April 08, 2014 11:11 AM > >>>>>>>> To: rsyslog-users > >>>>>>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > >>>>>>>> server > >>>>>>>> > >>>>>>>> I use rsyslog to pipe into sec, and then use logstash file input > >>>>>>>> > >>>>>>> to > >>> > >>>> index. > >>>>>>>> could be done without SEC as well. I don't like delivering > >>>>>>>> > >>>>>>> syslog > >>> > >>>> right into logstash. > >>>>>>>> On Apr 8, 2014 11:09 AM, "Sphonic" <[email protected]> > >>>>>>>> > >>>>>>> wrote: > >>> > >>>> > >>>>>>>> I use rsyslog to send all items to logstash which has a syslog > >>>>>>>>> listener enabled. > >>>>>>>>> > >>>>>>>>> Sent from my iPhone > >>>>>>>>> > >>>>>>>>> On 8 Apr 2014, at 18:05, Josh Bitto <[email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hello Everyone, > >>>>>>>>>> > >>>>>>>>>> I'm wanting to setup a syslog server that combines the three > >>>>>>>>>> programs > >>>>>>>>>> > >>>>>>>>> listed above with rsyslog. Has anyone had any success using > >>>>>>>>> > >>>>>>>> this? > >>> > >>>> I'm > >>>>>>>>> running on a CentOS 6.5 and finding adequate instructions on > >>>>>>>>> > >>>>>>>> how > >>> > >>>> to not only setup all three PLUS rsyslog has been somewhat of a > >>>>>>>>> challenge. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> This issue that I run into is on how to get > >>>>>>>>>> logstash/elasticsearch and > >>>>>>>>>> > >>>>>>>>> kibana to talk with rsyslog. Halp meh! Please! > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > >>>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >>>>>>> > >>>>>> of > >>>> > >>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>>>>> DON'T LIKE THAT. > >>>>>>> > >>>>>>> > >>>>>> -- > >>>>>> Rick Brown > >>>>>> Office of Information Technology > >>>>>> Georgia Institute of Technology > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com/professional-services/ > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > >>>>>> > >>>>> WELL: > >>>> > >>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > >>>>>> > >>>>> sites > >>> > >>>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > >>>>>> > >>>>> LIKE > >>>> > >>>>> THAT. > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com/professional-services/ > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>> > >>>>> myriad > >>>> > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>>>> > >>>>> you > >>> > >>>> DON'T LIKE THAT. > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Performance Monitoring * Log Analytics * Search Analytics > >>>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com/professional-services/ > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>> > >>>> myriad > >>> > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>>> DON'T LIKE THAT. > >>>>> > >>>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>>> > >>> > >>> > >>> -- > >>> Performance Monitoring * Log Analytics * Search Analytics > >>> Solr & Elasticsearch Support * http://sematext.com/ > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >>> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
