Hi Radu, I think using JASON at the logtash is not a solution for me, as we cannot use Ruby code as filter in the JASON. We have to place a lot of ruby code inside the Ruby filters. I just want to quote an answer -see below, which I got from logtash ML here which describe what we want to achieve with Ruby filters in Logtash (from a guy from Microsoft) -- in order to balance the performance we have Rsyslog with Logtash:
1. Yes, you can imbed a ruby filter 2. Actually, logstash has a small buffer in the pipeline.rb but does not persist to disk. Some of us have drastically increased this buffer to great benefit when processing large volumes of logs or dealing with bursts and output limitations (like rabbitmq). a. W.r.t. performance, it really depends on what your filters do. We have around 1000 lines of filter logic, but it has a lot of if/elseif ordering so that it escapes as soon as possible. We also do large dictionary lookups (40k entries) and are able to get around 250 msg/sec per CPU core with this "big" config. We scale out indexers as needed. Now how can we do the same in Rsyslog? Rainer: Performance is not an issue for Rsyslog but including support for languages like Ruby might not be possible for Rsyslog. Thanks On Fri, Apr 11, 2014 at 11:28 AM, Radu Gheorghe <[email protected]>wrote: > Hi, > > Rainer - here's where Grok seems to live: > https://github.com/jordansissel/grok > > It is indeed a library, and there are also other implementations (I know of > one in Ruby and one in Java). > > Masoom - I think Richard and David already answered your questions about > what you can use in rsyslog. If that's not enough, please give some > examples of what kind of filtering you need to do. > > Best regards, > Radu > > > On Fri, Apr 11, 2014 at 8:03 AM, Rainer Gerhards > <[email protected]>wrote: > > > I have yet to look into it, but maybe someone knows if grok is a > > stand-alone component. If so, we could probably very easily make it > > available in rsyslog via the new external message modification plugin > > capability. > > > > Anyone in the know (else I'll try to find out)? > > > > Rainer > > > > > > On Fri, Apr 11, 2014 at 5:51 AM, David Lang <[email protected]> wrote: > > > > > On Fri, 11 Apr 2014, masoom alam wrote: > > > > > > Very detailed answer. Thanks!!! > > >> > > >> Since it is related with both Rsyslog and Logtash, thats why I am > asking > > >> here. After your kind guidance, Its now clear that we should use JASON > > >> template in Rsyslog, and then use JASON in logtash. I did not find any > > >> significant difference at the logtash end regarding Grok and JASON, > > except > > >> the word Jason in the filter instead of Grok, am I right? -- I mean as > > for > > >> as the syntax is concerned. For the execution it will definitely have > > >> performance gains, as you suggested. > > >> > > >> Another thing which I think I did not explain well in my email is that > > we > > >> are thinking to place some regex at the Rsyslog end too. Suppose we > have > > >> > > >>> 200 filters defined in Logtash, so will happen that when a log entry > > will > > >>> > > >> arrive at the Logtash, it will have to match it against all the 200 > > >> filters > > >> -- worst case, and/or some thing matches earlier and we compose the > > >> configuration file of Logtash in a way that it escapes. Any ideas how > to > > >> optimize the log deep/fancy parsing at this end? > > >> > > > > > > I don't know about the logstash side, but I suspect that you are > correct. > > > On the rsyslog side, the equivalent would be mmlognorm, and with it the > > > number of rules doesn't matter because they get compiled into a parse > > tree, > > > you go through the log message once. > > > > > > David Lang > > > > > > > > > Once Again thanks Radu. You are very helpful. > > >> > > >> > > >> > > >> > > >> > > >> > > >> On Fri, Apr 11, 2014 at 12:36 AM, Radu Gheorghe > > >> <[email protected]>wrote: > > >> > > >> I've never actually tried this, but I think the best way for > > performance > > >>> is > > >>> to send over TCP, but make the template a JSON with everything > rsyslog > > >>> can > > >>> parse (by default, stuff like severity, date, etc). On the Logstash > > side, > > >>> you'll use the JSON that should parse much faster than grok can parse > > >>> syslog. After that, you'd set the rest of the Logstash filters you > want > > >>> to > > >>> use for fancy processing. > > >>> > > >>> Also, sending over TCP allows you to use rsyslog for buffering, and > if > > >>> you're using in-memory queues (or disk-assisted, assuming those > rarely > > >>> spill out to disk), this means you'll avoid the I/O penalty of > writing > > to > > >>> disks and having Logstash poll from disk periodically. > > >>> > > >>> If you need help with any of those, please write here (or on the > > Logstash > > >>> ML for the Logstash part, people are really helpful there). > > >>> > > >>> Best regards, > > >>> Radu > > >>> > > >>> > > >>> On Thu, Apr 10, 2014 at 6:13 PM, masoom alam <[email protected]> > > >>> wrote: > > >>> > > >>> Is it necessary to fill the templates inside rsyslog so that rsyslog > > >>>> > > >>> should > > >>> > > >>>> write each log source to a separate file for logtash - will be easy > > for > > >>>> > > >>> it > > >>> > > >>>> for parsing? - also due to the reason logrtash has to catch > rsyslog? . > > >>>> > > >>> What > > >>> > > >>>> is the alternative if we are doing extensive parsing in logtash? - > > >>>> simply > > >>>> directing log on to a port and ask logtash to pick it up - match it > > >>>> > > >>> against > > >>> > > >>>> 200 plugins? > > >>>> > > >>>> from phone thus brief. > > >>>> On Apr 10, 2014 5:06 PM, "Radu Gheorghe" < > [email protected]> > > >>>> wrote: > > >>>> > > >>>> Here's an article that explains how to configure squeeze > performance > > >>>>> > > >>>> from a > > >>>> > > >>>>> rsyslog>ES>Kibana setup, and the numbers I got (20-30K EPS on my > > >>>>> > > >>>> good-old > > >>> > > >>>> laptop): http://www.rsyslog.com/performance-tuning-elasticsearch/ > > >>>>> > > >>>>> You also have links there about other articles in this are (that > also > > >>>>> > > >>>> have > > >>>> > > >>>>> config snippets and explanations). > > >>>>> > > >>>>> On Tue, Apr 8, 2014 at 11:34 PM, Josh Bitto < > [email protected]> > > >>>>> wrote: > > >>>>> > > >>>>> If I'm reading this right your saying that you did > > >>>>>> Rsyslog->Elasticsearch->gui? > > >>>>>> > > >>>>>> I've tried installing the rpm on centos and it installs but > > >>>>>> > > >>>>> apparently > > >>> > > >>>> it > > >>>> > > >>>>> doesn't come with a config file and so the daemon starts it errors > > >>>>>> > > >>>>> out > > >>> > > >>>> in > > >>>> > > >>>>> the logs and just shuts down after that. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -----Original Message----- > > >>>>>> From: [email protected] [mailto: > > >>>>>> [email protected]] On Behalf Of Rick Brown > > >>>>>> Sent: Tuesday, April 08, 2014 11:31 AM > > >>>>>> To: rsyslog-users > > >>>>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > >>>>>> > > >>>>> server > > >>> > > >>>> > > >>>>>> Today I've setup my central rsyslog server to replay the logs via > > >>>>>> omudpspoof to a logstash server -> ES. It's already indexing > about > > >>>>>> > > >>>>> twice > > >>>> > > >>>>> as much as just rsyslog -> ES was using the recipe in the first > link > > >>>>>> > > >>>>> below, > > >>>>> > > >>>>>> and I haven't even begun to dig into the scads of plugins > available > > >>>>>> > > >>>>> for > > >>> > > >>>> logstash. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>> > > >>>> http://blog.sematext.com/2013/07/01/recipe-rsyslog- > > >>> elasticsearch-kibana/isagoodplace to start, although you can replace > > >>> the omelasticsearch OM > > >>> > > >>>> with omudpspoof if you want to do logstash. > > >>>>>> > > >>>>>> http://cookbook.logstash.net/recipes/rsyslog-agent/ is a good > place > > >>>>>> > > >>>>> to > > >>> > > >>>> start with rsyslog -> logstash, although I did UDP instead of TCP, > > >>>>>> > > >>>>> and > > >>> > > >>>> used > > >>>>> > > >>>>>> the elasticsearch output module instead of stdout, which is > > >>>>>> > > >>>>> documented > > >>> > > >>>> here: http://cookbook.logstash.net/recipes/central-syslog/ > > >>>>>> > > >>>>>> Good luck to you! Those three links is basically all I needed, > and > > >>>>>> > > >>>>> should > > >>>>> > > >>>>>> set you down the right path, regardless of how your path differs > > from > > >>>>>> > > >>>>> mine > > >>>>> > > >>>>>> ;) > > >>>>>> > > >>>>>> ----- Original Message ----- > > >>>>>> > > >>>>>>> From: "Orangepeel Beef" <[email protected]> > > >>>>>>> To: "rsyslog-users" <[email protected]> > > >>>>>>> Sent: Tuesday, April 8, 2014 2:17:42 PM > > >>>>>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > >>>>>>> > > >>>>>> server > > >>>> > > >>>>> > > >>>>>>> it works, but I find it overly complex for my environment. read: > > >>>>>>> > > >>>>>> I > > >>> > > >>>> don't need it ;) On Apr 8, 2014 11:13 AM, "Josh Bitto" > > >>>>>>> <[email protected]> wrote: > > >>>>>>> > > >>>>>>> I have read about Redis as being the "broker" thoughts? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -----Original Message----- > > >>>>>>>> From: [email protected] [mailto: > > >>>>>>>> [email protected]] On Behalf Of Orangepeel Beef > > >>>>>>>> Sent: Tuesday, April 08, 2014 11:11 AM > > >>>>>>>> To: rsyslog-users > > >>>>>>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > >>>>>>>> server > > >>>>>>>> > > >>>>>>>> I use rsyslog to pipe into sec, and then use logstash file input > > >>>>>>>> > > >>>>>>> to > > >>> > > >>>> index. > > >>>>>>>> could be done without SEC as well. I don't like delivering > > >>>>>>>> > > >>>>>>> syslog > > >>> > > >>>> right into logstash. > > >>>>>>>> On Apr 8, 2014 11:09 AM, "Sphonic" <[email protected]> > > >>>>>>>> > > >>>>>>> wrote: > > >>> > > >>>> > > >>>>>>>> I use rsyslog to send all items to logstash which has a syslog > > >>>>>>>>> listener enabled. > > >>>>>>>>> > > >>>>>>>>> Sent from my iPhone > > >>>>>>>>> > > >>>>>>>>> On 8 Apr 2014, at 18:05, Josh Bitto <[email protected]> > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>> Hello Everyone, > > >>>>>>>>>> > > >>>>>>>>>> I'm wanting to setup a syslog server that combines the three > > >>>>>>>>>> programs > > >>>>>>>>>> > > >>>>>>>>> listed above with rsyslog. Has anyone had any success using > > >>>>>>>>> > > >>>>>>>> this? > > >>> > > >>>> I'm > > >>>>>>>>> running on a CentOS 6.5 and finding adequate instructions on > > >>>>>>>>> > > >>>>>>>> how > > >>> > > >>>> to not only setup all three PLUS rsyslog has been somewhat of a > > >>>>>>>>> challenge. > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> This issue that I run into is on how to get > > >>>>>>>>>> logstash/elasticsearch and > > >>>>>>>>>> > > >>>>>>>>> kibana to talk with rsyslog. Halp meh! Please! > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>> rsyslog mailing list > > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>>>> http://www.rsyslog.com/professional-services/ > > >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhardsNOTE > > >>>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > >>>>>>> > > >>>>>> of > > >>>> > > >>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >>>>>>> DON'T LIKE THAT. > > >>>>>>> > > >>>>>>> > > >>>>>> -- > > >>>>>> Rick Brown > > >>>>>> Office of Information Technology > > >>>>>> Georgia Institute of Technology > > >>>>>> _______________________________________________ > > >>>>>> rsyslog mailing list > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>>> http://www.rsyslog.com/professional-services/ > > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > >>>>>> > > >>>>> WELL: > > >>>> > > >>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > >>>>>> > > >>>>> sites > > >>> > > >>>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > >>>>>> > > >>>>> LIKE > > >>>> > > >>>>> THAT. > > >>>>>> _______________________________________________ > > >>>>>> rsyslog mailing list > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>>> http://www.rsyslog.com/professional-services/ > > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >>>>>> > > >>>>> myriad > > >>>> > > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > >>>>>> > > >>>>> you > > >>> > > >>>> DON'T LIKE THAT. > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> Performance Monitoring * Log Analytics * Search Analytics > > >>>>> Solr & Elasticsearch Support * http://sematext.com/ > > >>>>> _______________________________________________ > > >>>>> rsyslog mailing list > > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>> http://www.rsyslog.com/professional-services/ > > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >>>>> > > >>>> myriad > > >>> > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > >>>>> DON'T LIKE THAT. > > >>>>> > > >>>>> _______________________________________________ > > >>>> rsyslog mailing list > > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>> http://www.rsyslog.com/professional-services/ > > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > >>>> DON'T LIKE THAT. > > >>>> > > >>>> > > >>> > > >>> > > >>> -- > > >>> Performance Monitoring * Log Analytics * Search Analytics > > >>> Solr & Elasticsearch Support * http://sematext.com/ > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com/professional-services/ > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > >>> DON'T LIKE THAT. > > >>> > > >>> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com/professional-services/ > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >> DON'T LIKE THAT. > > >> > > >> _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > > > > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
