You can do ruby plugins in 8.3.0. Sent from phone, thus brief. Am 11.04.2014 09:02 schrieb "masoom alam" <[email protected]>:
> Hi Radu, > > I think using JASON at the logtash is not a solution for me, as we cannot > use Ruby code as filter in the JASON. We have to place a lot of ruby code > inside the Ruby filters. I just want to quote an answer -see below, which I > got from logtash ML here which describe what we want to achieve with Ruby > filters in Logtash (from a guy from Microsoft) -- in order to balance the > performance we have Rsyslog with Logtash: > > 1. Yes, you can imbed a ruby filter > > 2. Actually, logstash has a small buffer in the pipeline.rb but does > not persist to disk. Some of us have drastically increased this buffer to > great benefit when processing large volumes of logs or dealing with bursts > and output limitations (like rabbitmq). > > a. W.r.t. performance, it really depends on what your filters do. We > have around 1000 lines of filter logic, but it has a lot of if/elseif > ordering so that it escapes as soon as possible. We also do large > dictionary lookups (40k entries) and are able to get around 250 msg/sec per > CPU core with this "big" config. We scale out indexers as needed. > > Now how can we do the same in Rsyslog? > > Rainer: Performance is not an issue for Rsyslog but including support for > languages like Ruby might not be possible for Rsyslog. > > > Thanks > > > > > On Fri, Apr 11, 2014 at 11:28 AM, Radu Gheorghe > <[email protected]>wrote: > > > Hi, > > > > Rainer - here's where Grok seems to live: > > https://github.com/jordansissel/grok > > > > It is indeed a library, and there are also other implementations (I know > of > > one in Ruby and one in Java). > > > > Masoom - I think Richard and David already answered your questions about > > what you can use in rsyslog. If that's not enough, please give some > > examples of what kind of filtering you need to do. > > > > Best regards, > > Radu > > > > > > On Fri, Apr 11, 2014 at 8:03 AM, Rainer Gerhards > > <[email protected]>wrote: > > > > > I have yet to look into it, but maybe someone knows if grok is a > > > stand-alone component. If so, we could probably very easily make it > > > available in rsyslog via the new external message modification plugin > > > capability. > > > > > > Anyone in the know (else I'll try to find out)? > > > > > > Rainer > > > > > > > > > On Fri, Apr 11, 2014 at 5:51 AM, David Lang <[email protected]> wrote: > > > > > > > On Fri, 11 Apr 2014, masoom alam wrote: > > > > > > > > Very detailed answer. Thanks!!! > > > >> > > > >> Since it is related with both Rsyslog and Logtash, thats why I am > > asking > > > >> here. After your kind guidance, Its now clear that we should use > JASON > > > >> template in Rsyslog, and then use JASON in logtash. I did not find > any > > > >> significant difference at the logtash end regarding Grok and JASON, > > > except > > > >> the word Jason in the filter instead of Grok, am I right? -- I mean > as > > > for > > > >> as the syntax is concerned. For the execution it will definitely > have > > > >> performance gains, as you suggested. > > > >> > > > >> Another thing which I think I did not explain well in my email is > that > > > we > > > >> are thinking to place some regex at the Rsyslog end too. Suppose we > > have > > > >> > > > >>> 200 filters defined in Logtash, so will happen that when a log > entry > > > will > > > >>> > > > >> arrive at the Logtash, it will have to match it against all the 200 > > > >> filters > > > >> -- worst case, and/or some thing matches earlier and we compose the > > > >> configuration file of Logtash in a way that it escapes. Any ideas > how > > to > > > >> optimize the log deep/fancy parsing at this end? > > > >> > > > > > > > > I don't know about the logstash side, but I suspect that you are > > correct. > > > > On the rsyslog side, the equivalent would be mmlognorm, and with it > the > > > > number of rules doesn't matter because they get compiled into a parse > > > tree, > > > > you go through the log message once. > > > > > > > > David Lang > > > > > > > > > > > > Once Again thanks Radu. You are very helpful. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> On Fri, Apr 11, 2014 at 12:36 AM, Radu Gheorghe > > > >> <[email protected]>wrote: > > > >> > > > >> I've never actually tried this, but I think the best way for > > > performance > > > >>> is > > > >>> to send over TCP, but make the template a JSON with everything > > rsyslog > > > >>> can > > > >>> parse (by default, stuff like severity, date, etc). On the Logstash > > > side, > > > >>> you'll use the JSON that should parse much faster than grok can > parse > > > >>> syslog. After that, you'd set the rest of the Logstash filters you > > want > > > >>> to > > > >>> use for fancy processing. > > > >>> > > > >>> Also, sending over TCP allows you to use rsyslog for buffering, and > > if > > > >>> you're using in-memory queues (or disk-assisted, assuming those > > rarely > > > >>> spill out to disk), this means you'll avoid the I/O penalty of > > writing > > > to > > > >>> disks and having Logstash poll from disk periodically. > > > >>> > > > >>> If you need help with any of those, please write here (or on the > > > Logstash > > > >>> ML for the Logstash part, people are really helpful there). > > > >>> > > > >>> Best regards, > > > >>> Radu > > > >>> > > > >>> > > > >>> On Thu, Apr 10, 2014 at 6:13 PM, masoom alam < > [email protected]> > > > >>> wrote: > > > >>> > > > >>> Is it necessary to fill the templates inside rsyslog so that > rsyslog > > > >>>> > > > >>> should > > > >>> > > > >>>> write each log source to a separate file for logtash - will be > easy > > > for > > > >>>> > > > >>> it > > > >>> > > > >>>> for parsing? - also due to the reason logrtash has to catch > > rsyslog? . > > > >>>> > > > >>> What > > > >>> > > > >>>> is the alternative if we are doing extensive parsing in logtash? - > > > >>>> simply > > > >>>> directing log on to a port and ask logtash to pick it up - match > it > > > >>>> > > > >>> against > > > >>> > > > >>>> 200 plugins? > > > >>>> > > > >>>> from phone thus brief. > > > >>>> On Apr 10, 2014 5:06 PM, "Radu Gheorghe" < > > [email protected]> > > > >>>> wrote: > > > >>>> > > > >>>> Here's an article that explains how to configure squeeze > > performance > > > >>>>> > > > >>>> from a > > > >>>> > > > >>>>> rsyslog>ES>Kibana setup, and the numbers I got (20-30K EPS on my > > > >>>>> > > > >>>> good-old > > > >>> > > > >>>> laptop): http://www.rsyslog.com/performance-tuning-elasticsearch/ > > > >>>>> > > > >>>>> You also have links there about other articles in this are (that > > also > > > >>>>> > > > >>>> have > > > >>>> > > > >>>>> config snippets and explanations). > > > >>>>> > > > >>>>> On Tue, Apr 8, 2014 at 11:34 PM, Josh Bitto < > > [email protected]> > > > >>>>> wrote: > > > >>>>> > > > >>>>> If I'm reading this right your saying that you did > > > >>>>>> Rsyslog->Elasticsearch->gui? > > > >>>>>> > > > >>>>>> I've tried installing the rpm on centos and it installs but > > > >>>>>> > > > >>>>> apparently > > > >>> > > > >>>> it > > > >>>> > > > >>>>> doesn't come with a config file and so the daemon starts it > errors > > > >>>>>> > > > >>>>> out > > > >>> > > > >>>> in > > > >>>> > > > >>>>> the logs and just shuts down after that. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> -----Original Message----- > > > >>>>>> From: [email protected] [mailto: > > > >>>>>> [email protected]] On Behalf Of Rick Brown > > > >>>>>> Sent: Tuesday, April 08, 2014 11:31 AM > > > >>>>>> To: rsyslog-users > > > >>>>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > > >>>>>> > > > >>>>> server > > > >>> > > > >>>> > > > >>>>>> Today I've setup my central rsyslog server to replay the logs > via > > > >>>>>> omudpspoof to a logstash server -> ES. It's already indexing > > about > > > >>>>>> > > > >>>>> twice > > > >>>> > > > >>>>> as much as just rsyslog -> ES was using the recipe in the first > > link > > > >>>>>> > > > >>>>> below, > > > >>>>> > > > >>>>>> and I haven't even begun to dig into the scads of plugins > > available > > > >>>>>> > > > >>>>> for > > > >>> > > > >>>> logstash. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>> http://blog.sematext.com/2013/07/01/recipe-rsyslog- > > > >>> elasticsearch-kibana/isagoodplace to start, although you can > replace > > > >>> the omelasticsearch OM > > > >>> > > > >>>> with omudpspoof if you want to do logstash. > > > >>>>>> > > > >>>>>> http://cookbook.logstash.net/recipes/rsyslog-agent/ is a good > > place > > > >>>>>> > > > >>>>> to > > > >>> > > > >>>> start with rsyslog -> logstash, although I did UDP instead of TCP, > > > >>>>>> > > > >>>>> and > > > >>> > > > >>>> used > > > >>>>> > > > >>>>>> the elasticsearch output module instead of stdout, which is > > > >>>>>> > > > >>>>> documented > > > >>> > > > >>>> here: http://cookbook.logstash.net/recipes/central-syslog/ > > > >>>>>> > > > >>>>>> Good luck to you! Those three links is basically all I needed, > > and > > > >>>>>> > > > >>>>> should > > > >>>>> > > > >>>>>> set you down the right path, regardless of how your path differs > > > from > > > >>>>>> > > > >>>>> mine > > > >>>>> > > > >>>>>> ;) > > > >>>>>> > > > >>>>>> ----- Original Message ----- > > > >>>>>> > > > >>>>>>> From: "Orangepeel Beef" <[email protected]> > > > >>>>>>> To: "rsyslog-users" <[email protected]> > > > >>>>>>> Sent: Tuesday, April 8, 2014 2:17:42 PM > > > >>>>>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > > >>>>>>> > > > >>>>>> server > > > >>>> > > > >>>>> > > > >>>>>>> it works, but I find it overly complex for my environment. > read: > > > >>>>>>> > > > >>>>>> I > > > >>> > > > >>>> don't need it ;) On Apr 8, 2014 11:13 AM, "Josh Bitto" > > > >>>>>>> <[email protected]> wrote: > > > >>>>>>> > > > >>>>>>> I have read about Redis as being the "broker" thoughts? > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> -----Original Message----- > > > >>>>>>>> From: [email protected] [mailto: > > > >>>>>>>> [email protected]] On Behalf Of Orangepeel > Beef > > > >>>>>>>> Sent: Tuesday, April 08, 2014 11:11 AM > > > >>>>>>>> To: rsyslog-users > > > >>>>>>>> Subject: Re: [rsyslog] Rsyslog w/ > logstash-elasticsearch-kibana > > > >>>>>>>> server > > > >>>>>>>> > > > >>>>>>>> I use rsyslog to pipe into sec, and then use logstash file > input > > > >>>>>>>> > > > >>>>>>> to > > > >>> > > > >>>> index. > > > >>>>>>>> could be done without SEC as well. I don't like delivering > > > >>>>>>>> > > > >>>>>>> syslog > > > >>> > > > >>>> right into logstash. > > > >>>>>>>> On Apr 8, 2014 11:09 AM, "Sphonic" <[email protected]> > > > >>>>>>>> > > > >>>>>>> wrote: > > > >>> > > > >>>> > > > >>>>>>>> I use rsyslog to send all items to logstash which has a > syslog > > > >>>>>>>>> listener enabled. > > > >>>>>>>>> > > > >>>>>>>>> Sent from my iPhone > > > >>>>>>>>> > > > >>>>>>>>> On 8 Apr 2014, at 18:05, Josh Bitto <[email protected] > > > > > >>>>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>> Hello Everyone, > > > >>>>>>>>>> > > > >>>>>>>>>> I'm wanting to setup a syslog server that combines the three > > > >>>>>>>>>> programs > > > >>>>>>>>>> > > > >>>>>>>>> listed above with rsyslog. Has anyone had any success using > > > >>>>>>>>> > > > >>>>>>>> this? > > > >>> > > > >>>> I'm > > > >>>>>>>>> running on a CentOS 6.5 and finding adequate instructions on > > > >>>>>>>>> > > > >>>>>>>> how > > > >>> > > > >>>> to not only setup all three PLUS rsyslog has been somewhat of a > > > >>>>>>>>> challenge. > > > >>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> This issue that I run into is on how to get > > > >>>>>>>>>> logstash/elasticsearch and > > > >>>>>>>>>> > > > >>>>>>>>> kibana to talk with rsyslog. Halp meh! Please! > > > >>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> rsyslog mailing list > > > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>>>> http://www.rsyslog.com/professional-services/ > > > >>>>>>> What's up with rsyslog? Follow > https://twitter.com/rgerhardsNOTE > > > >>>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > > >>>>>>> > > > >>>>>> of > > > >>>> > > > >>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > >>>>>>> DON'T LIKE THAT. > > > >>>>>>> > > > >>>>>>> > > > >>>>>> -- > > > >>>>>> Rick Brown > > > >>>>>> Office of Information Technology > > > >>>>>> Georgia Institute of Technology > > > >>>>>> _______________________________________________ > > > >>>>>> rsyslog mailing list > > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>>> http://www.rsyslog.com/professional-services/ > > > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhardsNOTE > > > >>>>>> > > > >>>>> WELL: > > > >>>> > > > >>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > > >>>>>> > > > >>>>> sites > > > >>> > > > >>>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > > > >>>>>> > > > >>>>> LIKE > > > >>>> > > > >>>>> THAT. > > > >>>>>> _______________________________________________ > > > >>>>>> rsyslog mailing list > > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>>> http://www.rsyslog.com/professional-services/ > > > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > > > >>>>>> > > > >>>>> myriad > > > >>>> > > > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > > > >>>>>> > > > >>>>> you > > > >>> > > > >>>> DON'T LIKE THAT. > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>>> -- > > > >>>>> Performance Monitoring * Log Analytics * Search Analytics > > > >>>>> Solr & Elasticsearch Support * http://sematext.com/ > > > >>>>> _______________________________________________ > > > >>>>> rsyslog mailing list > > > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>> http://www.rsyslog.com/professional-services/ > > > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > > >>>>> > > > >>>> myriad > > > >>> > > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > you > > > >>>>> DON'T LIKE THAT. > > > >>>>> > > > >>>>> _______________________________________________ > > > >>>> rsyslog mailing list > > > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>> http://www.rsyslog.com/professional-services/ > > > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > > myriad > > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > you > > > >>>> DON'T LIKE THAT. > > > >>>> > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> Performance Monitoring * Log Analytics * Search Analytics > > > >>> Solr & Elasticsearch Support * http://sematext.com/ > > > >>> _______________________________________________ > > > >>> rsyslog mailing list > > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>> http://www.rsyslog.com/professional-services/ > > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > > myriad > > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > you > > > >>> DON'T LIKE THAT. > > > >>> > > > >>> _______________________________________________ > > > >> rsyslog mailing list > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > >> http://www.rsyslog.com/professional-services/ > > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > >> DON'T LIKE THAT. > > > >> > > > >> _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > > DON'T LIKE THAT. > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > > > > > > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
