Cisco Pix firewall is a great in the sense that it contains a lot of if-then-else and multiple regex. We have done not-complete conversion as logtash plugin. Can we combine our synergies regarding logtash Plugin--> Rsyslog JASON?
Thanks. On Fri, Apr 11, 2014 at 6:34 AM, Brown, Richard A <[email protected] > wrote: > I like where this is going. I'd rather not have to involve logstash as > well.. But their contrib package of plugins/filters is compelling. If we > can build our own similar book of recipes, then let's do it! I'll start on > a few of my common use cases and report back. > > > Sent via mobile > > -------- Original message -------- > From: masoom alam <[email protected]> > Date: 04/10/2014 7:24 PM (GMT-05:00) > To: rsyslog-users <[email protected]> > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server > > Very detailed answer. Thanks!!! > > Since it is related with both Rsyslog and Logtash, thats why I am asking > here. After your kind guidance, Its now clear that we should use JASON > template in Rsyslog, and then use JASON in logtash. I did not find any > significant difference at the logtash end regarding Grok and JASON, except > the word Jason in the filter instead of Grok, am I right? -- I mean as for > as the syntax is concerned. For the execution it will definitely have > performance gains, as you suggested. > > Another thing which I think I did not explain well in my email is that we > are thinking to place some regex at the Rsyslog end too. Suppose we have > >200 filters defined in Logtash, so will happen that when a log entry will > arrive at the Logtash, it will have to match it against all the 200 filters > -- worst case, and/or some thing matches earlier and we compose the > configuration file of Logtash in a way that it escapes. Any ideas how to > optimize the log deep/fancy parsing at this end? > > Once Again thanks Radu. You are very helpful. > > > > > > > On Fri, Apr 11, 2014 at 12:36 AM, Radu Gheorghe > <[email protected]>wrote: > > > I've never actually tried this, but I think the best way for performance > is > > to send over TCP, but make the template a JSON with everything rsyslog > can > > parse (by default, stuff like severity, date, etc). On the Logstash side, > > you'll use the JSON that should parse much faster than grok can parse > > syslog. After that, you'd set the rest of the Logstash filters you want > to > > use for fancy processing. > > > > Also, sending over TCP allows you to use rsyslog for buffering, and if > > you're using in-memory queues (or disk-assisted, assuming those rarely > > spill out to disk), this means you'll avoid the I/O penalty of writing to > > disks and having Logstash poll from disk periodically. > > > > If you need help with any of those, please write here (or on the Logstash > > ML for the Logstash part, people are really helpful there). > > > > Best regards, > > Radu > > > > > > On Thu, Apr 10, 2014 at 6:13 PM, masoom alam <[email protected]> > > wrote: > > > > > Is it necessary to fill the templates inside rsyslog so that rsyslog > > should > > > write each log source to a separate file for logtash - will be easy for > > it > > > for parsing? - also due to the reason logrtash has to catch rsyslog? . > > What > > > is the alternative if we are doing extensive parsing in logtash? - > simply > > > directing log on to a port and ask logtash to pick it up - match it > > against > > > 200 plugins? > > > > > > from phone thus brief. > > > On Apr 10, 2014 5:06 PM, "Radu Gheorghe" <[email protected]> > > > wrote: > > > > > > > Here's an article that explains how to configure squeeze performance > > > from a > > > > rsyslog>ES>Kibana setup, and the numbers I got (20-30K EPS on my > > good-old > > > > laptop): http://www.rsyslog.com/performance-tuning-elasticsearch/ > > > > > > > > You also have links there about other articles in this are (that also > > > have > > > > config snippets and explanations). > > > > > > > > On Tue, Apr 8, 2014 at 11:34 PM, Josh Bitto <[email protected]> > > > > wrote: > > > > > > > > > If I'm reading this right your saying that you did > > > > > Rsyslog->Elasticsearch->gui? > > > > > > > > > > I've tried installing the rpm on centos and it installs but > > apparently > > > it > > > > > doesn't come with a config file and so the daemon starts it errors > > out > > > in > > > > > the logs and just shuts down after that. > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: [email protected] [mailto: > > > > > [email protected]] On Behalf Of Rick Brown > > > > > Sent: Tuesday, April 08, 2014 11:31 AM > > > > > To: rsyslog-users > > > > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > server > > > > > > > > > > Today I've setup my central rsyslog server to replay the logs via > > > > > omudpspoof to a logstash server -> ES. It's already indexing about > > > twice > > > > > as much as just rsyslog -> ES was using the recipe in the first > link > > > > below, > > > > > and I haven't even begun to dig into the scads of plugins available > > for > > > > > logstash. > > > > > > > > > > > > > > > > > > > > http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/isagoodplaceto > start, although you can replace the omelasticsearch OM > > > > > with omudpspoof if you want to do logstash. > > > > > > > > > > http://cookbook.logstash.net/recipes/rsyslog-agent/ is a good > place > > to > > > > > start with rsyslog -> logstash, although I did UDP instead of TCP, > > and > > > > used > > > > > the elasticsearch output module instead of stdout, which is > > documented > > > > > here: http://cookbook.logstash.net/recipes/central-syslog/ > > > > > > > > > > Good luck to you! Those three links is basically all I needed, and > > > > should > > > > > set you down the right path, regardless of how your path differs > from > > > > mine > > > > > ;) > > > > > > > > > > ----- Original Message ----- > > > > > > From: "Orangepeel Beef" <[email protected]> > > > > > > To: "rsyslog-users" <[email protected]> > > > > > > Sent: Tuesday, April 8, 2014 2:17:42 PM > > > > > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > > server > > > > > > > > > > > > it works, but I find it overly complex for my environment. read: > > I > > > > > > don't need it ;) On Apr 8, 2014 11:13 AM, "Josh Bitto" > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > I have read about Redis as being the "broker" thoughts? > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: [email protected] [mailto: > > > > > > > [email protected]] On Behalf Of Orangepeel > Beef > > > > > > > Sent: Tuesday, April 08, 2014 11:11 AM > > > > > > > To: rsyslog-users > > > > > > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > > > > > > server > > > > > > > > > > > > > > I use rsyslog to pipe into sec, and then use logstash file > input > > to > > > > > > > index. > > > > > > > could be done without SEC as well. I don't like delivering > > syslog > > > > > > > right into logstash. > > > > > > > On Apr 8, 2014 11:09 AM, "Sphonic" <[email protected]> > > wrote: > > > > > > > > > > > > > > > I use rsyslog to send all items to logstash which has a > syslog > > > > > > > > listener enabled. > > > > > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > > > > > On 8 Apr 2014, at 18:05, Josh Bitto < > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > > > > > > > I'm wanting to setup a syslog server that combines the > three > > > > > > > > > programs > > > > > > > > listed above with rsyslog. Has anyone had any success using > > this? > > > > > > > > I'm > > > > > > > > running on a CentOS 6.5 and finding adequate instructions on > > how > > > > > > > > to not only setup all three PLUS rsyslog has been somewhat > of a > > > > > > > > challenge. > > > > > > > > > > > > > > > > > > This issue that I run into is on how to get > > > > > > > > > logstash/elasticsearch and > > > > > > > > kibana to talk with rsyslog. Halp meh! Please! > > > > > > > > > > > > > > > > > > > > rsyslog mailing list > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > > http://www.rsyslog.com/professional-services/ > > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhardsNOTE > > > > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of > > > > > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > > > > DON'T LIKE THAT. > > > > > > > > > > > > > > > > -- > > > > > Rick Brown > > > > > Office of Information Technology > > > > > Georgia Institute of Technology > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com/professional-services/ > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > > WELL: > > > > > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > sites > > > > > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > > LIKE > > > > > THAT. > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com/professional-services/ > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > > myriad > > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > you > > > > > DON'T LIKE THAT. > > > > > > > > > > > > > > > > > > > > > -- > > > > Performance Monitoring * Log Analytics * Search Analytics > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > > DON'T LIKE THAT. > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > > > > > > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
