Hi Champ! My toes are fine and the more the merrier. Well - the more collaborators, not toes.
I've used Sagan's rulebase as a reference before - great stuff! It comes to mind that a coworker and I are currently working on a dockerized rsyslog + elasticsearch environment for doing rsyslog integration testing, that we were planning on tossing up github. Via docker compose it starts up a very small ES cluster ( 2 indexers in containers + 1 master, client, and kibana ) along with an rsyslog instance configured to receive over TCP and UDP and forward to ES. I'm currently imagining a system where people could check in mmnormalize rules + add to a sample rulebase and log(s), and travis CI could fire off, run the sample log for the rule through the rulebase, then verify the results. Such an environment could also be spun up locally for testing while developing new rules. If such a project is something that others would find useful, I could definitely get my employer to sponsor my time on it. I was thinking we could use a dev process we use in the ZeroMQ community that is designed for low friction and high amounts of collaboration without a lot of up front coordination ( http://rfc.zeromq.org/spec:42/C4/ ) so that people with good ideas who want to contribute can just jump in. People who contribute under this process are promoted to maintainers without any fuss, so no one has to worry about central ownership. Let me poke a couple of people - I can probably get what we have as far as the test environment up on github by the end of the week - the more the merrier. We were going to release at least the test environment regardless. Cheers, Brian On Thu, Jun 23, 2016 at 9:25 AM Champ Clark III <[email protected]> wrote: > I assist with a project that pretty heavily depends on liblognorm called > "Sagan" (http://sagan.io). > > While we have other "normalization" methods, we prefer liblognorm. Our > community rulebase file is at: > > https://github.com/beave/sagan-rules/blob/master/normalization.rulebase > > I agree with David, we don't want 10 different ways to normalize a Cisco > log. At the same time, Cisco logs sometimes differ just enough that you > _might_ need multiple ways to normalize them. > > We have talked about "market place" for rule normalization for years now. > It was always my impression that this would be part of the rsyslog team > efforts. It sounds like you have enough on your plate, keeping track for > rulebase isn't high on priority. I understand this. With Sagan, we are > doing this "anyways". That is, we are creating rulebases for different > types of logs either way. We commit them to the Sagan repo right now. > > I'd like to suggest the following for response: > > 1. Split off the "normalization.rules" base from Sagan and great a new, > separate github repo for it. > 2. If someone would like to add some rulebase "rules", they can do a > "pull" request. > 3. All rulebase "rules" need to have an example, anonymized log sample. > Used for testing. > 4. If the rules look good, then they can be merged. > > I'm certainly not trying to step on Brian's or anyone elses toe's. > IMHO, Sagan will benefit from a project like this. Obviously, rsyslog > will as well. This would likely bring other people outside rsyslog to the > project as well). > > Let me know your thoughts and thank you. > > > > ----- Original Message ----- > From: "Ryan Ward" <[email protected]> > To: "rsyslog-users" <[email protected]> > Sent: Thursday, June 23, 2016 8:51:48 AM > Subject: Re: [rsyslog] mmnormalize rule database Re: mmgrok packages > > All as a newbie to rsyslog I think this is a great idea and would find a > marketplace for rulebases and examples very beneficial. > > > > On Thu, Jun 23, 2016 at 7:06 AM, Brian Knox <[email protected]> > wrote: > > > David - I'm sure I could get some time to devote to shepherding this, > and I > > could get some time and resources from our community team to write some > > articles / tutorials about rsyslog + mmnormalize and generate some > > publicity for the project. Additionally I have access to a decently > large > > sampling of logs from a reasonably scaled environment for testing. > > > > If this is something people are interested in and the only blocker is > time > > and resources let me talk to a couple of people today and I'll update the > > list. > > > > Cheers, > > Brian > > > > On Wed, Jun 22, 2016 at 7:24 PM David Lang <[email protected]> wrote: > > > > > On Wed, 22 Jun 2016, Joe Blow wrote: > > > > > > > What about soliciting people to start sharing their mmnormalize > rules? > > > > I've already shared my checkpoint rules, I could see about sharing my > > > Cisco > > > > rules as well. I avoid regex engines like the plague (for obvious > > > > reasons), but would also like to see larger log source parsers > adopted > > > and > > > > open sourced. > > > > > > > > Thoughts? Should we try and start a larger repository for parsing > well > > > > adopted log sources via liblognorm? > > > > > > This thought keeps getting raised. Yes this shoudl be done. The problem > > is > > > that > > > nobody has stepped up to organize this. > > > > > > We don't want to have 50 different ways to handle the same Cisco > message, > > > but > > > how do we pick which of the many different versions we are going to > use? > > > > > > David Lang > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
