Thanks all for the great discussion and effort going forward! I am in preparation for a trip next week and so unfortunately had limited time to contribute (and will be unable next week), but I am more than interested in helping to move this forward.
Note that we currently have some rulebases inside liblognorm's git: https://github.com/rsyslog/liblognorm/tree/master/rulebases This might be the place where we can begin to actually gather a full set ... or we could create a new git repo. The latter might be a better idea, as the folks who primarily maintain it are probably quite different. Again, I am excited to see all this new activity. Also keep in mind that with v2 (finally to be released next month), we can have custom data types just like in grok, so building rules is also much easier. IMHO it would make sense to first build a set of custom data types (like we did in lognorm with the cisco address representation), and then base rules on those extended set of base types. This is a sample from the testbench of how custom types are defined: https://github.com/rsyslog/liblognorm/blob/master/tests/usrdef_twotypes.sh Also, the doc has good information on that topic: https://github.com/rsyslog/liblognorm/blob/master/doc/configuration.rst As I said, I will unfortunately be mostly silent up unitl begin of june - please don't treat this as sign of desinterest! Again, I think this is an extremely valuable approach. Rainer 2016-06-23 19:25 GMT+02:00 David Lang <[email protected]>: > On Thu, 23 Jun 2016, Champ Clark III wrote: > >> I assist with a project that pretty heavily depends on liblognorm called >> "Sagan" (http://sagan.io). >> >> While we have other "normalization" methods, we prefer liblognorm. Our >> community rulebase file is at: >> >> https://github.com/beave/sagan-rules/blob/master/normalization.rulebase >> >> I agree with David, we don't want 10 different ways to normalize a Cisco >> log. At the same time, Cisco logs sometimes differ just enough that you >> _might_ need multiple ways to normalize them. > > > as an example of what I'm talking about. > > take the log example %ASA-6-302014 (end of TCP session) > > a few variations of which are: > > %ASA-6-302014:Teardown TCP connection 42095195 for outside:2.2.9.2/5721 to > inside:192.168.1.1/54151 duration 0:00:30 bytes 0 SYN Timeout > > %ASA-6-302014: Teardown TCP connection 43363071 for > outside:192.168.2.5\/58949(LOCAL\\D.A) to > outside:192.168.2.3\/3283(LOCAL\\CP-9999G-SEP999999999999) duration 0:00:00 > bytes 0 TCP Reset-O (D.A) > %ASA-6-302014: Teardown TCP connection 51708532 for outside:10.1.5.5/54853 > to backup:192.168.2.1/4784(LOCALCP-9999G-SEPC99999999999) duration 0:00:00 > bytes 0 <snp_drop_none> > > some people will parse it so that they have the variables sourceif, > sourceip, sourceport, destif, destip, destport etc > > I do source:{interface,ip,port} dest:{interface,ip,port} > > this is making use of the v2 ciscointerface type > > prefix=%timestamp:date-rfc3164% %hostname:word% > > rule=cisco,disconnect: \x25ASA-6-302014\x3a Teardown %proto:word% connection > %connection-id:number% for %source:cisco-interface-spec% to > %dest:cisco-interface-spec% duration %duration:char-to: % bytes > %bytes:number% %reason:rest% > > So we will need to agree of if we are going to use nesting or not (I think > we should), and if we do it with Cisco, we need to do it across the board > > by the way, this also brings up the issue of tags for the message > >> We have talked about "market place" for rule normalization for years now. >> It was always my impression that this would be part of the rsyslog team >> efforts. It sounds like you have enough on your plate, keeping track for >> rulebase isn't high on priority. I understand this. With Sagan, we are >> doing this "anyways". That is, we are creating rulebases for different >> types of logs either way. We commit them to the Sagan repo right now. >> >> I'd like to suggest the following for response: >> >> 1. Split off the "normalization.rules" base from Sagan and great a new, >> separate github repo for it. >> 2. If someone would like to add some rulebase "rules", they can do a >> "pull" request. >> 3. All rulebase "rules" need to have an example, anonymized log sample. >> Used for testing. >> 4. If the rules look good, then they can be merged. > > > besides the pull request mechansim, I think we also need a way for people > who have rulesets to send them out for others to convert to pull requests. I > think that there is going to be a lot of tweaking/corrections to the > proposed rules, and a pull request isn't neccessarily the best way to handle > that. > > I think it would be a great thing to merge the existing scattered efforts. > Possibly under the liblognorm banner instead of rsyslog (not a big diffence) > >> I'm certainly not trying to step on Brian's or anyone elses toe's. IMHO, >> Sagan will benefit from a project like this. Obviously, rsyslog will as >> well. This would likely bring other people outside rsyslog to the project as >> well). > > > And best of all, I think it will give far more people the comfort to start > using the parsing. > > Given that liblognorm is pretty insensitive to the number of rules, it may > be that we can craft a combined rulebase that could ship by default with > liblognorm as a starter for people. > > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
