On Thu, 23 Jun 2016, Champ Clark III wrote:
I assist with a project that pretty heavily depends on liblognorm called
"Sagan" (http://sagan.io).
While we have other "normalization" methods, we prefer liblognorm. Our
community rulebase file is at:
https://github.com/beave/sagan-rules/blob/master/normalization.rulebase
I agree with David, we don't want 10 different ways to normalize a Cisco log.
At the same time, Cisco logs sometimes differ just enough that you _might_
need multiple ways to normalize them.
as an example of what I'm talking about.
take the log example %ASA-6-302014 (end of TCP session)
a few variations of which are:
%ASA-6-302014:Teardown TCP connection 42095195 for outside:2.2.9.2/5721 to
inside:192.168.1.1/54151 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 43363071 for
outside:192.168.2.5\/58949(LOCAL\\D.A) to
outside:192.168.2.3\/3283(LOCAL\\CP-9999G-SEP999999999999) duration 0:00:00
bytes 0 TCP Reset-O (D.A)
%ASA-6-302014: Teardown TCP connection 51708532 for
outside:10.1.5.5/54853 to backup:192.168.2.1/4784(LOCALCP-9999G-SEPC99999999999) duration 0:00:00
bytes 0 <snp_drop_none>
some people will parse it so that they have the variables sourceif, sourceip,
sourceport, destif, destip, destport etc
I do source:{interface,ip,port} dest:{interface,ip,port}
this is making use of the v2 ciscointerface type
prefix=%timestamp:date-rfc3164% %hostname:word%
rule=cisco,disconnect: \x25ASA-6-302014\x3a Teardown %proto:word% connection
%connection-id:number% for %source:cisco-interface-spec% to
%dest:cisco-interface-spec% duration %duration:char-to: % bytes %bytes:number%
%reason:rest%
So we will need to agree of if we are going to use nesting or not (I think we
should), and if we do it with Cisco, we need to do it across the board
by the way, this also brings up the issue of tags for the message
We have talked about "market place" for rule normalization for years now. It
was always my impression that this would be part of the rsyslog team efforts.
It sounds like you have enough on your plate, keeping track for rulebase isn't
high on priority. I understand this. With Sagan, we are doing this
"anyways". That is, we are creating rulebases for different types of logs
either way. We commit them to the Sagan repo right now.
I'd like to suggest the following for response:
1. Split off the "normalization.rules" base from Sagan and great a new,
separate github repo for it.
2. If someone would like to add some rulebase "rules", they can do a "pull"
request.
3. All rulebase "rules" need to have an example, anonymized log sample. Used
for testing.
4. If the rules look good, then they can be merged.
besides the pull request mechansim, I think we also need a way for people who
have rulesets to send them out for others to convert to pull requests. I think
that there is going to be a lot of tweaking/corrections to the proposed rules,
and a pull request isn't neccessarily the best way to handle that.
I think it would be a great thing to merge the existing scattered efforts.
Possibly under the liblognorm banner instead of rsyslog (not a big diffence)
I'm certainly not trying to step on Brian's or anyone elses toe's. IMHO,
Sagan will benefit from a project like this. Obviously, rsyslog will as well.
This would likely bring other people outside rsyslog to the project as well).
And best of all, I think it will give far more people the comfort to start using
the parsing.
Given that liblognorm is pretty insensitive to the number of rules, it may be
that we can craft a combined rulebase that could ship by default with liblognorm
as a starter for people.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
