Thanks for the nice review and instructions :) I have begun to work heavily on a message modfication module for rsyslog which will support liblognorm-style normalization inside rsyslog. In git there already is a branch "lognorm", which I will hopefully complete and merge into master soon. It provides some *very* interesting shortcuts of pulling specific information out of syslog messages. I'll probably promote it some more when it is available. IMHO it's the coolest and potentially most valuable feature I have added in the past three years. Once I have enabled tags in liblognorm/libee, you can even very easily classify log messages based on their content.
More inline below... > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Champ Clark III [Softwink] > Sent: Wednesday, December 01, 2010 9:39 PM > To: [email protected] > Subject: [rsyslog] Fun with liblognorm. > > > I had a chance to build and play a bit with liblognorm. Rainer > has done some good work on it! It was fun to play with and here are > some of my initial thoughts. > > First off, I was using a snapshots from the git repository for > libestr, libee and liblognorm. Compiling/building was pretty straight > forward. Untar the archives and run "autoreconf -vfi", and then run > "./configure && make && make install". When running 'autoreconf' with > liblognorm, it complains about the m4 directory not existing. Not a > big deal, simply create it and re-run 'autoreconf'. I did a couple of bug fixes yesterday. Frequently pull from git ;) > > Once everything was built, I started playing with Rainer's > 'normalizer' example that ships with liblognorm. That was sort of > nice > to have to 'tie everything' together. It basically takes the log input > via stdin and you point it to a log normalizer repository with the -r > flag.. This program (well, wrapper) is in the liblognorm/src > directory. You'd run it like this, assuming you're in the > liblognorm/src directory... > > cat log-messages | ./normalizer -r ../sample-db/messages.sampdb > > In the 'samble-db' are sort of the 'rules' for parsing. I wrote > a couple of test 'rules' just to play. Here's a quick little NTP rule > I wrote: > > :%date:date-rfc3164% %host:word% %tag:char-to:\x3a%: synchronized to > %ip:ipv4% > > My syslog "input" is.. > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10 > > Here's the normlizer output: > > To normalize: 'Dec 1 14:10:11 testbox ntpd[3821]: synchronized to > 192.168.0.1' > normalized: '[...@... ip="192.168.0.1" tag="ntpd[3821\]" host="testbox" > date="Dec 1 14:10:11"]' > > So, that work nicely. Nifty. I made a few more 'complex' > rules, and those worked fine as well. I added a capability to generate graphs of the actual call tree. I think this is *very* useful. An article on how to do that will be posted soon to the web site (will make sure a notification goes to the list). > However, if the rule is off a > bit, then you've got issue. Here's what I mean.. Back on my example > above.. If this: > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10 > > changes to this: > > Dec 1 14:10:11 testbox ntpd[3821]: synchronized to 192.168.0.10, > stratium 1 Well, that's by intention. The normalizer must know exactly which message it is dealing with. This is even more important when we use it for classification. So these two messages are definitely different, and I would consider it very dangerous to automatically merge them into a single one (which would not be a problem from a purely implementation PoV). The more fuzzy the recognition is, the higher is the chance of false recognition, something that would be really, really bad in the context of normalization. > The 'normalizer' will call the ",stratium 1" part of the message as > "unclassified". However, it doesn't appear that it'll grab the IP > address, tag, etc. > > Also, I thing the "real work" is going to be writing rules. That's > going to take some effort, in which I hope to assist with. Yes, that's definitely a lot of work -- and more than what a single person can do. In order to make the normalizer really useful, we need a community effort. If everyone contributes sample databases for their devices, we could gain good results fast. But the key is getting enough momentum, so please help spread the word! Rainer > Anyways, just wanted to share my brief bit of playing with liblognorm. > It looks very promising! I'm hoping to start playing with some test > Sagan + liblognorm code.. just for fun. :) > > > > -- > Champ Clark III | Softwink, Inc | 800-538-9357 x 101 > http://www.softwink.com > > GPG Key ID: 58A2A58F > Key fingerprint = 7734 2A1C 007D 581E BDF7 6AD5 0F1F 655F 58A2 A58F > If it wasn't for C, we'd be using BASI, PASAL and OBOL. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
