-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I have been a sysadmin for quite a while now and used to do a lot of perl. I haven't done much programming at all in the last couple of years but have been meaning to dump perl for python anyhow (for the usual reasons) and have finally gotten around to doing it.
The first itch that I wanted to scratch was something to help me find anomalies in our system logs. We have a lot of servers and need some automation for watching over them including logfiles. I have used logwatch and logcheck but they are both too complicated for what I really need not to mention practically unmaintained these days. So I decided to write my own. Features I wanted were: 1. A basic grep -v sort of functionality on the logfile where I have a file full of regexes/lines to be ignored that are applied to the logfiles to filter everything uninteresting out and whatever is left gets emailed to the admin. 2. A sort of macro system where I can substitute nasty long commonly used regexes in my filter/ignore file with something more easily read and typed. 3. Simplicity. Logcheck had several different levels of logfile events which it could report back on and if you wanted something ignored you had to put it in the right config file for whatever level it was popping up in. This drove me nuts because I often got the wrong one. Logwatch tries to analyze your logs and provide various summaries and statistics and other things I don't care about. I just want to see the interesting lines from the logfile. My code seems to have all of the above and a silly name as well: http://ultraviolet.org/Members/treed/lollerskates It has a list of logfiles to look at, a list of regexes to compare to each line to know what is uninteresting and to filter out/ignore, and emails anything left to the admin, and a simple macro facility. I would appreciate it if anyone interested would download and critique my code. Am I doing anything terribly un-pythonic? Questions/problems/TODO's: This is a fairly simple structured programming implementation. No OO. Should I be using some classes somewhere? The config file is just a module which I import which causes all of my configs to become globals. Globals are bad. Is there a better way or would just about anything else be overkill? A singleton config class or something? Overkill? I have several loops in this code for processing the logfiles. I once tried to convert these for loops to list comprehensions and totally confused myself and backed out the change (yeay svn!). Is there any benefit to list comprehensions in this case? I would kinda like to play with unit tests. Not sure how I would construct unit tests for this. And again, perhaps overkill. But some people tell me you should write the unit tests before you even begin coding and code until the tests are satisfied. So even on a small project you would have tests. I run into a couple nasty bugs I created which caused the script to never return anything from the logfile (so you don't immediately realize something is broken) where I thought "It sure would be nice to have a test to catch that if it ever happens again." Stuff tends to build up in the ignore file. If a line is typo'd and never gets used/matched you will never know. If a service is no longer running that we had lines in the ignore file for they will be there forever unused. I don't like this sort of clutter. I am really tempted to somehow make it keep track of when a rule is matched and if it doesn't match in a whole month email the admin because it doesn't need to be there. I am wondering if this is needless complexity or something useful. I think it could be useful. If so, how to save the info on when each line was last matched? I am thinking maybe of just having a dictionary keyed on the compiled regex object and the value being a tuple of the uncompiled regex string from the ignore file and a date string containing when it was last matched and pickling this data structure to disk to be read in each start of the program. Comments? I am currently pushing this code to around 20 systems with cfengine using just one ignore file for them all. If I use the scheme mentioned in the previous paragraph to alert me of unused rules in the ignore file I will get tons of such messages from machines not running a service that other machines of mine are. I guess that means perhaps I should split my ignore file out into ignore.hostname.conf or something like that so there is a separate one for each host. Anyhow, I appreciate any feedback on this potentially very handy sysadmin tool. Thanks in advance! - -- Tracy R Reed http://ultraviolet.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFFObPK9PIYKZYVAq0RAiflAJ9jprJgGnNRXkB+nKsljFUsAUGnFwCgiJg/ M3U24dNDtH+hAgSh7kZ40hQ= =nxQS -----END PGP SIGNATURE----- _______________________________________________ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
