[part 3 of: "Scalable amavisd-new installation"] There is also a self-written component I'm really proud of: the log aggregation system. I need to somehow offer detailed logs to our support team and to some of our customers. This is impossible to achieve using daily logfiles & grep as of performance reasons and as of privacy prob- lems (simply grepping a customers domain could show him log lines he shouldn't see: BCC etc).
Using the amavis database was also not an option as there are important information missing (front mx is blocking most email traffic not shown here) and performance can be bad on some types of searches, even if assigning large amounts of memory to MySQL and trying to somehow opti- mize table structure. The only way of solving this seemed to be to create a central, "real- time" log aggregation system - and that's what we have done. Syslog of all servers is sent to a central syslog server, and there a) written to daily rotated and compressed plain logfiles b) piped to our aggregation software Writing this software was not an easy task - and I've had to struggle a lot with thing's I would have never thought about without seeing them happening in real time. Log lines happening later very often happen to appear earlier (multiple cores, network latencies...), SMTP sessions are stopped / killed in various strange ways / states, the same mail can pass amavis multiple times etc. Right now it's running fine, even if there are some little issues I'm aware of, that still need to be fixed. We are slowly leaking memory, so we are monitoring the daemon's memory usage and restarting it from time to time. The reason is known, but as it is running fine I never forced myself to sit down and fix it - as we have a lot of other pro- jects with higher priority :-( I would really like to share this piece of software (as my very own contribution to the community ;-). I still have to get my employers permission for doing so - but I'm pretty confident that doing so would be fine. Some details about the software: - daemon is written in PHP5 (DON'T laugh, it's well done and behaves fine ;-) - even if written in PHP (CLI) it is handling POSIX signals, writing to syslog etc - my first attempt was focussed on raw speed - later I realized that no one would ever be able to debug this code, so I've rewritten it from scratch with an object-oriented approach - each mail is going to build a new object, each line add's itself to this object - and creates additional sub-objects (recipients etc) - "completed" mails are stored to a flat "manually partitioned" database, object is destroyed - database contains redundant information and many indexes - main goal is speed - and I can proudly report that it is VERY fast - no foreign keys, just stupid plain MyISAM - unhandled mails / log lines are discarded, information is written to syslog - disconnects / lost connections are handled, unfinished mails written to db and cleaned up - right now there are parser components for postfix and amavis - there is also a perdition parser, doing a similar job - and also allo- wing us to still provide SMTP-after-POP to customers using this service for historical reasons For sure I've forgotten some important details, but you should have some idea about the project right now. So please let me know: - is it worth the effort starting a small OS project for this piece of software? - are there similar projects I should look at before doing so? - are you (all of you!!) going to clean up and perfectionate my code once opensourced - so that I can lay back and let you do my job? That's all, folks ;o) Cheers, Thomas Gelf ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ AMaViS-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
