Rob, There is another issue with using multiple Amavisd child processes. From your description of how long it is taking to clear your backlogs, it could be affecting you.
One of my mail servers was running on a machine with similar horsepower to yours, and it was configured for 10+ Amavisd child processes and a matching number of MTA senders. However, the machine would not process more than 10 to 12 messages per minute through Amavisd. This regularly led to extreme backlogs of thousands of messages lasting several hours. After a lot of use of amavisd-agent, amavisd-nanny, and strace, it turned out the reason for the logjam was because SpamAssassin, as called by Amavisd, was experiencing extreme write-lock contention on two Berkeley Sleepycat databases. One of these was for Bayes checks and the other was for AutoWhitelist checks. This write-lock contention added 30 to 60 seconds of idle wait time per message. I found nothing in Amavisd that could fix this. Reconfiguring SpamAssassin was necessary to address this problem. In /etc/mail/spamassassin/local.cf I tried changing the locking method from NFS-compatible to Flock, which helped a small amount (12% drop in per message processing time). Then I tried turning on write-journaling for the Bayes db, which helped a small amount (25% drop in per message procesing time). However neither of these changes really made the problem go away. This is because the necessary inter-process write locking that needed to be done on Berkeley Sleepycat database files was still being done. What really fixed this problem and allowed me to scale-up the number of Amavisd child processes was to convert the underlying SpamAssassin to use a proper SQL database for Bayes and AutoWhitelist storage. This conversion all but eliminated the write lock contention between Amavisd child processes, and lead to an order of magnitude drop in message processing time. Since then I haven't experienced a single backlog. FYI, in SpamAssassin 3.1.x and 3.2.x there is a documented process using sa-learn for importing Bayes db data into one of the supported SQL databases. However, there is not currently a similar official documented process for converting the AutoWhitelist data from a long-running db. There was an alpha-release Perl script provided in the tools folder of the SpamAssassin 3.1.x tarball. This Perl script is missing and apparently deprecated in SpamAssassin 3.2.x, but it worked just fine for importing v3.2.x AutoWhitelist data. So in conclusion, what the others here have been saying about increasing max_servers in Amavisd (and a matching number of SMTP senders in your MTA) is correct, if you are using SQL (MySQL or PostgreSQL) for your underlying SpamAssassin processing. If you're still using the default Berkeley Sleepycat databases for Bayes and/or AutoWhitelist processing in SpamAssassin, I don't recommend increasing your Amavisd max_servers above 2 to 5 children. At 12:18 PM 2/6/2008, Rob Morin wrote: >Thanks for the info, you guys are great! > >Have a good one! > > >Rob Morin >Dido Internet Inc. >Montreal,Canada >http://www.dido.ca >514-990-4444 > > > >Gary V wrote: >> On 2/6/08, Rob Morin <[EMAIL PROTECTED]> wrote: >> >>> I do not see a ... >>> >>> $max_servers = >>> >>> setting in my conf files, i use Debian so i have a conf.d with a bunch of >>> different files but a grep for max_servers brings back nothing.... so what >>> is the default 1, i guess? >>> >>> I would be nice to up it to 4 or so.... >>> >>> should i added the option in my 50-user ?? >>> >>> >> >> Yes, if placed there, it will override a previous entry. 50-user is >> the last conf file read. >> >> >>> Rob Morin >>> >> >> > >------------------------------------------------------------------------- >This SF.net email is sponsored by: Microsoft >Defy all challenges. Microsoft(R) Visual Studio 2008. >http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ >_______________________________________________ >AMaViS-user mailing list >[email protected] >https://lists.sourceforge.net/lists/listinfo/amavis-user >AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 >AMaViS-HowTos:http://www.amavis.org/howto/ -- Douglas C. Stephens | Network/DNS/Unix/Windows Admin System Support Specialist | Email Postmaster Information Systems | Phone: (515) 294-6102 Ames Laboratory, US DOE | Email: [EMAIL PROTECTED] ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ AMaViS-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
