Thanks very much for your explanation.... i am intending to go with SQL as you mentioned in the near future, i now store spams in a database via Mailzu, this works great.... i do not use autowhitelisting... but i do use Bayes... i will take into consideration all you have mentioned with the other sysadmins... I only intend to use 4 child processes for now to see how it works out..
Once again thanks for all the info..... Have a great evening... :) Rob Morin Dido Internet Inc. Montreal,Canada http://www.dido.ca 514-990-4444 Douglas C. Stephens wrote: > Rob, > > There is another issue with using multiple Amavisd child processes. From your > description > of how long it is taking to clear your backlogs, it could be affecting you. > > One of my mail servers was running on a machine with similar horsepower to > yours, and it > was configured for 10+ Amavisd child processes and a matching number of MTA > senders. > However, the machine would not process more than 10 to 12 messages per minute > through > Amavisd. This regularly led to extreme backlogs of thousands of messages > lasting several > hours. > > After a lot of use of amavisd-agent, amavisd-nanny, and strace, it turned out > the reason > for the logjam was because SpamAssassin, as called by Amavisd, was > experiencing extreme > write-lock contention on two Berkeley Sleepycat databases. One of these was > for Bayes > checks and the other was for AutoWhitelist checks. This write-lock > contention added 30 > to 60 seconds of idle wait time per message. > > I found nothing in Amavisd that could fix this. Reconfiguring SpamAssassin > was necessary > to address this problem. In /etc/mail/spamassassin/local.cf I tried changing > the locking > method from NFS-compatible to Flock, which helped a small amount (12% drop in > per message > processing time). Then I tried turning on write-journaling for the Bayes db, > which helped > a small amount (25% drop in per message procesing time). However neither of > these changes > really made the problem go away. This is because the necessary inter-process > write > locking that needed to be done on Berkeley Sleepycat database files was still > being done. > > What really fixed this problem and allowed me to scale-up the number of > Amavisd child > processes was to convert the underlying SpamAssassin to use a proper SQL > database for > Bayes and AutoWhitelist storage. This conversion all but eliminated the > write lock > contention between Amavisd child processes, and lead to an order of magnitude > drop in > message processing time. Since then I haven't experienced a single backlog. > > FYI, in SpamAssassin 3.1.x and 3.2.x there is a documented process using > sa-learn for > importing Bayes db data into one of the supported SQL databases. However, > there is not > currently a similar official documented process for converting the > AutoWhitelist data from > a long-running db. There was an alpha-release Perl script provided in the > tools folder of > the SpamAssassin 3.1.x tarball. This Perl script is missing and apparently > deprecated in > SpamAssassin 3.2.x, but it worked just fine for importing v3.2.x > AutoWhitelist data. > > So in conclusion, what the others here have been saying about increasing > max_servers in > Amavisd (and a matching number of SMTP senders in your MTA) is correct, if > you are using > SQL (MySQL or PostgreSQL) for your underlying SpamAssassin processing. If > you're still > using the default Berkeley Sleepycat databases for Bayes and/or AutoWhitelist > processing > in SpamAssassin, I don't recommend increasing your Amavisd max_servers above > 2 to 5 > children. > > > At 12:18 PM 2/6/2008, Rob Morin wrote: > >> Thanks for the info, you guys are great! >> >> Have a good one! >> >> >> Rob Morin >> Dido Internet Inc. >> Montreal,Canada >> http://www.dido.ca >> 514-990-4444 >> >> >> >> Gary V wrote: >> >>> On 2/6/08, Rob Morin <[EMAIL PROTECTED]> wrote: >>> >>> >>>> I do not see a ... >>>> >>>> $max_servers = >>>> >>>> setting in my conf files, i use Debian so i have a conf.d with a bunch of >>>> different files but a grep for max_servers brings back nothing.... so what >>>> is the default 1, i guess? >>>> >>>> I would be nice to up it to 4 or so.... >>>> >>>> should i added the option in my 50-user ?? >>>> >>>> >>>> >>> Yes, if placed there, it will override a previous entry. 50-user is >>> the last conf file read. >>> >>> >>> >>>> Rob Morin >>>> >>>> >>> >>> >> ------------------------------------------------------------------------- >> This SF.net email is sponsored by: Microsoft >> Defy all challenges. Microsoft(R) Visual Studio 2008. >> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ >> _______________________________________________ >> AMaViS-user mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/amavis-user >> AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 >> AMaViS-HowTos:http://www.amavis.org/howto/ >> > > -- > Douglas C. Stephens | Network/DNS/Unix/Windows Admin > System Support Specialist | Email Postmaster > Information Systems | Phone: (515) 294-6102 > Ames Laboratory, US DOE | Email: [EMAIL PROTECTED] > > ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ AMaViS-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
