On Fri, Sep 02, 2005 at 12:34:36PM +0200, Gianluca Marcari wrote:
> We deployed a 14-host parallel postfix+amavis installation and are
> having performance definitely not in line with what we were expecting
> (and actually getting from a previos lower-scale installation).
>
> Scenario:
>
> Load balancer publishes a virtual IP which spreads the load on the 14
> content filtering servers (plus the 5 non-filtering servers which were
> previously handling the load, we had to bring them back into the group
> because the 14 servers can't cope with the traffic)
>
> The boxes have a postfix instance which performs some
> RBL+header+black/whitelist checks and then forwards to the local
> amavisd-new instance listening on the usual port 10024
>
> Amavisd does an LDAP query against our centralized ldap, then performs
> SpamAssassin + ClamAV checks
>
> Viruses get destroyed
> Spam gets destroyed unless the recipient is an AmavisVirusLover
> (400.000 users out of 6 millions in the LDAP)
> There is no quarantine at all, for anything (virus/spam/badheader/etc)
>
> Everything not destroyed (ham, spam to AmavisVirusLovers) is sent to
> the original 5 SMTP servers which handle it with no apparent proble
> (If we shunt all the traffic to the 5 original SMTP servers,
> everything flows smoothly).
>
> Boxes are 2x2.66Ghz HT Xeons, 1GB Ram, 2 Ultra-320 HDDs in hardware
> RAID1, Linux 2.6.12.3, amavis has its $TEMPBASE on a ramdrive
> (/dev/shm).
>
> The systems are processing between 7 and 14 emails per second, while
> receiving up to 40, thus falling behind constantly.
>
> Other relevant data:
> Perl v5.8.5
> Logging via syslog
> Filesystems are reiserfs v3
> SpamAssassin version 3.0.4
>
> Relevant data in /etc/amavisd:conf:
> $max_servers = 10;
> $max_requests = 10;
> $enable_ldap = 1;
My calculations based on the above are that on a comparable system,
excluding LDAP, you should be handling about 4.1 msgs/sec steady load
through each of the scanners, hence 57 msgs/sec throughput for the
cluster of 14. (This is based on my numbers for dual 3GHz Xeons on
FreeBSD.)
> Even though there is only 1 (mirrored) disk, we are seeing almost no
> iowait (2% max, often 0%). System load is split between 74%usr, 22%
> sys and 3% idle (fairly constant on most boxes)
Under load you should see virtually 0 idle time. If all free CPU is
not being soaked up by SA PCRE matching, something is suboptimal.
Are you running a local DNS cache on each machine? I'd install the
DJB tinydns dnscache process on every one (configured as a forwarder to
your main DNS server.) You might also raise the number of amavisd
processes slightly (and boost max_requests a lot, e.g. to 50 or 100, so
it's not constantly killing/respawning children under heavy load.)
> >From our logs, many mails get processed and handed over to the backend
> in 300-500 milliseconds, but some are in the 5000-150000 milliseconds
> range:
The former is right, the latter indicates something screwy is going
on.
...
> Mail which have long processing time exhibit an odd breakdown,
> spending a huge chunk of that time in fwd-rcpt-to and sometimes
> best_try_originator
>
>
> Sep 1 08:37:16 asav1 amavis[3682]: (03682-02-4) TIMING [total 5382
> ms] - lookup_ldap: 16 (0%)0, SMTP pre-DATA-flush: 1 (0%)0, SMTP DATA:
> 66 (1%)2, body_digest: 1 (0%)2, gen_mail_id: 0 (0%)2, mime_decode: 33
> (1%)2, get-file-type4: 16 (0%)2, decompose_part: 1 (0%)3,
> decompose_part: 3 (0%)3, get-file-type1: 14 (0%)3, decompose_part: 57
> (1%)4, parts_decode: 0 (0%)4, AV-scan-1: 10 (0%)4,
> read_snmp_variables: 1 (0%)4, best_try_originator: 7 (0%)4,
> update_cache: 1 (0%)4, fwd-connect: 24 (0%)5, fwd-mail-from: 2 (0%)5,
> fwd-rcpt-to: 5094 (95%)99, fwd-rundown: 3 (0%)99, deal_with_mail_size:
> 1 (0%)99, main_log_entry: 27 (1%)100, update_snmp: 2 (0%)100,
> unlink-4-files: 1 (0%)100, rundown: 0 (0%)100
There are a lot of possibilities here, but it's all happening in the
handoff back to Postfix. I think this could be indicating LDAP lookup
delays *or* inappropriate DNS check delays on the receiving Postfix
process. Look at load, concurrent requests, etc. on your LDAP server.
Look at the Postfix logs for reinjection processes, see if you can get
timing information on that side for the slow transactions and if it
tallies with what you see in the amavisd timing breakdown.
...
> Sorry for the big mail, I tried to strike a balance between providing
> enough useful information to troubleshoot the issue and flooding the
> list.
No problem, this was perfect. Thanks for providing so much relevant
information, and I hope somebody can point you to where the issue is!
-- Clifton
--
Clifton Royston -- [EMAIL PROTECTED]
Tiki Technologies Lead Programmer/Software Architect
"My own personal theory is that this is the very dawn of the world.
We're hardly more than an eyeblink away from the fall of Troy, and
scarcely an interglaciation removed from the Altamira cave painters. We
live in extremely interesting ancient times.
I like this idea. It encourages us to be earnest and ingenious and
brave, as befits ancestral peoples; but keeps us from deciding that
because we don't know all the answers, they must be unknowable and thus
unprofitable to pursue." -- Teresa Nielsen Hayden, 1995
-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
AMaViS-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
