Wendigo Thompson put forth on 1/14/2010 6:39 PM: > Okay, I have some updates. Reducing the concurrency down (and > ultimately to 1) increased performance: it seems the delays I was > seeing were related to concurrency inside SQL. However, I am still > seeing half second delays in local delivery while my insertion > application is showing ~.003s processing times. I was able to verify > that it's not my insertion application by making a simple script that > soft-bounces mail (exit code 75) and does nothing else -- there's > still a half a second delay on action. Tracing the individual > processes is proving to be a challenge as this is an Apple system and > dtruss can only be run as root. Any tips?
Not specifically. I was just going over the numbers, and with your previous configuration, before the current tweaking, the db process was consuming each email in about 10-20 seconds, which is extremely long but for large PDFs, Word docs, or binaries. Lets say the average was 10 seconds. If concurrency was 1, that comes out to ~34 days to consume 300,000 emails, if my math is correct, at 86,400 seconds per day. Obviously your concurrency was a little higher than 1. ;) > Jan 14 18:30:58 x postfix/local[27253]: 03B2A13E351: to=<xxx>, > relay=local, delay=40172, delays=37592/2580/0/0.51, dsn=4.3.0, > status=deferred (temporary failure) > Jan 14 18:30:59 x postfix/local[27253]: 03B31132541: to=<xxx>, > relay=local, delay=70394, delays=67813/2581/0/0.53, dsn=4.3.0, > status=deferred (temporary failure) > Jan 14 18:30:59 x postfix/local[27253]: 03B371416A7: to=<xxx>, > relay=local, delay=36726, delays=34144/2581/0/0.53, dsn=4.3.0, > status=deferred (temporary failure) > Jan 14 18:31:00 x postfix/local[27253]: 03B3714401A: to=<xxx>, > relay=local, delay=34562, delays=31980/2582/0/0.54, dsn=4.3.0, > status=deferred (temporary failure) It appears you've now dropped the db process consumption time down from ~10 seconds per email to about .5 seconds per email, a speedup of ~20:1. All things being equal, we could assume you'll start seeing messages sitting in the queue for a little over an hour now instead of a day, once things settle down. This is still pretty high. Due you have the postfix queue and the db files spread across more than one spindle? I ask because the xServe doesn't have an inbuilt RAID controller (optional card), and only has 3 disk bays for slow (7200rpm) SATA disks. The db insertion time seems to hint that disk throughput may not be an issue for the db subsystem. If you have the db files sitting on fast external RAID or SAN storage, but have the postfix queue on the single system boot disk along with a page file partition, etc, this may explain why you're sync'ing so quickly to db yet it takes so long to pipe the emails to the db insertion app. Your queue disk may not be fast enough for the load being generated on it. This one disk is trying to pump ~600,000 messages/day (in+out), and most of that will peak during two or three 1-2 hour intervals during the business day--arrival, after lunch, end of day. -- Stan
