Eric Ziegast wrote:
What we're seeing is that our network and RAID 5 IDE-based disk array on our central mail store server is not able to keep up with the 'client' servers doing the POP3, IMAP, Webmail, and SMTP legwork.
I've found an interesting bottleneck with webmail. When people use
POP or IMAP clients (Outlook, Mozilla, Opera, Thunderbird, etc.),
the client application caches alot of the information locally and
synchronizes occasionally with the server to see if there are new
messages. Things like browsing and searching run eally fast because
the user is utilizing the resources of their local PC to do most of
teh work. With webmail, the session state is not saved nor cached,
so with each new request, the mailbox can be rescanned.
I think, if you use sqwebmail, it *will* cache some information.
I've got a very large mailbox, with over 50000 messages (though split in 100 directories) amounting to over 350 MB of mail, mostly mailinglists like this one.
When I open a folder the first time in sqwebmail, it takes a lot of time, but the second time, it's rather quick (as quick as opening a folder with 3000 messages can be).
I like sqwebmail, though I sometimes think I'm the only one and the rest of the world wants squirrelmail and IMP ;-)
A relatively modest webmail application might only rescan all headers and show subject lines. A complex application might scan all content in a folder to present content more fully. Without anything to throttle back the webmail server, it's possible that the webmail server softwar can pound the mail spool server to death.
I used to run a Qmail-based infrastructure for 4000 clients on a single slow machine without much memory. They used POP as their only pickup mechanism. We recently reimplemented on a Dell 1750 with two Xeon procs, alot of RAM and a GigE backend to a NetApp filer with 14 fast disks, and I STILL notice that the machine sometimes slowed down while people tried to read their 140MB mailboxes via webmail. <sigh> I put some bottlenecks on the "search" and retrieval algorithms of the webmail software to help protect the filer from a flood of queries, and we've been better since then. The power users with super-large mailboxes complain that it's "slow", but now its a localized problem rather than a problem that affects everyone.
Good tip.
You can try to run up-imapproxy (if you don't do that allready) and see if it helps.
It will try to cache at least the IMAP-sessions.
Jeremy's comments are great for scaling the database, but it sounds to me that you're just maxed out on what you can serve over NFS. An SQL select might take at most a few kilobytes of data on the network whereas a webmail scan of a 30MB mailbox will take, well, 30MB. Doh!
I'd also like to add that people perhaps overestimate what IDE-RAID can do compared with a true SCSI-RAID - especially in cases where a horrendous amount of small, scattered files and highly concurrent access is envolved (hello qmail).
I always joke that nothing can beat a (current) IDE-disk when installing Windows and Office - they are optimized for rather large files and sequential access to these.
But mail-spool ("/var/qmail/queue/") and mail-storage (Â~vpopmail/") ain't an Office-installation....
So.... what to do?
Instead of the centralized NFS mail spool (where the central spool becomes the bottleneck), you might consider splitting the user base across several machines. Each machine would have its own RAID1 mail spool. Each machine would be responsible for its own Inbound SMTP and POP/IMAP/Webmail and use the local disk for the spool. Use lots of RAM for "buffer cache" to make sure your disk is hit less frequently. You might be able to centralize outbound SMTP. Once a machine "fills up", you add another machine. This is one way to scale.
The big boys in teh mailbox size wars (google, yahoo, hotmail) can't afford centralized storage for their mailboxes. Look for each to roll out racks of distribtuted storage where each "storage server" is a 1/2 U box with a couple large ATA disks in it. We might learn from this method of scaling.
I'd be interested to know how one can achieve this while still maintaining the "single-system-image"-nature that a central mail-storage with surrounding "mysql-slaves" provides.
Not that I want to start a "we're-bigger-than-google"-kind of freakshow, but just in case I hit the wall with the current system.
Before we take this costly step, what have you noticed for user / system loads before you start hitting the limits of your hardware?
Yes. I serve 6000 users right now. They used to all be POP, and life was good. Now a significant percentage of my new customers use webmail, and I'm not happy with how my current web-based mail reading software scales. I may have to hack it alot to get it to perform well.
Something that would help is if we rolled out spam/virus filtering out for everyone whih will cut 50% inbound mail and 10% viruses from being processed/stored/read and reread/reread/reread.
But the spamassassin + antivirus -scanning also take a big toll - you'd probably still have to do what Jeremy suggested... ;-)
cheers, Rainer
