Re: [vchkpw] NFS / Disk Access / Load Concerns on Vpopmail cluster
What we're seeing is that our network and RAID 5 IDE-based disk array on our central mail store server is not able to keep up with the 'client' servers doing the POP3, IMAP, Webmail, and SMTP legwork. I've found an interesting bottleneck with webmail. When people use POP or IMAP clients (Outlook, Mozilla, Opera, Thunderbird, etc.), the client application caches alot of the information locally and synchronizes occasionally with the server to see if there are new messages. Things like browsing and searching run eally fast because the user is utilizing the resources of their local PC to do most of teh work. With webmail, the session state is not saved nor cached, so with each new request, the mailbox can be rescanned. A relatively modest webmail application might only rescan all headers and show subject lines. A complex application might scan all content in a folder to present content more fully. Without anything to throttle back the webmail server, it's possible that the webmail server softwar can pound the mail spool server to death. I used to run a Qmail-based infrastructure for 4000 clients on a single slow machine without much memory. They used POP as their only pickup mechanism. We recently reimplemented on a Dell 1750 with two Xeon procs, alot of RAM and a GigE backend to a NetApp filer with 14 fast disks, and I STILL notice that the machine sometimes slowed down while people tried to read their 140MB mailboxes via webmail. sigh I put some bottlenecks on the search and retrieval algorithms of the webmail software to help protect the filer from a flood of queries, and we've been better since then. The power users with super-large mailboxes complain that it's slow, but now its a localized problem rather than a problem that affects everyone. Jeremy's comments are great for scaling the database, but it sounds to me that you're just maxed out on what you can serve over NFS. An SQL select might take at most a few kilobytes of data on the network whereas a webmail scan of a 30MB mailbox will take, well, 30MB. Doh! So what to do? Instead of the centralized NFS mail spool (where the central spool becomes the bottleneck), you might consider splitting the user base across several machines. Each machine would have its own RAID1 mail spool. Each machine would be responsible for its own Inbound SMTP and POP/IMAP/Webmail and use the local disk for the spool. Use lots of RAM for buffer cache to make sure your disk is hit less frequently. You might be able to centralize outbound SMTP. Once a machine fills up, you add another machine. This is one way to scale. The big boys in teh mailbox size wars (google, yahoo, hotmail) can't afford centralized storage for their mailboxes. Look for each to roll out racks of distribtuted storage where each storage server is a 1/2 U box with a couple large ATA disks in it. We might learn from this method of scaling. Before we take this costly step, what have you noticed for user / system loads before you start hitting the limits of your hardware? Yes. I serve 6000 users right now. They used to all be POP, and life was good. Now a significant percentage of my new customers use webmail, and I'm not happy with how my current web-based mail reading software scales. I may have to hack it alot to get it to perform well. Something that would help is if we rolled out spam/virus filtering out for everyone whih will cut 50% inbound mail and 10% viruses from being processed/stored/read and reread/reread/reread. BTW: I separate SMTP processing (/var/qmail local RAID1 fast SCSI with battery cache) from user mail spool storage (/home/vpopmail NFS mount to filer). Putting /var/qmail on the NFS server might be another source of overload. -- Eric Ziegast
[vchkpw] Delete accounts in mysql
Hi, I run vpopmail with mysql backend, and have a web front-end to admin stuff. The creation and drop of accounts works accessing the mysql table. The creation of accounts works ok, but when I delete a user in the mysql table and create it again, it can't log in. Imap tells login failed but vuserinfo show correct information. Any idea about ? Thanks -- David Sedeño Fernández [EMAIL PROTECTED]
Re: [vchkpw] SATA NAS for vpop cluster
On Wed, 30 Jun 2004 11:16:02 -0700, Jay Tortorelli [EMAIL PROTECTED] wrote: I'll chime in a say that I have had great success setting up Qmail/Vpopmail/etc that stores on an nfs mount from a two machine mirror using drbd/heartbeat to provide mirroring and failover. http://www.drbd.org/ Using commodity hardware, it really does save you a lot of money over the Netapps with clustering and I wouldn't even consider it a gamble...just based on my experiences. Jay I played with drbd when it was first being developed but haven't touched it since. I am just curious if have any statistics on the max and average NFS operations per second your drbd/nfs host is doing? At a previous job we had NetApp 840 Filers and were doing 35k nfs op/sec on average with spikes to 60k. -- Sean
Re: [vchkpw] RBLs
Jeremy Kitchen wrote: On Monday 05 July 2004 08:44 pm, Eduardo M. Bragatto wrote: What happens in the case that more than one RBL has the same blocked address? Which one actually blocks the smtp session? The one that answers first, or rblsmtpd waits until one to respond, before asking to another one? In that case, it askes in the same order that the parameters are given? the first one. If rblsmtpd finds a match, it doesn't bother wasting any more bandwith on the fool, and sends them a nice FU :) So, in that case, I may think that the first list given is more effective than others, since it will always be checked first and because of that, may blocks more than others... Is it right? I'm asking it because, like Simon (who started this thread), I also noticied more lists blocking than others... Best regards, Eduardo M. Bragatto.
Re: [vchkpw] RBLs
At 07:14 AM 7/6/2004, Eduardo M. Bragatto wrote: Jeremy Kitchen wrote: On Monday 05 July 2004 08:44 pm, Eduardo M. Bragatto wrote: What happens in the case that more than one RBL has the same blocked address? Which one actually blocks the smtp session? The one that answers first, or rblsmtpd waits until one to respond, before asking to another one? In that case, it askes in the same order that the parameters are given? the first one. If rblsmtpd finds a match, it doesn't bother wasting any more bandwith on the fool, and sends them a nice FU :) So, in that case, I may think that the first list given is more effective than others, since it will always be checked first and because of that, may blocks more than others... Is it right? I'm asking it because, like Simon (who started this thread), I also noticied more lists blocking than others... here's my call to the rbl's. i prefer to give some info in the 'FU' response, to at least give them a clue where to start. we dropped spamcop a while back, as they were listing some sites that - while they may at times be sources of spam - are not in the main spam sites - such as tropica. we had a number of customer complaints from people who were subscribed to legitimate mailing lists through tropica, when spamcop did a blanket blacklist of their address space. uncool. anyway, here's what we use: /usr/local/bin/rblsmtpd -C \ -a whitelist.example.com \ -r sbl-xbl.spamhaus.org:\ Probable spam connection rejected. Details at http://www.spamhaus.org; \ -r list.dsbl.org:\ Probable spam connection rejected. Details at http://www.dsbl.org; \ -r relays.ordb.org:\ Probable spam connection rejected. Details at http://www.ordb.org/faq; \ -t10 \ /var/qmail/bin/qmail-smtpd 21 the whitelist call is to put in some custom rules by customer request, though ultimately it's simpler to just list them in /service/smtpd/tcp. the '-t10' ensures that if one of the rbls isn't answering, it doesn't hang up smtp connections for a long time waiting. Paul Theodoropoulos http://www.anastrophe.com http://www.smileglobal.com
Re: [vchkpw] NFS / Disk Access / Load Concerns on Vpopmail cluster
Eric Ziegast wrote: What we're seeing is that our network and RAID 5 IDE-based disk array on our central mail store server is not able to keep up with the 'client' servers doing the POP3, IMAP, Webmail, and SMTP legwork. I've found an interesting bottleneck with webmail. When people use POP or IMAP clients (Outlook, Mozilla, Opera, Thunderbird, etc.), the client application caches alot of the information locally and synchronizes occasionally with the server to see if there are new messages. Things like browsing and searching run eally fast because the user is utilizing the resources of their local PC to do most of teh work. With webmail, the session state is not saved nor cached, so with each new request, the mailbox can be rescanned. I think, if you use sqwebmail, it *will* cache some information. I've got a very large mailbox, with over 5 messages (though split in 100 directories) amounting to over 350 MB of mail, mostly mailinglists like this one. When I open a folder the first time in sqwebmail, it takes a lot of time, but the second time, it's rather quick (as quick as opening a folder with 3000 messages can be). I like sqwebmail, though I sometimes think I'm the only one and the rest of the world wants squirrelmail and IMP ;-) A relatively modest webmail application might only rescan all headers and show subject lines. A complex application might scan all content in a folder to present content more fully. Without anything to throttle back the webmail server, it's possible that the webmail server softwar can pound the mail spool server to death. I used to run a Qmail-based infrastructure for 4000 clients on a single slow machine without much memory. They used POP as their only pickup mechanism. We recently reimplemented on a Dell 1750 with two Xeon procs, alot of RAM and a GigE backend to a NetApp filer with 14 fast disks, and I STILL notice that the machine sometimes slowed down while people tried to read their 140MB mailboxes via webmail. sigh I put some bottlenecks on the search and retrieval algorithms of the webmail software to help protect the filer from a flood of queries, and we've been better since then. The power users with super-large mailboxes complain that it's slow, but now its a localized problem rather than a problem that affects everyone. Good tip. You can try to run up-imapproxy (if you don't do that allready) and see if it helps. It will try to cache at least the IMAP-sessions. Jeremy's comments are great for scaling the database, but it sounds to me that you're just maxed out on what you can serve over NFS. An SQL select might take at most a few kilobytes of data on the network whereas a webmail scan of a 30MB mailbox will take, well, 30MB. Doh! I'd also like to add that people perhaps overestimate what IDE-RAID can do compared with a true SCSI-RAID - especially in cases where a horrendous amount of small, scattered files and highly concurrent access is envolved (hello qmail). I always joke that nothing can beat a (current) IDE-disk when installing Windows and Office - they are optimized for rather large files and sequential access to these. But mail-spool (/var/qmail/queue/) and mail-storage (~vpopmail/) ain't an Office-installation So what to do? Instead of the centralized NFS mail spool (where the central spool becomes the bottleneck), you might consider splitting the user base across several machines. Each machine would have its own RAID1 mail spool. Each machine would be responsible for its own Inbound SMTP and POP/IMAP/Webmail and use the local disk for the spool. Use lots of RAM for buffer cache to make sure your disk is hit less frequently. You might be able to centralize outbound SMTP. Once a machine fills up, you add another machine. This is one way to scale. The big boys in teh mailbox size wars (google, yahoo, hotmail) can't afford centralized storage for their mailboxes. Look for each to roll out racks of distribtuted storage where each storage server is a 1/2 U box with a couple large ATA disks in it. We might learn from this method of scaling. I'd be interested to know how one can achieve this while still maintaining the single-system-image-nature that a central mail-storage with surrounding mysql-slaves provides. Not that I want to start a we're-bigger-than-google-kind of freakshow, but just in case I hit the wall with the current system. Before we take this costly step, what have you noticed for user / system loads before you start hitting the limits of your hardware? Yes. I serve 6000 users right now. They used to all be POP, and life was good. Now a significant percentage of my new customers use webmail, and I'm not happy with how my current web-based mail reading software scales. I may have to hack it alot to get it to perform well. Something that would help is if we rolled out spam/virus filtering out for everyone whih will cut 50% inbound mail and 10% viruses from being
Re: [vchkpw] NFS / Disk Access / Load Concerns on Vpopmail cluster
On Mon, 5 Jul 2004 [EMAIL PROTECTED] wrote: Before we take this costly step, what have you noticed for user / system loads before you start hitting the limits of your hardware? Should we be having these issues with about 15,000 email users and 5 front-end 'work' servers? Well, you're making me feel better about having only 3500 or so accounts on one box. The whole ordeal is making me re-think a few things about the design: -vpopmaild will be nice - no need to have webmail on the same box -mysqld works best on it's own box, period. -looking into front-ending with postfix will probably let me squeeze more out of the same hardware; qmail really thrashes the box around, especially if you have many over-quota users clogging up your queue with bogus spam bounces. Charles -Simon
Re: [vchkpw] NFS / Disk Access / Load Concerns on Vpopmail cluster
Rainer Duffner wrote: Eric Ziegast wrote: What we're seeing is that our network and RAID 5 IDE-based disk array on our central mail store server is not able to keep up with the 'client' servers doing the POP3, IMAP, Webmail, and SMTP legwork. I've found an interesting bottleneck with webmail. When people use POP or IMAP clients (Outlook, Mozilla, Opera, Thunderbird, etc.), the client application caches alot of the information locally and synchronizes occasionally with the server to see if there are new messages. Things like browsing and searching run eally fast because the user is utilizing the resources of their local PC to do most of teh work. With webmail, the session state is not saved nor cached, so with each new request, the mailbox can be rescanned. I think, if you use sqwebmail, it *will* cache some information. I've got a very large mailbox, with over 5 messages (though split in 100 directories) amounting to over 350 MB of mail, mostly mailinglists like this one. When I open a folder the first time in sqwebmail, it takes a lot of time, but the second time, it's rather quick (as quick as opening a folder with 3000 messages can be). I like sqwebmail, though I sometimes think I'm the only one and the rest of the world wants squirrelmail and IMP ;-) There is at least one other sqwebmail user out here... I like the fact that it directly accesses maildirs rather than opening a connection via tcp/ip to retrieve mail. Rick
Re: [vchkpw] NFS / Disk Access / Load Concerns on Vpopmail cluster
Charles Sprickman wrote: On Mon, 5 Jul 2004 [EMAIL PROTECTED] wrote: -vpopmaild will be nice - no need to have webmail on the same box vpopmaild has nothing to do with webmail, although I guess you could use it to retrieve mail through the back door. I'd suggest IMAP or POP3 based webmail instead. vpopmaild is for administration of mail accounts. Rick
Re: [vchkpw] NFS / Disk Access / Load Concerns on Vpopmail cluster
On Mon, 5 Jul 2004, Jeremy Kitchen wrote: Also, on top of that, I would consider disabling auth logging as it performs an insert/update upon every authentication which, no matter what will go back to your central mysql server, and if you have mysql being replicated, will be replicated to the front machines, which will almost nearly negate any performance increases you may (and very likely will) see by switching to a replicated mysql configuration. Interesting... Refresh my memory on this, is it a compile-time switch? How does vpopmail behave if that table does not exist? Also! (last one, I promise!) if you're using vpopmail's roaming users support, stop now. completely disable roaming users in your vpopmail configuration and set up Bruce Guenter's relay-ctrl package (http://untroubled.org/relay-ctrl). No funky cronjob to run, no patches required to ucspi-tcp (there's a patch out there to make it talk to mysql, eek) no central cdb file to rebuild upon connection attempts, AND it's safe to mount the spool directory on NFS (I've done it) as it doesn't require locking or anything. Hmmm... Good suggestion, is there anything similar that will deal with Courier's pop3d? Do you have a rough feel for at what point trying to decrease updates will help things along? 2000 users? 10,000 users? Thanks, Charles Hope this helps. -Jeremy -- Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc. [EMAIL PROTECTED] ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail
Re: [vchkpw] NFS / Disk Access / Load Concerns on Vpopmail cluster
On Mon, 5 Jul 2004, Eric Ziegast wrote: With webmail, the session state is not saved nor cached, so with each new request, the mailbox can be rescanned. A relatively modest webmail application might only rescan all headers and show subject lines. A complex application might scan all content in a folder to present content more fully. Without anything to throttle back the webmail server, it's possible that the webmail server softwar can pound the mail spool server to death. There's two things I'm testing. The first is Turck-MMCache, which is a caching lib for php, similar to the commercial Zend stuff. It's free though. It saves all the overhead of compiling your webmail stuff on every hit (it caches pre-compiled code). This of course assumes you're using a php-based webmail app. The other thing I'm looking at is an IMAP cache, which could probably help with the problems you're seeing as well. The squirrelmail list has some good info on this. I used to run a Qmail-based infrastructure for 4000 clients on a single slow machine without much memory. The machine that this mail is going through is an AMD K6-2-450. It has maybe 20 or 30 mailboxes, tops. For years it has been more than adequate for this small task, but with a few catchall domains and no chkuser patch (I don't trust it as it rejects mail if mysql is not zippy enough) it can really get bogged down during spam runs. I'm sick of building boxes to accomodate all the spam; I need to build for the 85% of mail that just gets thrown away by SpamAssassin. That's my spam rant. :) Instead of the centralized NFS mail spool (where the central spool becomes the bottleneck), you might consider splitting the user base across several machines. Each machine would have its own RAID1 mail spool. Each machine would be responsible for its own Inbound SMTP and POP/IMAP/Webmail and use the local disk for the spool. Use lots of RAM for buffer cache to make sure your disk is hit less frequently. You might be able to centralize outbound SMTP. Once a machine fills up, you add another machine. This is one way to scale. That's been my plan, but my problem is that most of my users are all under one domain, so I'm not really sure how I can divvy up the users without yet another box doing mx in front and splitting the mail up... The big boys in teh mailbox size wars (google, yahoo, hotmail) can't afford centralized storage for their mailboxes. Look for each to roll out racks of distribtuted storage where each storage server is a 1/2 U box with a couple large ATA disks in it. We might learn from this method of scaling. Hopefully one of them will do a nice Usenix presentation like Earthlink did back in the day... Thanks, Charles -- Eric Ziegast
Re: [vchkpw] NFS / Disk Access / Load Concerns on Vpopmail cluster
On Tue, 6 Jul 2004, Rick Widmer wrote: Charles Sprickman wrote: On Mon, 5 Jul 2004 [EMAIL PROTECTED] wrote: -vpopmaild will be nice - no need to have webmail on the same box vpopmaild has nothing to do with webmail, although I guess you could use it to retrieve mail through the back door. I'd suggest IMAP or POP3 based webmail instead. In my case, it will. I have a few custom-built squirrelmail plugins that rely on being able to run various v* commands, so at this time I can't put webmail on a seperate box. When I finally upgrade vpopmail to 5.4.x, I can rewrite all that stuff to work with vpopmaild... Thanks, Charles vpopmaild is for administration of mail accounts. Rick
[vchkpw] chkuser patch
Hi, I believe there was some discussion about this some time ago, but recent events have made me think of this again... The standard chkuser patch that vpopmail uses (see Bill Shupp's update to the original: http://www.shupp.org/patches/chkuser-0.6.mysql.patch) has some rather nasty behaviours. If for some reason your mysql server is unavailable (load has shot so high that mysql can't return a prompt reply, you're upgrading mysql, taking db down for maintenance, etc.) the chkusr patch will start telling remote smtp clients that the user doesn't exist. This is not good; you never want to send a 550 on a user that really does exist; people get upset when things bounce, and mailing lists start looking at auto-removal. Has anyone found a better solution that integrates well with vpopmail? The goodrcptto patch is interesting, but too much work to maintain. I'm also wondering if there are any plans for the vpopmail stuff to support talking to multiple databases; ie: if no answer or timeout on one, hit a slave that's replicating off the master. Thanks, Charles
Re: [vchkpw] chkuser patch
Charles Sprickman wrote: I'm also wondering if there are any plans for the vpopmail stuff to support talking to multiple databases; ie: if no answer or timeout on one, hit a slave that's replicating off the master. I don't see anything happening with multiple databases beyond specifying a read database and an update database, but I have made some changes in vpopmail that will allow the patch to tell the difference between 'Unable to open database' and 'user does not exist'. Once that code gets released the patch author can return a temporary failure on 'can't open', and correctly report 'user does not exist' when appropriate. I am just starting a new job, so my open source efforts are a bit behind... Rick
