----- Original Message ----- From: "Eric Ziegast" <[EMAIL PROTECTED]> To: "Marcos Dutra" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Monday, July 21, 2003 11:12 PM Subject: Re: [vchkpw] Big server
> The client mail server would serve whatever combination > I would like make a big server with qmail +vpopmail +mysql +procmail. > I think in this structure: > Server 1: Mx domain + smtp delivery +filters (Antispam, user filter(procmail) > and antivirus) > This server basically is the mail gateway of all domains, where is passed in > the filters rules per domain and redirect all mails to server 2 > > Server 2: Pop3 accounts + mysql server > Here is created all accounts. > > This schema is good for multiple domains? Based on my experience, I agree, but I might split Server2 into Server2 (delivery/storage/database) and Server3 (pop/imap/webmail servers for clients). I include more details below for one economical infrastructure I worked with. It's not a HOWTO, but knowing what someone else has done might help guide you instead of figuring it out from scratch. > Another question is how do i do the message delivery the messages from > server 1 to server2? Qmail! :^) In /var/qmail/control/smtproutes, set it so that all mail goes to Server2 (eg: ":server2.mydomain.com"). If you're fancy, you can try QMTP instead of SMTP. -- Eric Ziegast A sample large server environment (hundreds of domains, thousands of users) I once helped with: The MX record points to multiple cheap parallel inbound mail servers: - Single CPU PC at the best Price/Performance cost. I've found that one can build these for $300 each. You will find that when doing Virus/Spam scanning that the first bottleneck that you hit (out of CPU/memory/disk/network) is CPU. All of the regular expression searching on an e-mail message takes processing power. Assuming you have enough RAM, disk I/O would be the next bottleneck. I found a good balance AMD 1800+ motherboard /w 512MB PC133 RAM and 7200RPM IDE. Another option is investing in a very fast multi-processor Intel screamer with lots of RAM, but the cheap and disposable dervers are linearly scalable. - RAM depends on how many simultaneous connections you want to be allowed for Spam/Virus filtering. I used 512MB on a cheap system becasue RAM is cheap these days. I usually ran out of CPU before memory. If the OS uses any significant amount of virtual memory, you need more RAM. Run vmstat. If "pi" or "po" is above 0, you need more RAM or need to lower the number of simultaneous connections allowed by qmail (eg: "concurrencyincoming" in /var/qmail/control). The inbound server is your "mail firewall" and doesn't have time for paging to disk when the message load is high. - Hardware or softare RAID1 7200+ RPM IDE drives is sufficient. I have been told by a Linux integrator that Linux software RAID1 can be faster than the RAID1 provided by hardware controllers. If you have a budget for SCSI, use it. You need merely a 9GB drive in an inbound relay server anyway because the mail doesn't sit on the server. In fact, you may see a disk I/O improvement if you limit /var/qmail/queue to a 2GB partition of the hard drive. If you don't need the space, you don't need to have the disk head potentially cross the entire disk to find data. If you select a hardware RAID controller, prefer a controller that has non-volatile RAM or RAM /w a battery. This will allow the controller to use write-back mode on write and significantly reduce response time between the computer and the hard drives. - While I love OpenBSD and FreeBSD, I've used Linux for Qmail services because I've had other Linux-capable staff that could help administer the servers. Another advantage to Linux is ReiserFS. I have used ReiserFS on /var/qmail/queue partitions with success after applying the fsync patches. (http://www.jedi.claranet.fr/qmail-reiserfs-howto.html) ReiserFS performs well with thousands of files in a directory and allows you to keep the default hash value (23) for the spool directory. If using ufs (Solaris/BSD), consider compiling a queue hash value of some large prime number (like 101). If using Linux without ReiserFS, at least use ext3 instead of ext2 so that you can recover after a crash. If using Solaris, consider VxFS if you have the ability to use it. A standard fsck of a non-journaled filsystems used for qmail REALLY sucks. Aside: I don't export ReiserFS over NFS - just use it for the mail relays themselves. For vpopmail directories, I use filesystems that are known to be tried and tested in heavy read/write environments under NFS. I hope ReiserFS gets to this state, but at the time of my implementation, it was easier for me to use ext3 for vpopmail dirs. - I followed instructions for using QmailScanner /w SpamAssassin (spamc -f -c) and a Virus checker. I found QmailScanner to be quite inefficient and significantly rewrote it to not break up the message into a zillion pieces for its internal scanning. SpamAssassin (spamd) does that for you anyway, and so do virus scanning software. I'd have the qmail-scanner programs mark the messages with a "X-Spam-Status" and "X-Virus-Status" header and forward the message to my central mail server. Filtering software (your choice is procmail) on the central mail server wouldn't have to scan messages, just headers to figure out what to do with the message. - One well-balanced server was able to handle 40 simultaneous connections while filtering each message. I started with two servers. If the load increased, I could add more. At $400 each, using generic mid-tower boxes, it was easy to justify new boxes. The central mail storage server: - The vpopmail directories and mysql server should be servered from a machine with a reliable motherboard and reliable storage. CPU load isn't necessary. Memory isn't necessary. 512MB RAM is still sufficient, and a 1GHz processor was still plenty. If you have access to a Network Appliance back-end, I recommend it highly, but if you're like me, you don't have much of a budget. I would use fast SCSI 73GB drives (10000+ RPM) in a RAID1 configuration. I would have a hot-spare drive. I would have NVRAM on the RAID controller if possible. If using Linux, I'd use ext3 if NFS were required (ReiserFS is this is also my POP/IMAP server). If I need more storage, I'd add more RAID1 pairs to the SCSI chain. You main bottleneck is likely to be disk I/O. Use iostat to see if read/write requests need to wait. If - In one setup, I put mysql and final qmail delivery into a large spool on one server. I then had POP3 clients and Web clients access the spool via NFS. If I needed more I/O between clients and theirmail, I culd add more cheap client servers. I used hardware RAID0+1 SCSI storage on the main Vpopmail home directory with ext3 (not ReiserFS because NFS reliability for ReiserFS had not yet been proven). I used 73GB partitions. If you have a good budget, consider using Network Appliance filers for your back end storage. They're fast and reliable and is amazingly good at random disk I/O when you use many spindles in a RAID group. - If something catostrophic ever happens to your qmail/vpopmail storage server, a user might live with an outage for a couple hours, but not for a full day. I would have a backup IDE disk that would be ready to to serve in the event that the primary storage is unavailable. I would do rsyncs between the primary storage and the backup drive as often as possible (at least once a day). The directory structure with alot of e-mail accounts would take alot of time to recreate. Having it available on a disk nearby helps. I haven't had to use a backup disk, but I sleep better knowing it's there. I know the backup disk would be slower than primary storage and add a bottleneck, but it's better than being down. - If you're going to use procmail or other filtering software, consider doing what you can to minimize the impact of the filtering on the storage server. - One method is having all filtering done on the inbound servers and perform final mailbox delivery from the filtering servers into an NFS vpopmail spool. Synchronization and idempotency of NFS writes shouldn't be an issue for qmail, but it could seriously lock up inbound delivery if NFS ever fails. ("mount -o rw,soft,intr,noatime ...") - Another method is having filtering and final delivery done on the mail storage server itself. There's less network traffic with this method compared to NFS, and qmail's queueing allows the inbound servers to handle an outage on the main mail server gracefully (sessions tempfail and queu up), but now there's significant processing being done on the central mail server. The central server could become a processing bottleneck, and it's possible that a high enough load for procmail filtering could make it less responsive to clients (POP, etc). You never want to overwhelm your file server doing non-file-server tasks. If you run procmail on the central storage server, at least use "nice 10 COMMAND". I'm also wary of the security implications of running promail on the same server central to mail processing for everyone. What if someone introduces a program as part of the procmail filters that has a security bug or sucks up all available CPU on the server? I generally shy away from this scenario. - At one site, I implemented a batch processing system. Mail would be "delivered" by vdelivermail program into a user's mail spool on the central storage server. The user would not be able to run any filters on inbound mail. By "delivered", I mean that the message would exist in ~user/Maildir/tmp/MESSAGE#. I would have vdelivermail stop before moving the message into the "new" subdirectory and instead append the message delivery information to a file that got rolled every minute. After a periodic delay of at least one minute, another program on the storage server would look at the list of new messages and serially process them through user-controlled filers (a jailed perl script). The customers are given some knobs for tuning (spam score threshold, sender address, recipient address, content search) and the filtering program would apply them to each batched message for final delivery. In final delivery, the message is moved into "~user/Maildir/new" or "~user/Maildir/junk" or some user-specified CourierIMAP folder. The MySql database contains a list of user preferences for their mail filtering with this program. The message is optionally tagged by the filter with "spam" in case a POP user wants to use that subject tag for filtering in their outlook mail client. The web mail and POP and IMAP servers are simply scalable via NFS: - Platform: up to 1GHz, 1GB RAM, single-disk PCs ($200) or more PCs similar to the inbound relays. - User vpopmail directories are seen via an NFS mount of the central mail storage server. - Install user-based web services (eg: pop3d, imapd, apache, sqwebmail), and SSL wrappers. Note: The nubmer of simultaneous sessions depends most upon available RAM. More sessions -> more RAM. - This server could also be the outbound SMTP server for users. One could take the POP logs and use them for POP-before-SMTP authentication, or the qmail-smtpd service could be configured for SMTP-auth with queries against the cenral MySQL server on the mail storage server. With multiple servers, SMTP-auth works better, but it is possible to create a summary of POP traffic on all servers to build a central POP-before-SMTP database. Scaling: - The database and storage server must be reliable. This server is not linearly scalable, but if properly designed, one can process a significant amount of storage before needing to upgrade. To scale this out to multiple servers, one needs a way of hashing/routing inbound mail to the correct storage server - think of how "vdominfo -d" or "vuserinfo -d" or /var/qmail/users/access are used by vdelivermail/qmail and how one might be able to use symlinks to point users into different NFS mounts. Adding a user or domain becomes more complicated if one can't use a single directory tree for their mail storage. - The mail filtering servers must have good CPU, but don't need to be reliable. They are linearly scalable. - The client access processors don't need to be reliable if you have some method for failover (IP address takeover, LVS) or load balancing (Alteon, Foundry, BigIP, etc). They are linearly scalable. Outbound smtp works better with a local "dnscache" DNS resolver (http://cr.yp.to/djbdns) than a local nameserver on the network. This is an example that can service 20000+ users well. One could decide to scale the single database/storage server with some sort of server redundancy, or create clusters of cheaper storage/database servers paired with relay/access servers. One would need a method, though, of managing accounts across multiple clusters. Scaling to 1000000 users or more would involve commercial storage systems (eg: Celerra or NetApp), load balancers (eg: Alteon), hashing users or domains into multiple clusters of access servers and storage/database managers, and custom scripting to manage the creation and removal of domains and users. One might also start looking at the cost/benefit of commercial software packages when dealing with large user bases. "Life With Qmail" is your friend: http://LifewithQmail.org . "Qmail-Scanner" is a good beginning Virus/Spam integration tool for Qmail - http://qmail-scanner.sourceforge.net/ . Hi Can you pleas explain to me how you got the whole procmail thing to work? I have qmail+ Maildir, and vpopmail setup, yet I cannot redirect mail comming in for any other user except the default user who downloads the pop3 ETRN email. Please help? Kind Regards Rudi Ahlers OcÚ Software Support +27 (11) 661-9000 "The basic difference between an ordinary man and a warrior is that a warrior takes everything as a challenge while an ordinary man takes everything either as a blessing or a curse."