Re: New mail server setup
Michael K. Smith - Adhost wrote: Steve Bertrand wrote: If anyone has a setup that has redundancy for their IMAP/POP services, and a method to keep the changing data relatively up-to-date, I'd love to hear about it. [ big snip ] Another approach would be a cluster of Postfix servers and Dovecot servers behind PF load balancers. We have 3 POP servers (IMAP/POP), 9 Mail Servers, 2 Defer servers and 5 Filter servers that process over 20 million messages a day without a blip. We can take individual servers out of the pool for maintenance, etc. Everything is fed to a set of redundant NAS for the data storage and common configuration files. Thanks Mike, I'm interested to learn a little more about your setup. I was going to take it off-list, but if you can provide some further details, it would probably add long-term value to keep it here. So, a couple of questions: - can your PF load balancers 'sense' when one of the Postfix/Dovecot units are down, or is this a manual change in config to prevent any time-out conditions? I like this load balancer idea. In my environment, it would be trivial to set up a couple of them, throw Quagga on them, and integrate them directly into our iBGP setup. On the other side, I could use VRRP or the like to ensure redundancy from front to back. - do the Postfix/Dovecot servers communicate with each other, or are they simply stand-alone units that don't know/care that they have other peers helping with the workload? - are your filter servers in front of, or behind the load balancers (iow, is all of your inbound email passed through the balancers, and then filtered/processed/delivered in behind them)? - how do all of the pieces communicate with the NAS...NFS? - could you share a small snip of your PF config in relation to load-balancing, so I can get a bit of a better understanding config-wise on how that piece hangs together? (I've never used PF, only IFPW ;) Thanks, and regards, Steve smime.p7s Description: S/MIME Cryptographic Signature
RE: New mail server setup
Hello Steve: I'll try to answer your questions in line. snip Another approach would be a cluster of Postfix servers and Dovecot servers behind PF load balancers. We have 3 POP servers (IMAP/POP), 9 Mail Servers, 2 Defer servers and 5 Filter servers that process over 20 million messages a day without a blip. We can take individual servers out of the pool for maintenance, etc. Everything is fed to a set of redundant NAS for the data storage and common configuration files. Thanks Mike, I'm interested to learn a little more about your setup. I was going to take it off-list, but if you can provide some further details, it would probably add long-term value to keep it here. So, a couple of questions: - can your PF load balancers 'sense' when one of the Postfix/Dovecot units are down, or is this a manual change in config to prevent any time-out conditions? Not natively. When we initially implemented this setup, ifstated wasn't up to snuff, so we wrote some PERL scripts that make connections to the required ports and, if no connection is established, pull the server from the table and send us an alarm. We also have scripts so that we can pull servers out when we're doing maintenance. I like this load balancer idea. In my environment, it would be trivial to set up a couple of them, throw Quagga on them, and integrate them directly into our iBGP setup. On the other side, I could use VRRP or the like to ensure redundancy from front to back. We use two PF boxes and CARP with PFSync for failover, so no dynamic protocols are needed. - do the Postfix/Dovecot servers communicate with each other, or are they simply stand-alone units that don't know/care that they have other peers helping with the workload? They are standalone. All of the user authentication is handled from a centralized database, so there are no local credentials stored on the server. - are your filter servers in front of, or behind the load balancers (iow, is all of your inbound email passed through the balancers, and then filtered/processed/delivered in behind them)? They are behind the PF boxes. We have other hooks in PF that we use to block SPAM in PF, including Cloudmark and some custom stuff that looks for multiple mails to non-existent addresses. We also use the overload tables for abusive connections. - how do all of the pieces communicate with the NAS...NFS? Yes. Originally we used TCP but we found performance to be much better with UDP. NFSv3 by the way. - could you share a small snip of your PF config in relation to load-balancing, so I can get a bit of a better understanding config- wise on how that piece hangs together? (I've never used PF, only IFPW ;) That might be difficult because it's about 720 lines. :-) Here are some highlights, though. 1) Our customers use mail.adhost.com for everything - SMTP, POP and IMAP. We use redirects in PF so that traffic coming in on the associated ports goes to the appropriate servers. 2) We have our load-balanced DNS servers behind the same PF boxes so we localize the tons of DNS queries related to mail. 3) We do a lot of our rejecting in PF, including Spamhaus, Cloudmark, check scripts for Phishing, Porn and Viruses, as well as our own list of Nefarious IP's culled from various sources. When traffic matches these originators, we send them to mail reject servers that send out a 550 message with the group name so we can find false positives more quickly. 4) Because 3 does have false positives, we have a whitelist that we can add to that will pass traffic to the mail servers before they match against any of the tables in 3. 5) We use POP before SMTP, so once we authenticate a user to send, their IP address is also added to an allow table. 6) The filter servers are load balanced to and from the mail servers so we can take them in and out of their pool for maintenance. If you have a particular scenario you're thinking about I could help you with the rules to make it work. Regards, Mike ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: New mail server setup
Michael K. Smith - Adhost wrote: - can your PF load balancers 'sense' when one of the Postfix/Dovecot units are down, or is this a manual change in config to prevent any time-out conditions? Not natively. When we initially implemented this setup, ifstated wasn't up to snuff, so we wrote some PERL scripts that make connections to the required ports and, if no connection is established, pull the server from the table and send us an alarm. We also have scripts so that we can pull servers out when we're doing maintenance. Ok. I've done the above in similar situations numerous times, so that works. I like this load balancer idea. In my environment, it would be trivial to set up a couple of them, throw Quagga on them, and integrate them directly into our iBGP setup. On the other side, I could use VRRP or the like to ensure redundancy from front to back. We use two PF boxes and CARP with PFSync for failover, so no dynamic protocols are needed. I'll have to review this further. I'm not overly familiar with CARP (ie I've never used it), nor PFSync. My mentality for infrastructure gear (the balancers, not the servers) is always make each device connect to two different switches/routers, and try to make it dynamic in a way that it fits into our OSPF/iBGP design, so if necessary, we can move the entire thing to a different network segment, and not have to renumber. I'm getting a mental picture how I can have load balancing failover with the two devices, and network resiliency by having each balancer connected to different network segments (between buildings over fibre if I want). - do the Postfix/Dovecot servers communicate with each other, or are they simply stand-alone units that don't know/care that they have other peers helping with the workload? They are standalone. All of the user authentication is handled from a centralized database, so there are no local credentials stored on the server. Perfect...do your auth/acct db's generally reside on the same storage mechanism that the data does, in order to keep 'email related stuff' altogether? - are your filter servers in front of, or behind the load balancers (iow, is all of your inbound email passed through the balancers, and then filtered/processed/delivered in behind them)? They are behind the PF boxes. We have other hooks in PF that we use to block SPAM in PF, including Cloudmark and some custom stuff that looks for multiple mails to non-existent addresses. We also use the overload tables for abusive connections. Ok. We have a Barracuda cluster hanging off of one of our Internet facing edge routers, that filters then passes what it allows back into the network, and to the servers. The only reason I don't aggregate all of the mail systems together, is so that I can filter the spam as soon as possible upon ingress to our network, instead of having it traverse the core. - how do all of the pieces communicate with the NAS...NFS? Yes. Originally we used TCP but we found performance to be much better with UDP. NFSv3 by the way. Ok. [ snip ] If you have a particular scenario you're thinking about I could help you with the rules to make it work. I do, and that would be fantastic! I'll draw up a diagram this afternoon of what I envision. Where I'll need a bit of advice will likely be in the details, as opposed to the design, especially if I migrate completely away from our existing mail platform(s). Cheers! Steve smime.p7s Description: S/MIME Cryptographic Signature
Re: New mail server setup
Steve Bertrand wrote: I'm looking potentially to try a different mail server setup. I'm requesting honest feedback from experienced mail ops. My minimum requirements: - IPv6 for all protocols - SPF - IMAP|POP3 must support SSL - SMTP AUTH - submit on 587 - MySQL backend for un/pw, vpopmail preferred, but not mandatory - Maildir storage preferred - easy (ie: well documented) integration with SA/clam - integration with maildrop .mailfiter preferred Right now I use a system wrapped around Qmail, and honestly, I just don't want to patch for IPv6 anymore. I've broken my personal system, so while I work on re-hacking everything, I thought I'd solicit some new ideas. I've been using the same email system pretty much across the board for seven years or so, so perhaps I should look at other options. Please cc me, as this addr isn't subscribed. I won't be receiving my list email from my backup mx until tomorrow, as it were ;) For an MTA: postfix does everything you want, it's not too shabby speed wise and the config files are reasonably comprehensible. For an IMAP/POP3 server: dovecot has the required functionality and unless you're dealing with thousands of user accounts it's probably a better alternative for you than the nuclear option of cyrus-imapd. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: New mail server setup
Matthew Seaman wrote: Steve Bertrand wrote: My minimum requirements: - IPv6 for all protocols - SPF - IMAP|POP3 must support SSL - SMTP AUTH - submit on 587 - MySQL backend for un/pw, vpopmail preferred, but not mandatory - Maildir storage preferred - easy (ie: well documented) integration with SA/clam - integration with maildrop .mailfiter preferred For an MTA: postfix does everything you want, it's not too shabby speed wise and the config files are reasonably comprehensible. For an IMAP/POP3 server: dovecot has the required functionality and unless you're dealing with thousands of user accounts it's probably a better alternative for you than the nuclear option of cyrus-imapd. Ok, I'm back up and rolling again. Thanks Matthew, and the others who replied off-list for all of the feedback. One thing that I forgot to ask in my original post was that of clustering. In our production network, we have a cluster of perimeter MX's, and a similar setup for our submission boxes (it's been a couple of years since we've strictly enforced AUTH for all clients). What I don't have, and have always wondered about, is live redundancy for the IMAP/POP services. I know that this would be a challenge to some degree considering the high volume of data changes. Perhaps a carp(4) setup between a couple of MDA's, where when the primary is up, a constant rsync pushes the data to the backup. Or perhaps a combination of rsync for manual changes, and a method to have the primary write the emails to a local disk, and a network disk simultaneously? If anyone has a setup that has redundancy for their IMAP/POP services, and a method to keep the changing data relatively up-to-date, I'd love to hear about it. Cheers, Steve smime.p7s Description: S/MIME Cryptographic Signature
Re: New mail server setup
Steve Bertrand wrote: What I don't have, and have always wondered about, is live redundancy for the IMAP/POP services. I know that this would be a challenge to some degree considering the high volume of data changes. Perhaps a carp(4) setup between a couple of MDA's, where when the primary is up, a constant rsync pushes the data to the backup. Or perhaps a combination of rsync for manual changes, and a method to have the primary write the emails to a local disk, and a network disk simultaneously? If anyone has a setup that has redundancy for their IMAP/POP services, and a method to keep the changing data relatively up-to-date, I'd love to hear about it. Now, that is a different kettle of fish. This is a job for cyrus imap. I suggest googling for 'cyrus murder' -- this is almost, but not quite, a fully resilient mail store / IMAP system. Your mail store is divided into frontend IMAP protocol servers which handle user auth etc. and back-end mail stores. The protocol layer servers are fully resilient and you can fail over a user session at will, but the mailstores don't quite get there: mail is replicated across different stores, but actions modifying the mail store are not transactional across all the mail stores. Or in other words, you can lose a small amount of data if one of the mail stores goes bang at precisely the wrong moment. Even so, it will do better at keeping multiple copies of a mailstore in synch than any locally scripted rsync setup. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. Flat 3 7 Priory Courtyard PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW, UK signature.asc Description: OpenPGP digital signature
Re: New mail server setup
Matthew Seaman wrote: Steve Bertrand wrote: If anyone has a setup that has redundancy for their IMAP/POP services, and a method to keep the changing data relatively up-to-date, I'd love to hear about it. Now, that is a different kettle of fish. This is a job for cyrus imap. I suggest googling for 'cyrus murder' -- this is almost, but not quite, a fully resilient mail store / IMAP system. Your mail store is divided into frontend IMAP protocol servers which handle user auth etc. and back-end mail stores. The protocol layer servers are fully resilient and you can fail over a user session at will, but the mailstores don't quite get there: mail is replicated across different stores, but actions modifying the mail store are not transactional across all the mail stores. Or in other words, you can lose a small amount of data if one of the mail stores goes bang at precisely the wrong moment. Even so, it will do better at keeping multiple copies of a mailstore in synch than any locally scripted rsync setup. This is *EXACTLY* what I was looking for! The possibility of loosing an extremely small amount of data far outweighs the possibility of a multi-hour outage where 3,000 users are receiving can't reach the POP3 server errors. Besides, our incoming SMTP gateway boxes cache all incoming email for 24 hours, and we can re-deliver any message to the back-end we wish during that window. I really try my best to design/implement all the systems I can like our networks... multiple paths and extremely quick convergence. Being able to take a box down to test/perform an upgrade, or during a failure without client impact is well worth any initial large learning curve imho. Thanks, Steve smime.p7s Description: S/MIME Cryptographic Signature
RE: New mail server setup
-Original Message- From: owner-freebsd-questi...@freebsd.org [mailto:owner-freebsd- questi...@freebsd.org] On Behalf Of Steve Bertrand Sent: Wednesday, September 16, 2009 7:09 AM To: Matthew Seaman Cc: questi...@freebsd.org Subject: Re: New mail server setup Matthew Seaman wrote: Steve Bertrand wrote: If anyone has a setup that has redundancy for their IMAP/POP services, and a method to keep the changing data relatively up-to-date, I'd love to hear about it. Now, that is a different kettle of fish. This is a job for cyrus imap. I suggest googling for 'cyrus murder' -- this is almost, but not quite, a fully resilient mail store / IMAP system. Your mail store is divided into frontend IMAP protocol servers which handle user auth etc. and back-end mail stores. The protocol layer servers are fully resilient and you can fail over a user session at will, but the mailstores don't quite get there: mail is replicated across different stores, but actions modifying the mail store are not transactional across all the mail stores. Or in other words, you can lose a small amount of data if one of the mail stores goes bang at precisely the wrong moment. Even so, it will do better at keeping multiple copies of a mailstore in synch than any locally scripted rsync setup. This is *EXACTLY* what I was looking for! The possibility of loosing an extremely small amount of data far outweighs the possibility of a multi-hour outage where 3,000 users are receiving can't reach the POP3 server errors. Besides, our incoming SMTP gateway boxes cache all incoming email for 24 hours, and we can re-deliver any message to the back-end we wish during that window. I really try my best to design/implement all the systems I can like our networks... multiple paths and extremely quick convergence. Being able to take a box down to test/perform an upgrade, or during a failure without client impact is well worth any initial large learning curve imho. Thanks, Steve Hello Steve: Another approach would be a cluster of Postfix servers and Dovecot servers behind PF load balancers. We have 3 POP servers (IMAP/POP), 9 Mail Servers, 2 Defer servers and 5 Filter servers that process over 20 million messages a day without a blip. We can take individual servers out of the pool for maintenance, etc. Everything is fed to a set of redundant NAS for the data storage and common configuration files. Regards, Mike -- Michael K. Smith - CISSP, GISP Chief Technical Officer - Adhost Internet LLC mksm...@adhost.com w: +1 (206) 404-9500 f: +1 (206) 404-9050 PGP: B49A DDF5 8611 27F3 08B9 84BB E61E 38C0 (Key ID: 0x9A96777D) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
New mail server setup
I'm looking potentially to try a different mail server setup. I'm requesting honest feedback from experienced mail ops. My minimum requirements: - IPv6 for all protocols - SPF - IMAP|POP3 must support SSL - SMTP AUTH - submit on 587 - MySQL backend for un/pw, vpopmail preferred, but not mandatory - Maildir storage preferred - easy (ie: well documented) integration with SA/clam - integration with maildrop .mailfiter preferred Right now I use a system wrapped around Qmail, and honestly, I just don't want to patch for IPv6 anymore. I've broken my personal system, so while I work on re-hacking everything, I thought I'd solicit some new ideas. I've been using the same email system pretty much across the board for seven years or so, so perhaps I should look at other options. Please cc me, as this addr isn't subscribed. I won't be receiving my list email from my backup mx until tomorrow, as it were ;) Steve smime.p7s Description: S/MIME Cryptographic Signature
