Re: OT - Scalable email server solution needed
robert [EMAIL PROTECTED] writes: Have a look at qmail, It is very scalable and well supported with various sites and mailing lists. Iv'e been using it for well over a year now. Most important thing, IMHO, is uptime. If you use the Maildir mailbox format you can put it on a solid NFS server like a NetApp and front it with any number of MTAs and IMAP servers. Maildir is NFS-safe. If a (used) netApp is too expensive for you, the same approach still keeps your complicated services off your most critical file server. I used qmail-ldap (qmail with LDAP for virtualization) and a handfull of 1U SMTP/IMAP/POP/Squirrelmail servers. Each server had a local read-only replica of the LDAP data sync'd from the LDAP master. Zero downtime in well over a year, even with taking individual boxes down (one at a time) for upgrades and such. You could also look at Vpopmail virtualization for qmail. I've not tried to scale that across multiple boxes so I don't know how you'd replicate the account info. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
Chris Shenton wrote: robert [EMAIL PROTECTED] writes: Have a look at qmail, It is very scalable and well supported with various sites and mailing lists. Iv'e been using it for well over a year now. Most important thing, IMHO, is uptime. If you use the Maildir mailbox format you can put it on a solid NFS server like a NetApp and front it with any number of MTAs and IMAP servers. Maildir is NFS-safe. If a (used) netApp is too expensive for you, the same approach still keeps your complicated services off your most critical file server. I used qmail-ldap (qmail with LDAP for virtualization) and a handfull of 1U SMTP/IMAP/POP/Squirrelmail servers. Each server had a local read-only replica of the LDAP data sync'd from the LDAP master. Zero downtime in well over a year, even with taking individual boxes down (one at a time) for upgrades and such. You could also look at Vpopmail virtualization for qmail. I've not tried to scale that across multiple boxes so I don't know how you'd replicate the account info. Vpopmail can use a SQL store for account info. Like you we have several 1U SMTP/IMAP/POP/Squirrelmail toasters running with all mail stored on a common NFS server. Which also is home to SQL for vpopmail, and for spamd. I have the NFS server (mail store is Raid5) on a second 1 gb network. Maildir delivery and SQL lookups are very very fast, even under a heavy load. 2+ years with the only issue being disk failure on the OS mirror of the toasters. I can recommend it. DAve -- This message was checked by forty monkeys and found to not contain any SPAM whatsoever. Your monkeys may vary ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
Duane Whitty wrote: Hi, My appologies in advance for the OT post. This probably more appropriately belongs on ISP@ but there doesn't seem to be much traffic there. To the point: I'm about to provide hosting services, including but not limited to email, to a few local business. Right now I have Sendmail configured and I was about to install Courier-IMAP. My concern is future scalability. I'm not sure sendmail is the right back-end MTA. Right now Sendmail and Procmail will work fine I believe with the addition of an IMAP server. But what about later on? I was looking at DBmail but I'm not sure just how much activity that project has so I'm a little leary, but perhaps that's unjustified? So now I'm considering Exim or POSFIX. The Sun Java Communications Server looks appealing but unless I misunderstood I don't have anywhere near the bankroll for that yet and it doesn't look like a good fit for a start-up operation anyhow. I like that POPular port and the way it uses proxies etc., but it doesn't seem to support IMAP. Is there anyone on the list who might be serving many business customers each with many users who would be willing to share their insights or opinions on this? From my past experiences, transitioning from a low volume solution to a high volume solution, regardless of the application domain, is usually best solved by making the right choice in the beginning. It would seem like an important marketing tool as well. If I am way of base here or missing something really important I'd appreciate the heads-up. Thanks in advance for any help. Sincerely, Duane Hi, I would like to say thank you to everyone who has replied to my query, both on and off this list. Your input is much appreciated. I've read everyones answers and I have found them very educational and helpful. I have learned two things as a result of the help people have offered. First, everyone has their favourite mail solution, and secondly, I am not in a position to decide which email system is best for me at this time because I have not yet educated with regards to the issues. I believe that before I move away from the base email system provided with FreeBSD, and the support and maintenance attention it receives as part of the base system, I should learn more about it. I also need to learn more about it before I agree to support an email solution for other people. For this reason I have decided to use an external provider, for the short term, to meet the needs of those who have approached me for hosting services. I am basically taking my own advice. In an earlier post on a different thread I advised that people need to learn the tools they are using before they can make conclusions about those tools shortcomings. If I can't make educated conclusions for myself regarding Sendmail's strengths and shortcomings then how can I reasonably expect to evaluate its replacement? I don't believe I can. So I must learn the details of SMTP, IMAP, Sendmail, mailfilters, and any other tools I will need to use. Then after using these tools for a while I will be in a better position to evaluate them. Right now I have a working installation of Sendmail that seems reasonably secure. It uses the DNS and after several tests it appears to be closed to relaying. I feel somewhat confident about my DNS: it uses internal and external views, it doesn't leak any internal information, it only does recursion for clients on my private net, it only allows zone transfers to my authorized secondary, and my secondary is geographically removed (somewhat). With this foundation behind me I have started to do some more reading. I have read, from start to finish, RFCs 4234 (ABNF), 2821 (SMTP), 2822 (Internet Message Format), and 2476 (Message Submission). I am currently reading the Bat Book and the Sendmail Cookbook. From the start I will be heeding the advice of several posters who recommended using maildir format. To implement that I will be using Procmail, at least to start, as soon as I learn a little more about Sendmail. I hope no one feels I have wasted their time and I would truly appreciate further comments that you believe might aid me on my path. Sincerely, Duane Whitty -- [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
Marc G. Fournier [EMAIL PROTECTED] wrote: On Sat, 8 Apr 2006, Bill Moran wrote: One advantage of Maildir over mbox and the Cyrus db (that, for some reason, I seldom see touted) is that you can make a safe backup of a Maildir without shutting down the SMTP, POP, or IMAP server. With both mbox and the Cyrus mail system, you have to shut the mail system down to back up the mail boxes. Why with Cyrus IMAPd? I just had a major hardware failure, had to restore 30 mail spools from backup, all of which were 'backed up live', and the only thing I had to do once restored was run the 'reconstruct' command to make sure the various databases were sync'd up ... and the reconstruct command can be run while the system is live too ... I have only a small amount of experience with Cyrus. However: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/Backup Based on that document, it appears as if you're dodging the bullet with backups. My interpretation is that Cyrus keeps mailboxes in some sort of db file. If a db file is being modified while you're backing it up, the backed up version will be inconsistent, thus the entire mailbox unusable. Compare this to Maildir, where each message is a seperate file. If you're backing up during access to a mailbox, a single message could end up corrupt, but this will not affect the rest of the mailbox. I set up backup for a Cyrus server for a client. We've yet to have to restore however, so all of my knowledge is based on that and other documents on the web regarding Cyrus' operation. It might be one of those things where _theoretically_ the backups could be useless, but it almost never happens in practice? -- Bill Moran ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
On Sun, 9 Apr 2006, Bill Moran wrote: I have only a small amount of experience with Cyrus. However: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/Backup Based on that document, it appears as if you're dodging the bullet with backups. My interpretation is that Cyrus keeps mailboxes in some sort of db file. If a db file is being modified while you're backing it up, the backed up version will be inconsistent, thus the entire mailbox unusable. Not quite. Cyrus maps the imap hierarchy onto directories, with one (written once only) file per message. As flat-file as you can get. On top of this are per-folder index files (which can be recreated using reconstruct) holding stuff like preformatted IMAP responses, header indexes etc. That's where cyrus gets its speed from. Additionally there's a per-server mailboxes DB holding folder information (including ACLs), per-user seen/subscription databases and a deliverdb for duplicate suppression. The latter can all be stored in multiple formats, including bdb; hot backups for these work much the same way as for anything other bdb. It's not perfect, but there's minimal bullet-dodging. jan -- jan grant, ISYS, University of Bristol. http://www.bris.ac.uk/ Tel +44 (0)117 3317661 http://ioctl.org/jan/ ioctl(2): probably the coolest Unix system call in the world ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
On Sun, 9 Apr 2006, Bill Moran wrote: Marc G. Fournier [EMAIL PROTECTED] wrote: On Sat, 8 Apr 2006, Bill Moran wrote: One advantage of Maildir over mbox and the Cyrus db (that, for some reason, I seldom see touted) is that you can make a safe backup of a Maildir without shutting down the SMTP, POP, or IMAP server. With both mbox and the Cyrus mail system, you have to shut the mail system down to back up the mail boxes. Why with Cyrus IMAPd? I just had a major hardware failure, had to restore 30 mail spools from backup, all of which were 'backed up live', and the only thing I had to do once restored was run the 'reconstruct' command to make sure the various databases were sync'd up ... and the reconstruct command can be run while the system is live too ... I have only a small amount of experience with Cyrus. However: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/Backup Based on that document, it appears as if you're dodging the bullet with backups. My interpretation is that Cyrus keeps mailboxes in some sort of db file. If a db file is being modified while you're backing it up, the backed up version will be inconsistent, thus the entire mailbox unusable. Agreed, but this db file is stored within the maibox itself, and which is what Cyrus has a 'reconstruct' command to rebuild ... so, after a restore, the first thing you run in 'reconstruct' to make sure you database(s) match the mailbox ... Compare this to Maildir, where each message is a seperate file. If you're backing up during access to a mailbox, a single message could end up corrupt, but this will not affect the rest of the mailbox. Oh, wait, I misunderstood your first paragraph ... Cyrus IMAP stores its individual mail messages in a seperate file (old 'mh' kinda thing), but there is a db file that maintains the 'state' informatin for the folder (what messages have been read, marked deleted but not purged, etc) ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
Marc G. Fournier [EMAIL PROTECTED] wrote: On Sun, 9 Apr 2006, Bill Moran wrote: [snip] I have only a small amount of experience with Cyrus. However: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/Backup Based on that document, it appears as if you're dodging the bullet with backups. My interpretation is that Cyrus keeps mailboxes in some sort of db file. If a db file is being modified while you're backing it up, the backed up version will be inconsistent, thus the entire mailbox unusable. Agreed, but this db file is stored within the maibox itself, and which is what Cyrus has a 'reconstruct' command to rebuild ... so, after a restore, the first thing you run in 'reconstruct' to make sure you database(s) match the mailbox ... Compare this to Maildir, where each message is a seperate file. If you're backing up during access to a mailbox, a single message could end up corrupt, but this will not affect the rest of the mailbox. Oh, wait, I misunderstood your first paragraph ... Cyrus IMAP stores its individual mail messages in a seperate file (old 'mh' kinda thing), but there is a db file that maintains the 'state' informatin for the folder (what messages have been read, marked deleted but not purged, etc) ... Hmm ... I apologize for the misinformation then. It seems like the link listed above could use some updating. -- Bill Moran Potential Technologies http://www.potentialtech.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
Postfix + Cyrus IMAPd ... Cyrus IMAPd has built-in virtual domain support ... only solution I've used for almost 10 years now ... On Sat, 8 Apr 2006, Duane Whitty wrote: Hi, My appologies in advance for the OT post. This probably more appropriately belongs on ISP@ but there doesn't seem to be much traffic there. To the point: I'm about to provide hosting services, including but not limited to email, to a few local business. Right now I have Sendmail configured and I was about to install Courier-IMAP. My concern is future scalability. I'm not sure sendmail is the right back-end MTA. Right now Sendmail and Procmail will work fine I believe with the addition of an IMAP server. But what about later on? I was looking at DBmail but I'm not sure just how much activity that project has so I'm a little leary, but perhaps that's unjustified? So now I'm considering Exim or POSFIX. The Sun Java Communications Server looks appealing but unless I misunderstood I don't have anywhere near the bankroll for that yet and it doesn't look like a good fit for a start-up operation anyhow. I like that POPular port and the way it uses proxies etc., but it doesn't seem to support IMAP. Is there anyone on the list who might be serving many business customers each with many users who would be willing to share their insights or opinions on this? From my past experiences, transitioning from a low volume solution to a high volume solution, regardless of the application domain, is usually best solved by making the right choice in the beginning. It would seem like an important marketing tool as well. If I am way of base here or missing something really important I'd appreciate the heads-up. Thanks in advance for any help. Sincerely, Duane ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
On Sat, 2006-04-08 at 00:45 -0300, Duane Whitty wrote: Hi, My appologies in advance for the OT post. This probably more appropriately belongs on ISP@ but there doesn't seem to be much traffic there. To the point: I'm about to provide hosting services, including but not limited to email, to a few local business. Right now I have Sendmail configured and I was about to install Courier-IMAP. My concern is future scalability. I'm not sure sendmail is the right back-end MTA. Right now Sendmail and Procmail will work fine I believe with the addition of an IMAP server. But what about later on? I was looking at DBmail but I'm not sure just how much activity that project has so I'm a little leary, but perhaps that's unjustified? So now I'm considering Exim or POSFIX. The Sun Java Communications Server looks appealing but unless I misunderstood I don't have anywhere near the bankroll for that yet and it doesn't look like a good fit for a start-up operation anyhow. I like that POPular port and the way it uses proxies etc., but it doesn't seem to support IMAP. Is there anyone on the list who might be serving many business customers each with many users who would be willing to share their insights or opinions on this? From my past experiences, transitioning from a low volume solution to a high volume solution, regardless of the application domain, is usually best solved by making the right choice in the beginning. It would seem like an important marketing tool as well. If I am way of base here or missing something really important I'd appreciate the heads-up. Thanks in advance for any help. Sincerely, Duane Duane, Have a look at qmail, It is very scalable and well supported with various sites and mailing lists. Iv'e been using it for well over a year now. Sites: http://www.lifewithqmail.org/ http://www.qmailrocks.org/ http://qmail.jms1.net/ Etc Rob ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
Duane Whitty [EMAIL PROTECTED] wrote: Hi, My appologies in advance for the OT post. This probably more appropriately belongs on ISP@ but there doesn't seem to be much traffic there. To the point: I'm about to provide hosting services, including but not limited to email, to a few local business. Right now I have Sendmail configured and I was about to install Courier-IMAP. My concern is future scalability. I'm not sure sendmail is the right back-end MTA. Right now Sendmail and Procmail will work fine I believe with the addition of an IMAP server. But what about later on? I was looking at DBmail but I'm not sure just how much activity that project has so I'm a little leary, but perhaps that's unjustified? So now I'm considering Exim or POSFIX. The Sun Java Communications Server looks appealing but unless I misunderstood I don't have anywhere near the bankroll for that yet and it doesn't look like a good fit for a start-up operation anyhow. I like that POPular port and the way it uses proxies etc., but it doesn't seem to support IMAP. Is there anyone on the list who might be serving many business customers each with many users who would be willing to share their insights or opinions on this? From my past experiences, transitioning from a low volume solution to a high volume solution, regardless of the application domain, is usually best solved by making the right choice in the beginning. It would seem like an important marketing tool as well. There are a number of excellent tools available, I will give a few opinions on what I like best. I think the most important key, however, is that your data be in a format that's flexible enough to scale, while also giving you the ability to choose tools. This being said, standardize on Maildir as your backend storage system right from the start. Maildir has a number of advantages over mbox. A few quick links: http://www.courier-mta.org/mbox-vs-maildir http://cr.yp.to/proto/maildir.html One advantage of Maildir over mbox and the Cyrus db (that, for some reason, I seldom see touted) is that you can make a safe backup of a Maildir without shutting down the SMTP, POP, or IMAP server. With both mbox and the Cyrus mail system, you have to shut the mail system down to back up the mail boxes. You may or may not have a need to make mail box backups at this time, but it'll be nice to be able to do it if a customer is willing to pay for it! With both mbox and Cyrus, if you back up without stopping the server, and entire mailbox will be corrupted if the file holding it's mail is in use during backup and restoring will be difficult or impossible. With Maildir, only the one email that was in use will be corrupted on the backup. Once you've chosen to use Maildir, you can choose which softwares you want to use to get mail into and out of your Maildirs. Almost all Unix mail software supports both mbox and Maildir, so you have a LOT of choices. Additionaly, if you find that you picked a poor IMAP server for your needs, you can easily switch later without needing to convert all your data. You can even run multiple different IMAP and SMTP systems to try them out - on the same Maildirs. I recommend you set up an SQL database to keep your users in. This will result in a bit more work up front, but it scales much better. You'll be able to use tools like PHP to create easy to use scripts to manage your email users, and you don't have to spam your /etc/password file with hundreds (or thousands) of users that shouldn't be allowed to log in anyway. In my experience, this gets forgotten, and user accidentally end up with a login shell - since the admins aren't securing the mail server against local attacks, these accidental login shells become a security risk. Now, once you've decided to use Maildirs with an SQL user database, it's time to pick software. I recommend PostgreSQL for the DB. It has all the performance of MySQL, but it also has a lot of advanced features that MySQL doesn't that will allow you to scale up, such as mature transaction support, point in time recovery of databases and mature stored procedures. For the SMTP system I recommend Postfix. It has all the security advantages of qmail, with an easier to understand configuration and a lot more commonly used features built in. It supports Maildir and PostgreSQL databases. For the POP/IMAP system, I recommend Dovecot. I've been using it since it was beta and it just works. The config is fairly straight forward. It supports Maildir and PostgreSQL databases, and is written with a careful focus on security and performance. In my experience it works well with all MUAs, even those from Microsoft. I've never found it lacking for features, and I've noticed that my IMAP server is cleaner to use than a lot of others I've seen. As I said before, however: if you go with a good backend storage format (Maildir) you'll find that if you don't like Postfix or Dovecot,
Re: OT - Scalable email server solution needed
Duane Whitty writes: My appologies in advance for the OT post. This probably more appropriately belongs on ISP@ but there doesn't seem to be much traffic there. Although there isn't much traffic, people to read it, and so far the advice I have gotten from that list is pretty good. business. Right now I have Sendmail configured and I was about to install Courier-IMAP. My concern is future scalability. What volume are you looking to handle? How many users? I'm not sure sendmail is the right back-end MTA. I am more of a Postfix kind of person, but can't imagine that you would have a performance issue because of using sendmail. However you need to give details such as how many machines you plan to use, what configuration in the machines, how many users, how many domains.. and anything you can specify. I'm considering Exim or POSFIX. I personally find Postfix easy to use, learn and very stable. Is there anyone on the list who might be serving many business customers each with many users who would be willing to share their insights or opinions on this? Until you give more info about your volume it's hard to give you any good information. I find Postfix to be very scalable, easy to use and maintain. I find Courier to be easy to use and maintain, but not necessary easy to scale.. HOWEVER, I am talking about THOUSANDS of email accounts.. so unless you reply saying that you will have large volumes.. in the thousands, performance may be a non issue. The hardware you use will also have a significant impact.. the architecture (ie single machine vs multiple machines splitting functions such as MX, outboud mail, webmail, mail storage.. ) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
Bill Moran writes: One advantage of Maildir over mbox and the Cyrus db (that, for some reason, I seldom see touted) is that you can make a safe backup of a Maildir without shutting down the SMTP, POP, or IMAP server. For a small/simple setup I think Maildir is most definitely the way to go. From what I have seen so far working with Cyrus.. and from what I read... Cyrus scales far beyond what can be easily done with most Maildir format. With both mbox and the Cyrus mail system, you have to shut the mail system down to back up the mail boxes. Although that is true, using a database such as in Cyrus can in theory be a big speed booster. it! With both mbox and Cyrus, if you back up without stopping the server, and entire mailbox will be corrupted if the file holding it's mail is in use during backup and restoring will be difficult or impossible. Maildir can also get corrupted. :-( At least with Courier.. I have seen several folders go bad and Courier did not have enough functionality to easily find which folders had problems. Once you've chosen to use Maildir, you can choose which softwares you want to use to get mail into and out of your Maildirs. Agree 100% that this is one great appeal of using Maildir. The ability to easily switch different alternatives. I recommend PostgreSQL for the DB. Until I started to work for an email provider I had never used Mysql, having used PostgreSQL for many, many years.. I must say that after using Mysql... I became to appreciate even more PostgreSQL. Coming from database administrator background I felt completely at home with PostgreSQL. For the SMTP system I recommend Postfix. I find postfix to be easy to use, easy to learn.. and highly stable and scalable. Great mailing list too. For the POP/IMAP system, I recommend Dovecot. I've been using it since it was beta and it just works. Does it scale better than Courier? In particular I find Courier's footprint is about 3 to 5 MB per connection. A bit on the high side when one has hundreds of connections per machine. over NFS, you can even run multiple computers all off the same backend. I can attest to that. :-) PostgreSQL is a little more of a commitment, but it seems as if support for PostgreSQL is growing It's a good choice. :-) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: OT - Scalable email server solution needed
On Sat, 8 Apr 2006, Bill Moran wrote: One advantage of Maildir over mbox and the Cyrus db (that, for some reason, I seldom see touted) is that you can make a safe backup of a Maildir without shutting down the SMTP, POP, or IMAP server. With both mbox and the Cyrus mail system, you have to shut the mail system down to back up the mail boxes. Why with Cyrus IMAPd? I just had a major hardware failure, had to restore 30 mail spools from backup, all of which were 'backed up live', and the only thing I had to do once restored was run the 'reconstruct' command to make sure the various databases were sync'd up ... and the reconstruct command can be run while the system is live too ... Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
