Re: [Dovecot] Migration questions...
pod wrote: Richard Hobbs richard.ho...@crl.toshiba.co.uk writes: 19. Once everything is working perfectly, send an email to the entire company instructing them what to do after the outage and arrange an outage and do the following steps as soon as the outage begins: a. Unplug DMZ switch from firewall to make delivered mail wait at the sender. [...] i. Plug DMZ switch back into firewall and begin monitoring exim logs to check that mail is being delivered. If I'm not misunderstanding the steps between 19.a -- 19.i are going to be done while not network connected? I'd be slightly concerned that these steps may involve anything some that needs to do DNS lookups or the like at which point they may hit long(ish) timeouts or just fail completely. Because the mail servers are in a DMZ, they have their own DNS running locally. Local DNS lookups, therefore, shouldn't be a problem. Good point though, so thank you for that - i ought to adjust my instructions to include installation of the DNS service! lol Thanks again, Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
Hello again, people, I have been reading all of the emails sent back and forth recently, and reading many web pages, and I have a new draft of my migration plan. If anyone has the time and generosity to read through this latest plan for me and let me know if you spot any potential problems with it, i'd really appreciate it. FYI, we're moving from a Debian 3.1/exim/uw-imapd/qpopper/mbox setup to a Debian 5.0/exim/dovecot/maildir setup, and the final version of this document can hopefully become a useful guide for lots of people in future too, if I get it right :-) Anyway... here's the plan... 1. Install Debian with exim, spamassassin, mysqld (for Horde/IMP) and mailman. Use ReiserFS for the indexes and data. XFS would be second choice for filesystem, and this is something of a religious argument, but Dovecot's own page states that ReiserFS is a better choice, and loads of pages on the Internet say that one or the other is better - there's no clear winner, really, so ReiserFS (the FS we already know and trust) wins the battle. 2. Run an online update. 3. Setup IPMI and monitor it from Nagios. 4. Install HotSaNIC to monitor historical system stats. 5. Install and configure Dell OpenManage Server Administrator (or whatever it's called) to monitor hardware, and make sure we will be emailed if a hard drive fails. 6. Rsync homedirs and inboxes onto new server, ready for initial exim configuration. 7. Configure exim as per existing mail server and test that mailing lists and normal email works. You should now have the existing mail delivery solution on the brand new hardware. 8. Configure Spamassassin as per current configuration and test. 9. Configure mailman as per current configuration and test. 10. Once mail delivery is sorted, add deb http://www.backports.org/debian lenny-backports main contrib non-free into /etc/apt/sources.list and run aptitude update aptitude install debian-backports-keyring aptitude update. 11. Install dovecot (at the time of writing, this was version 1.1.13-2). 12. Run the conversion utility (use mb2md.py from http://wiki.dovecot.org/Migration/MailFormat because convert-tool doesn't preserve msg UIDs) to convert all existing mbox mailboxes into maildir mailboxes, for use with Dovecot. 13. Setup exim to use Dovecot's delivery agent (deliver) so the indexes are updated on mail delivery instead of when the mailbox is requested and also to use the new maildir format. 14. Setup dovecot for IMAP and POP3 and test. *THIS IS WHERE THE REPEATABLE SECTION BEGINS* 15. Stop exim and dovecot services. 16. Rsync homedirs and inboxes onto new server again. 17. Run the conversion utility (use mb2md.py from http://wiki.dovecot.org/Migration/MailFormat because convert-tool doesn't preserve msg UIDs) to convert all existing mbox mailboxes into maildir mailboxes. 18. Start exim and dovecot services and test SMTP, Spamassassin, mailman, IMAP and POP3. *THIS IS WHERE THE REPEATABLE SECTION ENDS* 19. Once everything is working perfectly, send an email to the entire company instructing them what to do after the outage and arrange an outage and do the following steps as soon as the outage begins: a. Unplug DMZ switch from firewall to make delivered mail wait at the sender. b. Stop exim, IMAP, POP3 and dovecot services on both mail servers. c. Rsync homedirs and inboxes onto new server again. d. Run the conversion utility (use mb2md.py from http://wiki.dovecot.org/Migration/MailFormat because convert-tool doesn't preserve msg UIDs) to convert all existing mbox mailboxes into maildir mailboxes. e. Start exim and dovecot services and test SMTP, Spamassassin, mailman, IMAP and POP3. f. Reconfigure my own mail client to point to the new mail server (192.168.2.3). g. Reconfigure firewall to think the mail server is now on 192.168.2.3 instead of 192.168.2.2. h. Prepare an email in webmail somewhere in the world, ready to send to a user on our mail server. i. Plug DMZ switch back into firewall and begin monitoring exim logs to check that mail is being delivered. j. Send the email from webmail and check that is goes through exim perfectly and is sitting in the users Inbox. NOTE: We now have a working mail delivery solution! k. Start dovecot IMAP and POP3 and test both. l. Once exim and dovecot are both working against live mailboxes, tell everyone the outage is over. 20. Install and configure Horde IMP to access email via IMAP. 21. Reconfigure backup script to make sure new server is backed up. 22. Modify Nagios config to monitor new server. 23. DONE! Thanks in advance, everyone! :-) Richard. Brandon Lamb wrote: On Fri, May 15, 2009 at 5:02 AM, Richard Hobbs richard.ho...@crl.toshiba.co.uk wrote: Seth Mattinen wrote: Richard Hobbs wrote: Seth Mattinen wrote: Phillip Macey wrote: On 14/05/2009 5:11 PM, Steffen Kaiser wrote: On Wed,
Re: [Dovecot] Migration questions...
Richard Hobbs richard.ho...@crl.toshiba.co.uk writes: 19. Once everything is working perfectly, send an email to the entire company instructing them what to do after the outage and arrange an outage and do the following steps as soon as the outage begins: a. Unplug DMZ switch from firewall to make delivered mail wait at the sender. [...] i. Plug DMZ switch back into firewall and begin monitoring exim logs to check that mail is being delivered. If I'm not misunderstanding the steps between 19.a -- 19.i are going to be done while not network connected? I'd be slightly concerned that these steps may involve anything some that needs to do DNS lookups or the like at which point they may hit long(ish) timeouts or just fail completely.
Re: [Dovecot] Migration questions...
Curtis Maloney wrote: Phillip Macey wrote: Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) At first glance this sounds like a large folder is being indexed... are you using Dovecot deliver (which updates indices on deliver)? This raises an interesting question for me actually... given that we've now decided dovecot and maildir is the way forward for us, which delivery method should we use in exim? exim can support maildir, (right?) and so can dovecot, so should i use dovecot's deliver mechanism, or exim's own internal mechanism? Thanks again! Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
Seth Mattinen wrote: Phillip Macey wrote: On 14/05/2009 5:11 PM, Steffen Kaiser wrote: On Wed, 13 May 2009, Richard Hobbs wrote: The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Are you using ext3 by chance? Vanilla ext3 without directory indexing (or whatever it's called) *hates* directories with a lot of files - like maildir. Personally, I use XFS, which doesn't suffer from this problem since it uses b-trees instead of a table(!) like ext3 does. This raises another question for me actually... We will have one volume for indexes and another volume for data (using maildir). We will be using the latest stable Debian Linux distro. Any opinions on the best filesystem to use? We would normally go ReiserFS for large volumes, and ext3 for small volumes because of the unlimited inodes in reiserfs, but i understand that support for that is beginning to disappear given that Hans Reiser got into a bit of trouble a few years ago. Anyway... that would leave ext3, but is there a better choice i could make in this instance? We do want performance, of course, but also complete reliability and resilience when it comes to system crashes etc... we do *not* want data corruption, and ext3 we know has a very good journalling and data recovery methods. Well... they're very mature, anyway. Any opinions? Thanks again! Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
Richard Hobbs wrote: Seth Mattinen wrote: Phillip Macey wrote: On 14/05/2009 5:11 PM, Steffen Kaiser wrote: On Wed, 13 May 2009, Richard Hobbs wrote: The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Are you using ext3 by chance? Vanilla ext3 without directory indexing (or whatever it's called) *hates* directories with a lot of files - like maildir. Personally, I use XFS, which doesn't suffer from this problem since it uses b-trees instead of a table(!) like ext3 does. This raises another question for me actually... We will have one volume for indexes and another volume for data (using maildir). We will be using the latest stable Debian Linux distro. Any opinions on the best filesystem to use? We would normally go ReiserFS for large volumes, and ext3 for small volumes because of the unlimited inodes in reiserfs, but i understand that support for that is beginning to disappear given that Hans Reiser got into a bit of trouble a few years ago. Anyway... that would leave ext3, but is there a better choice i could make in this instance? We do want performance, of course, but also complete reliability and resilience when it comes to system crashes etc... we do *not* want data corruption, and ext3 we know has a very good journalling and data recovery methods. Well... they're very mature, anyway. I used to use ext3, ran into its horrible performance even with directory indexing enabled, went to XFS and never looked back. All of my systems are Debian stable. Reiser3 is part of the kernel so I wouldn't worry about that; Namesys considered it complete and stopped work on it long before the whole murder thing. Both Reiser3 and XFS have worse reputations than ext3, however, I've seen ext3 filesystems hosed beyond repair, too. All my XFS filesystems have battery-backed cache controllers, so it hasn't happened to me yet, hopefully never. ;) One catch with XFS (such as with LVM) to keep in mind is you can't ever shrink it, only grow. ext3 is mature but IMHO completely unsuitable for a busy mail server or any situation where you'll have a bajillion of files in one directory. The exact point at which ext3 will screw you over obviously depends on many factors. But when it happens it'll probably be painful to reformat to something better. ~Seth
Re: [Dovecot] Migration questions...
On F 15 May, 2009, at 09:39 , Seth Mattinen wrote: This raises an interesting question for me actually... given that we've now decided dovecot and maildir is the way forward for us, which delivery method should we use in exim? exim can support maildir, (right?) and so can dovecot, so should i use dovecot's deliver mechanism, or exim's own internal mechanism? Only dovecot 'deliver' will update the index on delivery. well, using an LDA makes a little more cumbersome to check the local recipient at RCPT time. What repercussion has not updating the index on delivery? My systems use exim to deliver and they seem to act normally... Giuliano
Re: [Dovecot] Migration questions...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, 15 May 2009, Giuliano Gavazzi wrote: well, using an LDA makes a little more cumbersome to check the local recipient at RCPT time. Huh? exim won't try local deliver unless it has decided it is a local recipient. You won't get overquota status this way, did you mean that? Bye, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBSg1EP3WSIuGy1ktrAQJtVAgAqH3G9syzKPtmLJ1axrGtAAKb6NS5knma 8081aI9LXsOHU/MiPJ0KCzRER80gJAbNhyYlFNaHU2nc9ryOPFO3+HnJ/tkpSqkG 3P8m7PCYa1+pupknU+ZH9YarcA7qxFT/wPHKCvhnlXEqQttvImn0ROL4EUN1waqP QmFivY26FJuhrn4BFm8t/1Unl5AJ5kknajZ7Q/WuNcwhxzewfNF4+CG29VOmJzs9 V87kwc4mpUEWwnkG7YkL4RJWhnBE9dnoZBYaCMR3GyqiL2vS1qGHJwYY6WC0tObQ bym38B7+OV9cOjiU0RYj8ZB4wa9UiGvNEokCzvS1uSjhx9bKl7JyGA== =CSiD -END PGP SIGNATURE-
Re: [Dovecot] Migration questions...
Seth Mattinen wrote: Richard Hobbs wrote: Curtis Maloney wrote: Phillip Macey wrote: Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) At first glance this sounds like a large folder is being indexed... are you using Dovecot deliver (which updates indices on deliver)? This raises an interesting question for me actually... given that we've now decided dovecot and maildir is the way forward for us, which delivery method should we use in exim? exim can support maildir, (right?) and so can dovecot, so should i use dovecot's deliver mechanism, or exim's own internal mechanism? Only dovecot 'deliver' will update the index on delivery. Do does this mean that it's slightly slower to actually deliver the mail with dovecot (because it's writing two places instead of one), but it saves the files having to be indexed again, so overall potentially faster? And one more question... given that we're going to be using maildir, should i still use dovecot's POP3 server, or whatever the standard one is? I've heard (through google mainly), that dovecot's POP3 server, in terms of performance, is actually quite bad compared to other ones, but what's your take on this? Thanks again, people! Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, May 15, 2009 at 11:28:42AM +, Richard Hobbs wrote: [...] Only dovecot 'deliver' will update the index on delivery. Do does this mean that it's slightly slower to actually deliver the mail with dovecot (because it's writing two places instead of one), but it saves the files having to be indexed again, so overall potentially faster? Things get re-indexed on client request if Dovecot sees that index is stale. So you are buying faster response times for clients with somewhat higher server load at mail delivery. I don't know how this piecemeal update of the index stacks up against a complete re-index, but I'd assume it to be more efficient (only having to do new mails instead of whole mailbox). Still, I find the re-index to be almost instantaneous on not-so-small mailboxes (hundreds of MB) and fairly modest hardware, by today's standards. Regards - -- tomás -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFKDVeEBcgs9XrR2kYRAtBGAJ4zHYA23C5SxJRcS5khH5cskZGfSACdFs3r 7tmWdF5ShGkTOiqVXWQIYjs= =swId -END PGP SIGNATURE-
Re: [Dovecot] Migration questions...
Seth Mattinen wrote: Richard Hobbs wrote: Seth Mattinen wrote: Phillip Macey wrote: On 14/05/2009 5:11 PM, Steffen Kaiser wrote: On Wed, 13 May 2009, Richard Hobbs wrote: The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Are you using ext3 by chance? Vanilla ext3 without directory indexing (or whatever it's called) *hates* directories with a lot of files - like maildir. Personally, I use XFS, which doesn't suffer from this problem since it uses b-trees instead of a table(!) like ext3 does. This raises another question for me actually... We will have one volume for indexes and another volume for data (using maildir). We will be using the latest stable Debian Linux distro. Any opinions on the best filesystem to use? We would normally go ReiserFS for large volumes, and ext3 for small volumes because of the unlimited inodes in reiserfs, but i understand that support for that is beginning to disappear given that Hans Reiser got into a bit of trouble a few years ago. Anyway... that would leave ext3, but is there a better choice i could make in this instance? We do want performance, of course, but also complete reliability and resilience when it comes to system crashes etc... we do *not* want data corruption, and ext3 we know has a very good journalling and data recovery methods. Well... they're very mature, anyway. I used to use ext3, ran into its horrible performance even with directory indexing enabled, went to XFS and never looked back. All of my systems are Debian stable. Reiser3 is part of the kernel so I wouldn't worry about that; Namesys considered it complete and stopped work on it long before the whole murder thing. Both Reiser3 and XFS have worse reputations than ext3, however, I've seen ext3 filesystems hosed beyond repair, too. All my XFS filesystems have battery-backed cache controllers, so it hasn't happened to me yet, hopefully never. ;) One catch with XFS (such as with LVM) to keep in mind is you can't ever shrink it, only grow. Trouble is... i've been googling this as well, just now, and loads of people say XFS has the better performance, but loads of other people say ReiserFS has the better performance. We have battery backed up RAID controllers too, in this new system, and the systems are UPSd, so on that basis i'm still none the wiser! lol I appreciate your experience with XFS is a positive one, but even the dovecot web site says XFS might now be a good choice... http://wiki.dovecot.org/MailboxFormat/Maildir What a tough decision! I know it probably won't make too much difference in my situation, but i want this to be a very long-term solution, so want to do it right first time! Any other opinions on XFS vs Reiserfs with Dovecot maildir? Thanks again! Richard. ext3 is mature but IMHO completely unsuitable for a busy mail server or any situation where you'll have a bajillion of files in one directory. The exact point at which ext3 will screw you over obviously depends on many factors. But when it happens it'll probably be painful to reformat to something better. ~Seth __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
Seth Mattinen, 2009-05-15 09:39: (right?) and so can dovecot, so should i use dovecot's deliver mechanism, or exim's own internal mechanism? Only dovecot 'deliver' will update the index on delivery. That's obvious. But dovecot (AFAIK) updates (not rebuilds) indexes when it sees new messages, so it should not matter that much. At least with maildir and not too much messages in poll intervalls, so usual always on imap usage should be ok. It might be even better not to use dovecot-lda for heavy-usage systems, because you save the fork/exec and indexes are updated on demand (i.e. when the folder is polled), not for every single message inserted. With exim, I had also the problem that messages got bounced because the lda crashed occasionally (was a svn version, so I don't complain) and stupid exim treated this as a permanent error.
Re: [Dovecot] Migration questions...
On F 15 May, 2009, at 12:30 , Steffen Kaiser wrote: On Fri, 15 May 2009, Giuliano Gavazzi wrote: well, using an LDA makes a little more cumbersome to check the local recipient at RCPT time. Huh? exim won't try local deliver unless it has decided it is a local recipient. You won't get overquota status this way, did you mean that? No. The problem is the try... or at least, it was when I had a cyrus + exim setup five years ago. With cyrus it was impossible to easily check if the recipient was an existing local user. It required either the use of the mbpath utility or a recipient callout with lmtp over tcp. I suppose that with a virtual users setup this is a moot point, but with system users nothing beats the simplicity of an all exim setup! (If you use exim, that is). Giuliano
Re: [Dovecot] Migration questions...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 13 May 2009, Richard Hobbs wrote: difference for you. A single server with recent hardware should be able to easily handle hundreds of simultaneous users. That's also good to know... i like to do a job right instead of relying on faster hardware, as i'm sure you all do too, but it's good to know that if i make one or two non-optimal choices along the way, it'll probably be lightning fast anyway! Well, IMHO there is some advantage to split index data and mail data on separate disks, if they are located at different channels, so the load per channel drops. But this is probably not so significant, if your OS uses one CPU for all I/O anyway The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: a) hits a 2GB limit on a mailbox in the mail client, b) try to move all their 20'000+ messages at once to an archive folder once a year, when their quota limit is exceeded. This can make the whole server irresponsible slow, because I have the mail_log plugin running as well. With maildir, and especially dovecot, this problem effectively disappears! That's the theory anyway, right? Bye, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBSgvEHnWSIuGy1ktrAQLsswgAvW1yUeDUVCvivf/67C47aJF4Vahuovzu kMBcUlrB9K2kxxz4yP2F+UVSnsQAVyo5I7XztFQ81mCZdNKStFO8k4booI75zpW/ Qwm6d9jezReMMzL/D73NKNwi1hQwK+CDsSIuuIFRfFuHkH7q02BAT2SF66dTeqM3 e08g66b0Ki2qlSWfE4smAjZFrVZCPgDlYgSRYrUK0VixV6ljyZLU+KFrl97RkHOY qMvxM6WfQ7TsejcU5UrFGL/J0Wyh5TOzO6iYSAXhsWM6AEthfDvz1C5rpcBmk1V4 BYQYe4cnG7b7slb0SsMed6SVuyzVfKnoZo+Wm5vEy27DGy8JfVljuA== =wzNa -END PGP SIGNATURE-
Re: [Dovecot] Migration questions...
Richard Hobbs wrote: That's also good to know... i like to do a job right instead of relying on faster hardware, as i'm sure you all do too, but it's good to know that if i make one or two non-optimal choices along the way, it'll probably be lightning fast anyway! The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Most of my mailing list folders that I work with on a daily basis like NANOG (over 20,000) and cisco-nsp (over 35,000) are no slower than folders with a handful of messages in them. It's not a private server either, it's the same one my customers use. If you're seeing a slowdown at 5k messages, either your server is woefully underpowered or something isn't quite right. ~Seth
Re: [Dovecot] Migration questions...
On 5/14/2009, Steffen Kaiser (skdove...@smail.inf.fh-brs.de) wrote: b) try to move all their 20'000+ messages at once to an archive folder once a year, when their quota limit is exceeded. This can make the whole server irresponsible slow, because I have the mail_log plugin running as well. Hmmm... it would be nice if there was some way to check for mass moves like that and just log one line like: imap(user): copy - Archive: uid=908, IDIOT_USER moved 20K+ messages at once - FLOG HIM!! Don't know if its possible to detect a mass move though... -- Best regards, Charles
Re: [Dovecot] Migration questions...
I'd point out that the big *practical* issue with mbox is the reality of big inboxes. While you can restrict the hoi polloi to something limited like a quota of under 60MB (and remember that inbox is one big honking file), the powers that be will not allow themselves to be so limited...nor will they be particularly good about cleaning up. I dunno how it is with you and your hardware/OS implementation, but there is a serious CPU hit when somebody with a 1GB inbox (one big file, remember) deletes a message...or gets new mail...or searches their inbox (I call this the python swallowing the pig). The first two will be trivial when we switch to maildir. OTOH, boy is it quick to do a backup with mbox. I dread that part of our move from mbox to maildir format. We will probably go from 2 hours to a day in the switch from 3000 inboxes of one file each (mbox) to 3000 directories with hundreds or thousands of files in each (maildir). Pick your poison Timo Sirainen wrote: On May 13, 2009, at 9:57 AM, Richard Hobbs wrote: OK... so Dovecot is certainly significantly faster that uw-imapd in both cases, but is dovecot fastest with mbox or maildir? I would assume maildir, but you never know... It's not that simple to answer. With mbox it's probably faster to read through all mails, because they're in a single file. With Maildir it's faster to delete mails, because it only needs to delete a single file, instead of moving data around in the mbox file. But Maildir has less problems and it's much less likely to get corrupted, so even if mbox performance would be better in some cases I'd recommend Maildir. -- One must think like a hero to behave like a merely decent human being. - May Sarton Stewart Dean, Unix System Admin, Bard College, New York 12504 sd...@bard.edu voice: 845-758-7475, fax: 845-758-7035
Re: [Dovecot] Migration questions...
Phillip Macey wrote: Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) At first glance this sounds like a large folder is being indexed... are you using Dovecot deliver (which updates indices on deliver)? With maildir, and especially dovecot, this problem effectively disappears! That's the theory anyway, right? Theory is wonderful when it works :-) In theory, Theory and Practice are the same. In practice, this isn't always the case. :) -- Curtis Maloney cmalo...@cardgate.net
Re: [Dovecot] Migration questions...
Phillip Macey wrote: On 14/05/2009 5:11 PM, Steffen Kaiser wrote: On Wed, 13 May 2009, Richard Hobbs wrote: The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Are you using ext3 by chance? Vanilla ext3 without directory indexing (or whatever it's called) *hates* directories with a lot of files - like maildir. Personally, I use XFS, which doesn't suffer from this problem since it uses b-trees instead of a table(!) like ext3 does. ~Seth
Re: [Dovecot] Migration questions...
On Fri, 2009-05-15 at 09:35 +1000, Phillip Macey wrote: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Do you have POP3 users? What clients do your users typically use? BTW. Kind of nasty to hijack a pretty much unrelated thread to your question.. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Migration questions...
On 15/05/2009 9:49 AM, Timo Sirainen wrote: On Fri, 2009-05-15 at 09:35 +1000, Phillip Macey wrote: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Do you have POP3 users? What clients do your users typically use? 50% pop. 50% imap.. Roughly. BTW. Kind of nasty to hijack a pretty much unrelated thread to your question.. Apologies, I wasn't intending it to be a hijack. Steffen said he had no problems. I was only making trying to point out that maildir can also have its problems as well. I dont know how MBox would handle the same situation because we dont have any MBox storage. I may post another thread if we still have problems after I have finished tidying up some peoples maildirs. -- Thanks, Phill Macey begin:vcard fn:Phillip Macey n:Macey;Phillip org:Canon Information Systems Research Australia (CISRA);IT Services adr:;;3 Thomas Holt Drive;North Ryde;;2113;Australia email;internet:phillip.ma...@cisra.canon.com.au title:Systems Administrator tel;work:(02) 9805 2645 x-mozilla-html:FALSE version:2.1 end:vcard
Re: [Dovecot] Migration questions...
On May 14, 2009, at 8:19 PM, Phillip Macey wrote: On 15/05/2009 9:49 AM, Timo Sirainen wrote: On Fri, 2009-05-15 at 09:35 +1000, Phillip Macey wrote: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Do you have POP3 users? What clients do your users typically use? 50% pop. 50% imap.. Roughly. With POP3 users those kinds of I/O bursts can happen when a lot of messages don't have their virtual size known. See Maildir performance in http://wiki.dovecot.org/POP3Server
Re: [Dovecot] Migration questions...
On 15/05/2009 10:33 AM, Timo Sirainen wrote: On May 14, 2009, at 8:19 PM, Phillip Macey wrote: On 15/05/2009 9:49 AM, Timo Sirainen wrote: Do you have POP3 users? What clients do your users typically use? 50% pop. 50% imap.. Roughly. With POP3 users those kinds of I/O bursts can happen when a lot of messages don't have their virtual size known. See Maildir performance in http://wiki.dovecot.org/POP3Server Ok. Thanks. I will have a read and keep that in mind ;-)
Re: [Dovecot] Migration questions...
Hi All, Replies to everyone below (to keep the number of emails down)... Ed W wrote: Richard Hobbs wrote: Hi All, We are soon to migrate our mail server from one piece of hardware to another and we would like to take this opportunity to optimize things. Can I recommend you add virtualisation to your todo list. I use linux-vserver, but there are plenty other ideas out there. It's just superb that later on you can migrate services between physical hardware with MUCH less hassle. You can easily test upgrades in a sandbox first. etc I personally split all the tasks into different virtual servers. Right now I actually still have quite a few mail related services in a single vserver, but ideally you would split everything up and then later if you needed to upgrade a single service or move it to a new machine it would have minimal effect. Your general process sounds about right though - I think it may be possible for you to preserve pop uidls, but see the wiki for more notes on that I appreciate that the world is going crazy for virtualization at the moment and I agree that it is a truly great concept. However, in this instance, where we want absolute ultimate performance and we know for a fact that this machine will never be used for anything else, we are shying away from virtualization. In some way, it's another layer to potentially go wrong, and another layer to potentially slow things down. I appreciate the advice though, it's just that i'm not sure it's for us in this particular instance. Regarding keeping msg UIDs, i know that can happen now, which is great :-) Lou Duchez wrote: I don't know whether this would help with the migration, but I routinely solve a similar problem. I have implemented mail failover between two servers -- which are configured with identical sets of mailboxes -- and every 10 minutes or so, a script grabs any E-Mails from the other server and stores them locally in the proper mailboxes. This script relies on IMAP connectivity and passwords in plaintext in a passwd-file. Here it is: snip Blast, forgot something: the Simple.pm referenced in the script is this thing: http://search.cpan.org/~jpaf/Net-IMAP-Simple-0.93/Simple.pm Download it, compile it, put it somewhere that the script can find it. Thank you very much for that script - i will certainly take a good look through that when the time comes. Does it rely on IMAP connectivity at both ends? i.e. does it read from old IMAP and write to new IMAP, or does it deal with the filesystem at one end or the other? Timo Sirainen wrote: On May 12, 2009, at 6:41 AM, Richard Hobbs wrote: Single-dbox is the highest performing, but note that it's not as much tested as mbox and Maildir code. I think it should work ok, but I'm not aware of any larger installations using dbox currently. So in case you find a problem, you might have to upgrade/patch Dovecot to get it fixed and that would require compiling from sources. In that case (and with a little further investigation which i've just done) we've decided to go with maildir! That is still going to be significantly better performing than mbox, right? Depends on the usage, but it's significantly better performing than UW-IMAP. Dovecot+mbox is also significantly faster than UW-IMAP+mbox. OK... so Dovecot is certainly significantly faster that uw-imapd in both cases, but is dovecot fastest with mbox or maildir? I would assume maildir, but you never know... Also... do you know how uw-imapd maildir compares to dovecot maildir in terms of performance? Maildir is a patch on top of the official UW-IMAP distribution. I don't know how well it performs, but it doesn't use any indexes and indexes are what makes Dovecot fast. Does dovecot still use indices with maildir? Yes. Excellent :-) mbox - Maildir conversion can preserve both IMAP and POP3 UIDLs using an external script. Maildir - dbox conversion can also preserve both, but that causes Dovecot to use this hybrid Maildir-dbox format, which is slower than the full native dbox. That'd good to know. Do you happen to know where I can get a copy of this external script you speak of? Will it simply be included in the debian package (probably)? http://wiki.dovecot.org/Migration/MailFormat - mb2md.py how does this differ from the convert-tool script packaged with dovecot itself? Also, given that i'm going to have to test this, i will obviously be running the conversion on a copy of the live data, and then i'll have to run the conversion again during the migration outage - will i need to delete all the data and basically start again, or is it incremental? I don't think it can do incremental, but I've never looked at the script myself. My guess is that two RAID-1s would be faster, but I haven't really done any benchmarking. Anyway index files are 10-30% of the mailbox size, so the index-disks would be using a lot less disk space. I assume you are talking
Re: [Dovecot] Migration questions...
On May 13, 2009, at 9:57 AM, Richard Hobbs wrote: Depends on the usage, but it's significantly better performing than UW-IMAP. Dovecot+mbox is also significantly faster than UW-IMAP+mbox. OK... so Dovecot is certainly significantly faster that uw-imapd in both cases, but is dovecot fastest with mbox or maildir? I would assume maildir, but you never know... It's not that simple to answer. With mbox it's probably faster to read through all mails, because they're in a single file. With Maildir it's faster to delete mails, because it only needs to delete a single file, instead of moving data around in the mbox file. But Maildir has less problems and it's much less likely to get corrupted, so even if mbox performance would be better in some cases I'd recommend Maildir. http://wiki.dovecot.org/Migration/MailFormat - mb2md.py how does this differ from the convert-tool script packaged with dovecot itself? convert-tool doesn't preserve UIDs, mb2md.py does. Would it be best to put all data on one array, and the indices on the other? We're basically after the fastest way to distribute the data! :-) Last I heard it was faster to keep index files in a separate disk than mailbox data. I've never verified this myself, but it sounds reasonable. Same here - i assume dovecot gives you a choice as to where it puts its index files, right? Also... where are the maildir data files kept? homedirs or elsewhere? mail_location setting specifies where both of them are kept. For example: mail_location = maildir:~/Maildir:INDEX=/var/index/%u However, does dovecot allow configuration of where to store both the index files *and* cache files? Whenever something is talking about index files, it also means cache files. Given that we have 2 RAID arrays, should we put index files *and* cache files on one array, and the data on another, or should put either the index files *or* cache files on the same array as the data? There's no choice, it's both or neither. BTW. I don't these optimizations will make much of a noticeable difference for you. A single server with recent hardware should be able to easily handle hundreds of simultaneous users.
Re: [Dovecot] Migration questions...
Timo Sirainen wrote: On May 13, 2009, at 9:57 AM, Richard Hobbs wrote: Depends on the usage, but it's significantly better performing than UW-IMAP. Dovecot+mbox is also significantly faster than UW-IMAP+mbox. OK... so Dovecot is certainly significantly faster that uw-imapd in both cases, but is dovecot fastest with mbox or maildir? I would assume maildir, but you never know... It's not that simple to answer. With mbox it's probably faster to read through all mails, because they're in a single file. With Maildir it's faster to delete mails, because it only needs to delete a single file, instead of moving data around in the mbox file. But Maildir has less problems and it's much less likely to get corrupted, so even if mbox performance would be better in some cases I'd recommend Maildir. OK... so in both cases, the files are indexed and headers etc... cached, so in both scenarios: maildir - slow to read mails, but all indexed, so slowness kinda disappears. mbox - slow to delete mails - indexing will help this problem, but the filesystem will still have work to do in order to join the two halves of the file. My take on the above is that overall, maildir will probably be faster, and if it's less likely to corrupt and has fewer problems, i think that's the format for us! http://wiki.dovecot.org/Migration/MailFormat - mb2md.py how does this differ from the convert-tool script packaged with dovecot itself? convert-tool doesn't preserve UIDs, mb2md.py does. Ah... that's a fairly critical piece of info, so thank you for that! :-) Would it be best to put all data on one array, and the indices on the other? We're basically after the fastest way to distribute the data! :-) Last I heard it was faster to keep index files in a separate disk than mailbox data. I've never verified this myself, but it sounds reasonable. Same here - i assume dovecot gives you a choice as to where it puts its index files, right? Also... where are the maildir data files kept? homedirs or elsewhere? mail_location setting specifies where both of them are kept. For example: mail_location = maildir:~/Maildir:INDEX=/var/index/%u Excellent - that makes life very simple! However, does dovecot allow configuration of where to store both the index files *and* cache files? Whenever something is talking about index files, it also means cache files. Given that we have 2 RAID arrays, should we put index files *and* cache files on one array, and the data on another, or should put either the index files *or* cache files on the same array as the data? There's no choice, it's both or neither. BTW. I don't these optimizations will make much of a noticeable difference for you. A single server with recent hardware should be able to easily handle hundreds of simultaneous users. That's also good to know... i like to do a job right instead of relying on faster hardware, as i'm sure you all do too, but it's good to know that if i make one or two non-optimal choices along the way, it'll probably be lightning fast anyway! The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. With maildir, and especially dovecot, this problem effectively disappears! That's the theory anyway, right? Thanks again! Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
on 5-13-2009 8:55 AM Richard Hobbs spake the following: Timo Sirainen wrote: On May 13, 2009, at 9:57 AM, Richard Hobbs wrote: Depends on the usage, but it's significantly better performing than UW-IMAP. Dovecot+mbox is also significantly faster than UW-IMAP+mbox. OK... so Dovecot is certainly significantly faster that uw-imapd in both cases, but is dovecot fastest with mbox or maildir? I would assume maildir, but you never know... It's not that simple to answer. With mbox it's probably faster to read through all mails, because they're in a single file. With Maildir it's faster to delete mails, because it only needs to delete a single file, instead of moving data around in the mbox file. But Maildir has less problems and it's much less likely to get corrupted, so even if mbox performance would be better in some cases I'd recommend Maildir. OK... so in both cases, the files are indexed and headers etc... cached, so in both scenarios: maildir - slow to read mails, but all indexed, so slowness kinda disappears. mbox - slow to delete mails - indexing will help this problem, but the filesystem will still have work to do in order to join the two halves of the file. Actually, I think a new file is written with everything re-written except the deleted message and then linked over the old file or renamed to the old file. That is why many clients will just mark them deleted and then you run a separate purge step, or the client is set to purge on exit. That is more efficient because the big write step is only done once. signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Migration questions...
On Wed, 2009-05-13 at 09:56 -0700, Scott Silva wrote: mbox - slow to delete mails - indexing will help this problem, but the filesystem will still have work to do in order to join the two halves of the file. Actually, I think a new file is written with everything re-written except the deleted message and then linked over the old file or renamed to the old file. That is why many clients will just mark them deleted and then you run a separate purge step, or the client is set to purge on exit. That is more efficient because the big write step is only done once. It's neither actually. It's about moving data inside the mbox file to get rid of the expunged messages and then truncating the mbox file. So this means that if you have a 1 GB mbox file and you delete the first message Dovecot needs to write 1 GB of data, but if you delete near the end of file it's going to be fast. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Migration questions...
Hello, Timo Sirainen wrote: On Mon, 2009-05-11 at 15:05 +, Richard Hobbs wrote: Hi All, We are soon to migrate our mail server from one piece of hardware to another and we would like to take this opportunity to optimize things. As a result, we would like to replace uw-imapd and qpopper with dovecot. The version we will be installing is 1.1.13-2, as this is what is available through the latest Debian stable backports. We will also be using exim to deliver mail (through dovecot's deliver mechanism, of course). So... We are currently using the mbox format with uw-imapd, and would like to migrate to the fastest solution possible with dovecot on the new hardware. My understanding is that multi-dbox is not an option in this version anyway, maildir is OK, but not great, and single-dbox is therefore going to be the highest performing solution. Is single-dbox the fastest way to go, considering we're going to be using email in the following ways: Single-dbox is the highest performing, but note that it's not as much tested as mbox and Maildir code. I think it should work ok, but I'm not aware of any larger installations using dbox currently. So in case you find a problem, you might have to upgrade/patch Dovecot to get it fixed and that would require compiling from sources. In that case (and with a little further investigation which i've just done) we've decided to go with maildir! That is still going to be significantly better performing than mbox, right? Also... do you know how uw-imapd maildir compares to dovecot maildir in terms of performance? Does dovecot still use indices with maildir? One problem that may arise is the fact that when we migrate, all msg UIDs will be lost. If i'm not mistaken, this means that all emails will be treated by the mail client as brand new, and if through IMAP, will all go bold, and if through POP3, will all be downloaded again (if still on the server) and therefore duplicated in the mail client. If this is the case, is there anything we can do to stop this happening? Does the Convert plugin does this job well? mbox - Maildir conversion can preserve both IMAP and POP3 UIDLs using an external script. Maildir - dbox conversion can also preserve both, but that causes Dovecot to use this hybrid Maildir-dbox format, which is slower than the full native dbox. That'd good to know. Do you happen to know where I can get a copy of this external script you speak of? Will it simply be included in the debian package (probably)? Also, given that i'm going to have to test this, i will obviously be running the conversion on a copy of the live data, and then i'll have to run the conversion again during the migration outage - will i need to delete all the data and basically start again, or is it incremental? 8. Configure the convert plugin to begin converting the mail to dbox format. 9. Run something manually (if possible) to convert mailboxes before people connect, so the task is already done by the time the outage is over. There's convert-tool that you could use. I don't know if Debian packages it. OK, fair enough... if this is a different script to the one you mention above, could you pls let me know where i can get this too? Basically, i suppose i'm asking, with the highest performing mailbox option, is dovecot going to run faster with 2 individual arrays each made from 2 disks, or a single 4-disk array (in which case we'd go RAID 0+1)? My guess is that two RAID-1s would be faster, but I haven't really done any benchmarking. Anyway index files are 10-30% of the mailbox size, so the index-disks would be using a lot less disk space. I assume you are talking about dovecot with maildir here, right? Also, what would we put on each array? Are the inboxes still stored separately to the IMAP folders when using dovecot and maildir? Would it be best to put all data on one array, and the indices on the other? We're basically after the fastest way to distribute the data! :-) Also... would it be useful to turn off atime when we mount the volume(s) or does dovecot rely on this? Dovecot doesn't rely on atime updates, so turn them off. Will do! Thanks again - your help is greatly appreciated :-) Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
Sorry people - i'm replying to my own email again... my reply is below! Richard Hobbs wrote: Hello, Timo Sirainen wrote: On Mon, 2009-05-11 at 15:05 +, Richard Hobbs wrote: Hi All, We are soon to migrate our mail server from one piece of hardware to another and we would like to take this opportunity to optimize things. As a result, we would like to replace uw-imapd and qpopper with dovecot. The version we will be installing is 1.1.13-2, as this is what is available through the latest Debian stable backports. We will also be using exim to deliver mail (through dovecot's deliver mechanism, of course). So... We are currently using the mbox format with uw-imapd, and would like to migrate to the fastest solution possible with dovecot on the new hardware. My understanding is that multi-dbox is not an option in this version anyway, maildir is OK, but not great, and single-dbox is therefore going to be the highest performing solution. Is single-dbox the fastest way to go, considering we're going to be using email in the following ways: Single-dbox is the highest performing, but note that it's not as much tested as mbox and Maildir code. I think it should work ok, but I'm not aware of any larger installations using dbox currently. So in case you find a problem, you might have to upgrade/patch Dovecot to get it fixed and that would require compiling from sources. In that case (and with a little further investigation which i've just done) we've decided to go with maildir! That is still going to be significantly better performing than mbox, right? Also... do you know how uw-imapd maildir compares to dovecot maildir in terms of performance? Does dovecot still use indices with maildir? One problem that may arise is the fact that when we migrate, all msg UIDs will be lost. If i'm not mistaken, this means that all emails will be treated by the mail client as brand new, and if through IMAP, will all go bold, and if through POP3, will all be downloaded again (if still on the server) and therefore duplicated in the mail client. If this is the case, is there anything we can do to stop this happening? Does the Convert plugin does this job well? mbox - Maildir conversion can preserve both IMAP and POP3 UIDLs using an external script. Maildir - dbox conversion can also preserve both, but that causes Dovecot to use this hybrid Maildir-dbox format, which is slower than the full native dbox. That'd good to know. Do you happen to know where I can get a copy of this external script you speak of? Will it simply be included in the debian package (probably)? Also, given that i'm going to have to test this, i will obviously be running the conversion on a copy of the live data, and then i'll have to run the conversion again during the migration outage - will i need to delete all the data and basically start again, or is it incremental? 8. Configure the convert plugin to begin converting the mail to dbox format. 9. Run something manually (if possible) to convert mailboxes before people connect, so the task is already done by the time the outage is over. There's convert-tool that you could use. I don't know if Debian packages it. OK, fair enough... if this is a different script to the one you mention above, could you pls let me know where i can get this too? Basically, i suppose i'm asking, with the highest performing mailbox option, is dovecot going to run faster with 2 individual arrays each made from 2 disks, or a single 4-disk array (in which case we'd go RAID 0+1)? My guess is that two RAID-1s would be faster, but I haven't really done any benchmarking. Anyway index files are 10-30% of the mailbox size, so the index-disks would be using a lot less disk space. I assume you are talking about dovecot with maildir here, right? Also, what would we put on each array? Are the inboxes still stored separately to the IMAP folders when using dovecot and maildir? Would it be best to put all data on one array, and the indices on the other? We're basically after the fastest way to distribute the data! :-) My colleague has mentioned something of interest... can dovecot keep the index files in RAM? If so, the performance will obviously be *so* much better than running them off the hard disks. This also raises questions about what happens if the machine is powered off etc... but it's UPSd etc... so if it were to rebuild it's indexes every time it was booted up, that wouldn't be the end of the world. Anyway... thoughts on this topic are also appreciated :-) Thanks again! Also... would it be useful to turn off atime when we mount the volume(s) or does dovecot rely on this? Dovecot doesn't rely on atime updates, so turn them off. Will do! Thanks again - your help is greatly appreciated :-) Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web:
Re: [Dovecot] Migration questions...
I don't know whether this would help with the migration, but I routinely
solve a similar problem. I have implemented mail failover between two
servers -- which are configured with identical sets of mailboxes -- and
every 10 minutes or so, a script grabs any E-Mails from the other server
and stores them locally in the proper mailboxes. This script relies on
IMAP connectivity and passwords in plaintext in a passwd-file. Here it is:
---
#!/usr/bin/perl
require '/vmail/Simple.pm';
use Net::SMTP;
$passwdfile = /path/to/passwd-file;
$remoteserver = myotherserver.com;
open(FIN, $passwdfile);
@pwlines = (FIN);
close(FIN);
for ($i = 0; $i = $#pwlines; $i++)
{
@pwflds = split(/[:]/, $pwlines[$i]);
if ($#pwflds == 3)
{
@pwparts = split(/[}]/, $pwflds[1]);
if ($#pwparts == 1)
{
$thislogin = $pwflds[0];
$thispassword = $pwparts[1];
transfermail($remoteserver, $thislogin, $thispassword);
}
}
}
sub transfermail
{
my $remoteserver = $_[0];
my $thislogin = $_[1];
my $thispassword = $_[2];
print $remoteserver, - , $thislogin, - , $thispassword, \n;
my $server = new Net::IMAP::Simple( $remoteserver );
my $login_status = $server-login( $thislogin, $thispassword );
if ($login_status)
{
my $number_of_messages = $server-select(INBOX);
print $thislogin, - , $number_of_messages, messages.\n;
my $msg;
for ($msg = 1; $msg = $number_of_messages; $msg++)
{
$ok = 1;
$lines = $server-get( $msg ) or $ok = 0;
$smtp = Net::SMTP-new(127.0.0.1) or $ok = 0;
$smtp-mail($thislogin) or $ok = 0;
$smtp-recipient($thislogin) or $ok = 0;
$smtp-data() or $ok = 0;
$smtp-datasend(@$lines) or $ok = 0;
$smtp-dataend() or $ok = 0;
$smtp-quit or $ok = 0;
if ($ok)
{
$server-delete( $msg );
}
}
}
else
{
print $thislogin, - could not log in.\n;
}
$server-quit();
}
1;
Re: [Dovecot] Migration questions...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, May 12, 2009 at 10:09:06AM +, Richard Hobbs wrote: Hello, [...] That'd good to know. Do you happen to know where I can get a copy of this external script you speak of? Will it simply be included in the debian package (probably)? | to...@floh:~$ apt-file search convert-tool | dovecot-common: /usr/bin/convert-tool Seems to be there :-) HTH - -- tomás -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFKCXZKBcgs9XrR2kYRAoSyAJ4omHYXeXcOGoy0i64ZAf7fOOlF9gCcCYnQ tI3hPvMdhCxJ3z8y3LlMBa4= =aXxn -END PGP SIGNATURE-
Re: [Dovecot] Migration questions...
Blast, forgot something: the Simple.pm referenced in the script is
this thing:
http://search.cpan.org/~jpaf/Net-IMAP-Simple-0.93/Simple.pm
Download it, compile it, put it somewhere that the script can find it.
I don't know whether this would help with the migration, but I
routinely solve a similar problem. I have implemented mail failover
between two servers -- which are configured with identical sets of
mailboxes -- and every 10 minutes or so, a script grabs any E-Mails
from the other server and stores them locally in the proper
mailboxes. This script relies on IMAP connectivity and passwords in
plaintext in a passwd-file. Here it is:
---
#!/usr/bin/perl
require '/vmail/Simple.pm';
use Net::SMTP;
$passwdfile = /path/to/passwd-file;
$remoteserver = myotherserver.com;
open(FIN, $passwdfile);
@pwlines = (FIN);
close(FIN);
for ($i = 0; $i = $#pwlines; $i++)
{
@pwflds = split(/[:]/, $pwlines[$i]);
if ($#pwflds == 3)
{
@pwparts = split(/[}]/, $pwflds[1]);
if ($#pwparts == 1)
{
$thislogin = $pwflds[0];
$thispassword = $pwparts[1];
transfermail($remoteserver, $thislogin, $thispassword);
}
}
}
sub transfermail
{
my $remoteserver = $_[0];
my $thislogin = $_[1];
my $thispassword = $_[2];
print $remoteserver, - , $thislogin, - , $thispassword, \n;
my $server = new Net::IMAP::Simple( $remoteserver );
my $login_status = $server-login( $thislogin, $thispassword );
if ($login_status)
{
my $number_of_messages = $server-select(INBOX);
print $thislogin, - , $number_of_messages, messages.\n;
my $msg;
for ($msg = 1; $msg = $number_of_messages; $msg++)
{
$ok = 1;
$lines = $server-get( $msg ) or $ok = 0;
$smtp = Net::SMTP-new(127.0.0.1) or $ok = 0;
$smtp-mail($thislogin) or $ok = 0;
$smtp-recipient($thislogin) or $ok = 0;
$smtp-data() or $ok = 0;
$smtp-datasend(@$lines) or $ok = 0;
$smtp-dataend() or $ok = 0;
$smtp-quit or $ok = 0;
if ($ok)
{
$server-delete( $msg );
}
}
}
else
{
print $thislogin, - could not log in.\n;
}
$server-quit();
}
1;
Re: [Dovecot] Migration questions...
On May 12, 2009, at 6:41 AM, Richard Hobbs wrote: Single-dbox is the highest performing, but note that it's not as much tested as mbox and Maildir code. I think it should work ok, but I'm not aware of any larger installations using dbox currently. So in case you find a problem, you might have to upgrade/patch Dovecot to get it fixed and that would require compiling from sources. In that case (and with a little further investigation which i've just done) we've decided to go with maildir! That is still going to be significantly better performing than mbox, right? Depends on the usage, but it's significantly better performing than UW- IMAP. Dovecot+mbox is also significantly faster than UW-IMAP+mbox. Also... do you know how uw-imapd maildir compares to dovecot maildir in terms of performance? Maildir is a patch on top of the official UW-IMAP distribution. I don't know how well it performs, but it doesn't use any indexes and indexes are what makes Dovecot fast. Does dovecot still use indices with maildir? Yes. mbox - Maildir conversion can preserve both IMAP and POP3 UIDLs using an external script. Maildir - dbox conversion can also preserve both, but that causes Dovecot to use this hybrid Maildir-dbox format, which is slower than the full native dbox. That'd good to know. Do you happen to know where I can get a copy of this external script you speak of? Will it simply be included in the debian package (probably)? http://wiki.dovecot.org/Migration/MailFormat - mb2md.py Also, given that i'm going to have to test this, i will obviously be running the conversion on a copy of the live data, and then i'll have to run the conversion again during the migration outage - will i need to delete all the data and basically start again, or is it incremental? I don't think it can do incremental, but I've never looked at the script myself. My guess is that two RAID-1s would be faster, but I haven't really done any benchmarking. Anyway index files are 10-30% of the mailbox size, so the index-disks would be using a lot less disk space. I assume you are talking about dovecot with maildir here, right? The same applies to all mailbox formats Dovecot supports. Also, what would we put on each array? Are the inboxes still stored separately to the IMAP folders when using dovecot and maildir? Inboxes are stored inside Maildir like all other mailboxes. Would it be best to put all data on one array, and the indices on the other? We're basically after the fastest way to distribute the data! :-) Last I heard it was faster to keep index files in a separate disk than mailbox data. I've never verified this myself, but it sounds reasonable. My colleague has mentioned something of interest... can dovecot keep the index files in RAM? If so, the performance will obviously be *so* much better than running them off the hard disks. Last I heard it didn't really help much. Assuming your OS works properly it already keeps the necessary indexes in memory anyway. Also I wrote a patch that tries to tell OS to do that by dropping message files' data from cache after reading the messages: http://dovecot.org/patches/1.1/fadvise.diff But no one has told me if that helps or makes things worse.. This also raises questions about what happens if the machine is powered off etc... but it's UPSd etc... so if it were to rebuild it's indexes every time it was booted up, that wouldn't be the end of the world. Well, there are two parts of index rebuilding: dovecot.index files which are quick to rebuild and dovecot.index.cache files that contain the useful fields that clients want. The cache file is especially useful with webmails and if it's gone it could mean opening user's all messages and reading their headers and perhaps even bodies. So depends on what clients your users use, but in some cases it could be 10-100x slower to open the mailbox if the cache file is gone.
Re: [Dovecot] Migration questions...
Richard Hobbs wrote: Hi All, We are soon to migrate our mail server from one piece of hardware to another and we would like to take this opportunity to optimize things. Can I recommend you add virtualisation to your todo list. I use linux-vserver, but there are plenty other ideas out there. It's just superb that later on you can migrate services between physical hardware with MUCH less hassle. You can easily test upgrades in a sandbox first. etc I personally split all the tasks into different virtual servers. Right now I actually still have quite a few mail related services in a single vserver, but ideally you would split everything up and then later if you needed to upgrade a single service or move it to a new machine it would have minimal effect. Your general process sounds about right though - I think it may be possible for you to preserve pop uidls, but see the wiki for more notes on that Good luck Ed W
Re: [Dovecot] Migration questions...
Richard Hobbs wrote: My colleague has mentioned something of interest... can dovecot keep the index files in RAM? If so, the performance will obviously be *so* much better than running them off the hard disks. My understanding was that in-memory indicies are discarded on logout. They're of benefit generally in cases where (a) you are distributing your servers and don't want to risk indicies over NFS, or (b) your users are doing POP drive-bys, and not storing on the server (though I thought there was a don't bother indexing at all for this). This also raises questions about what happens if the machine is powered off etc... but it's UPSd etc... so if it were to rebuild it's indexes every time it was booted up, that wouldn't be the end of the world. Dovecot will rebuild its indices when it needs to -- either they're not there, or they're deemed invalid (corrupted, etc). Except in the case of dbox, they are a bonus only -- not essential. You could even reduce your backup footprint by omitting them. -- Curtis Maloney cmalo...@cardgate.net
Re: [Dovecot] Migration questions...
Hello, Apologies for replying to my own email so soon, but I've had other thoughts as well... Our server is going to have 4 hard disks. These can be configured into 2 RAID 1 (mirror) arrays, a single RAID 5 array, or a single RAID 0+1 array. Previously, we thought that two RAID 1 arrays would be best because the inboxes can sit on one set of disks, and the IMAP folders on another. This is so both types of user don't annoy each other. However, with no knowledge of how the highest performing mailbox format works with dovecot, perhaps this isn't the best option. Basically, i suppose i'm asking, with the highest performing mailbox option, is dovecot going to run faster with 2 individual arrays each made from 2 disks, or a single 4-disk array (in which case we'd go RAID 0+1)? Also... would it be useful to turn off atime when we mount the volume(s) or does dovecot rely on this? Thanks again, people... :-) Richard. Richard Hobbs wrote: Hi All, We are soon to migrate our mail server from one piece of hardware to another and we would like to take this opportunity to optimize things. As a result, we would like to replace uw-imapd and qpopper with dovecot. The version we will be installing is 1.1.13-2, as this is what is available through the latest Debian stable backports. We will also be using exim to deliver mail (through dovecot's deliver mechanism, of course). So... We are currently using the mbox format with uw-imapd, and would like to migrate to the fastest solution possible with dovecot on the new hardware. My understanding is that multi-dbox is not an option in this version anyway, maildir is OK, but not great, and single-dbox is therefore going to be the highest performing solution. Is single-dbox the fastest way to go, considering we're going to be using email in the following ways: - IMAP connections with all email in the Inbox (Gmail-style). - IMAP connections with email split into many IMAP folders. - POP3 connections with no email left on the server. - POP3 connections with *all* email left on the server. All connections check for new mail every 5 minutes (on average) and there are 50-60 users). Also, we are not able to change user behaviour in this instance, unfortunately. Can anyone see any problems with the above proposal? Hopefully not... One problem that may arise is the fact that when we migrate, all msg UIDs will be lost. If i'm not mistaken, this means that all emails will be treated by the mail client as brand new, and if through IMAP, will all go bold, and if through POP3, will all be downloaded again (if still on the server) and therefore duplicated in the mail client. If this is the case, is there anything we can do to stop this happening? Does the Convert plugin does this job well? Finally, I have a rough draft of our migration plan - is there anything horribly wrong in it that's going to cause lots of problems, that anyone can spot by any chance? 1. Install Debian with exim, mysqld (for Horde/IMP) and mailman. 2. Run an online update. 3. Rsync homedirs and inboxes onto new server, ready for initial exim configuration. 4. Configure exim as per existing mail server and test that mailing lists and normal email works. You should now have the existing mail delivery solution on the brand new hardware. 5. Once mail delivery is sorted, add deb http://www.backports.org/debian lenny-backports main contrib non-free into /etc/apt/sources.list and run aptitude update aptitude install debian-backports-keyring aptitude update. 6. Install dovecot (at the time of writing, this was version 1.1.13-2) and configure to use existing mbox files (inboxes in /var/spool/mail/ and IMAP folders in /home/user/mail/) 7. Setup exim to use dovecot's deliver mechanism for interacting with the inboxes (which are still in mbox format). 8. Configure the convert plugin to begin converting the mail to dbox format. 9. Run something manually (if possible) to convert mailboxes before people connect, so the task is already done by the time the outage is over. 10. Give staff access to new speedy mail server! Thanks in advance, people - any help is greatly appreciated! :-) Richard. -- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.ho...@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999Mobile: +44 7811 803377 smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Migration questions...
On Mon, 2009-05-11 at 15:05 +, Richard Hobbs wrote: Hi All, We are soon to migrate our mail server from one piece of hardware to another and we would like to take this opportunity to optimize things. As a result, we would like to replace uw-imapd and qpopper with dovecot. The version we will be installing is 1.1.13-2, as this is what is available through the latest Debian stable backports. We will also be using exim to deliver mail (through dovecot's deliver mechanism, of course). So... We are currently using the mbox format with uw-imapd, and would like to migrate to the fastest solution possible with dovecot on the new hardware. My understanding is that multi-dbox is not an option in this version anyway, maildir is OK, but not great, and single-dbox is therefore going to be the highest performing solution. Is single-dbox the fastest way to go, considering we're going to be using email in the following ways: Single-dbox is the highest performing, but note that it's not as much tested as mbox and Maildir code. I think it should work ok, but I'm not aware of any larger installations using dbox currently. So in case you find a problem, you might have to upgrade/patch Dovecot to get it fixed and that would require compiling from sources. One problem that may arise is the fact that when we migrate, all msg UIDs will be lost. If i'm not mistaken, this means that all emails will be treated by the mail client as brand new, and if through IMAP, will all go bold, and if through POP3, will all be downloaded again (if still on the server) and therefore duplicated in the mail client. If this is the case, is there anything we can do to stop this happening? Does the Convert plugin does this job well? mbox - Maildir conversion can preserve both IMAP and POP3 UIDLs using an external script. Maildir - dbox conversion can also preserve both, but that causes Dovecot to use this hybrid Maildir-dbox format, which is slower than the full native dbox. 8. Configure the convert plugin to begin converting the mail to dbox format. 9. Run something manually (if possible) to convert mailboxes before people connect, so the task is already done by the time the outage is over. There's convert-tool that you could use. I don't know if Debian packages it. Basically, i suppose i'm asking, with the highest performing mailbox option, is dovecot going to run faster with 2 individual arrays each made from 2 disks, or a single 4-disk array (in which case we'd go RAID 0+1)? My guess is that two RAID-1s would be faster, but I haven't really done any benchmarking. Anyway index files are 10-30% of the mailbox size, so the index-disks would be using a lot less disk space. Also... would it be useful to turn off atime when we mount the volume(s) or does dovecot rely on this? Dovecot doesn't rely on atime updates, so turn them off. signature.asc Description: This is a digitally signed message part
