Move of mailboxes
Hallo ALL, we have 2 backends in a murder environment (running cyrus 2.3.11) which works. And now for the problem: We want to replace both backends by new hardware. Therefore the mailboxes from both must be moved to the replacement machines. Is there a recommended procedure to do the move? Any pointers (even to pitfalls) are welcome. Thanks in advance and kind regards, Frank Elsner Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Move of mailboxes
On 2011-05-05 10:15, Frank Elsner wrote: Hallo ALL, we have 2 backends in a murder environment (running cyrus 2.3.11) which works. And now for the problem: We want to replace both backends by new hardware. Therefore the mailboxes from both must be moved to the replacement machines. Is there a recommended procedure to do the move? Any pointers (even to pitfalls) are welcome. I am in the way of replacing a old server with a new one. I just rsync and let cyrus 2.4.8 upgrade all databases on its own on first start. Does look fine so far. Thanks in advance and kind regards, Frank Elsner MfG, Lars Schimmer -- - TU Graz, Institut für ComputerGraphik WissensVisualisierung Tel: +43 316 873-5405 E-Mail: l.schim...@cgv.tugraz.at Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Move of mailboxes
On 5 May 2011, at 09:15, Frank Elsner wrote: we have 2 backends in a murder environment (running cyrus 2.3.11) which works. And now for the problem: We want to replace both backends by new hardware. Therefore the mailboxes from both must be moved to the replacement machines. Is there a recommended procedure to do the move? Any pointers (even to pitfalls) are welcome. You should upgrade Cyrus on the new machines because there's a quota bug 2.3.14 (I think) where the quota on the new machine counts expunged messages or something similar so you end up needing to run quota -f to fix it. For a live migration, where downtime per mailbox is minimal - add the new hardware into the murder as machines 3 and 4. Use XFER to move the mailboxes to the new machines. XFER from 2.3.12 to 2.4.8 worked fine here. The problem with this method is you have to issue an XFER command for each mailbox separately - fine if you have a separate database with mailboxes in it, but not so helpful if you only have the cyradm listmailbox output. Advantages are that you can transfer a test mailbox over first to check that the new server works ok (authentication etc) and you can do the migration over the course of a few days rather than having to do them all at once. If you can afford to take the servers offline for a few hours to half a day or more depending on spool size, you can simply copy the data over. This has the advantage that you can name the new server exactly the same as the old server. Stop cyrus (and exim/sendmail/postfix) on both old and new servers, use rsync to copy the spool and conf directories over to the new servers and then start Cyrus. Pitfalls here include things like mismatched BDB versions (convert mailboxes.db and others to skiplist format first). I've successfully migrated 2.2.x (CentOS4) to 2.4.8 using this method. Simon -- Simon Amor si...@leaky.org http://www.leaky.org/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Move of mailboxes
Is there a recommended procedure to do the move? Any pointers (even to pitfalls) are welcome. I'm actually in the middle of migrating two backends of a Murder (2.2.13p1) to new machines (and Cyrus version, 2.3.16, though it doesn't sound like you're switching Cyrus versions in your case). Full disclosure, no idea if this is the recommended procedure and 2.2.13's probably too old to be that useful, but it's been working for me. The way I've been doing it is writing a small script that: 1) Checks $cyrus_var/proc to see if the user is logged in. 2) If they are not, connects to one of the proxy frontends, changes permissions on the mailbox to (temporarily) stop the user from manipulating it if they were to log in mid transfer. 3) Issue a rename (which I think ultimately just makes an XFER call to the appropriate backend?) of the format: user.melson user.melson mailboxes3.example.com!partition1 (for example). 4) Set the permissions back to normal after success. (I may be changing this permission switching to temporarily deny the user authentication, that method just didn't fit my environment when I started this). The script's just some perl using Cyrus::IMAP::Admin, and it should be fairly easily to replicate in anything with an IMAP library. (It's basically the same process I use to shuffle people between backends in the murder environment normally, which was handy from a laziness standpoint; the process is used for moving from 2.3.16-2.3.16 backends as well) It's been pretty successful so far, no user has noticed anything amiss (that said, most of my users are relatively light users), and delivered email just gets queued up during the transfer as near as I can tell. I've found it to be an ideal way to go about migration/upgrades - I can introduce users into the new environment slowly to catch configuration glitches, make sure I didn't misjudge the resources I would require, etc, etc. Very handy. It's been good about catching problems, even (I have a customization that extends the valid characters in mailboxes I forgot to apply on the new machines; the XFER noticed right away and left the user's mailbox unmoved and unmolested) Oh, one thing I forgot - I'm fairly sure you *do* have to set allowusermoves: 1 in your configuration if you're using this method. The gotchas I've run into are probably more quirks of the environment I'm working with or relics of a relatively old Cyrus at this point, but I'll share them in case they are useful. *) I've had some weirdness if the transfer gets interrupted in the middle (I forget the exact symptom, but the mailbox on the home spool was flagged as REMOTE (or something like that), but it hadn't actually made it over to the new machine - a simple ctl_mboxlist -m on the backend fixed it up since the frontend had the right information). *) This is probably more my environment than anything else, but I've had to throttle the moves and not run too many simultaneous moves in parallel or I basically overwhelm the server I'm migrating from - it seems to actually be the delete of the mailbox from its old home that hits the hardest. I've not looked into it much, because, old crusty server (2.4 kernel, ext3 file system; like I said, probably my environment :P). -- Matt Elson mel...@fastmail.net Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Move of mailboxes
try this: http://cynici.wordpress.com/2010/12/06/how-to-migrate-32-bit-cyrus-imapd-mailboxes-to-64-bit/ On 5/5/2011 1:08 μμ, Matt Elson wrote: Is there a recommended procedure to do the move? Any pointers (even to pitfalls) are welcome. I'm actually in the middle of migrating two backends of a Murder (2.2.13p1) to new machines (and Cyrus version, 2.3.16, though it doesn't sound like you're switching Cyrus versions in your case). Full disclosure, no idea if this is the recommended procedure and 2.2.13's probably too old to be that useful, but it's been working for me. The way I've been doing it is writing a small script that: 1) Checks $cyrus_var/proc to see if the user is logged in. 2) If they are not, connects to one of the proxy frontends, changes permissions on the mailbox to (temporarily) stop the user from manipulating it if they were to log in mid transfer. 3) Issue a rename (which I think ultimately just makes an XFER call to the appropriate backend?) of the format: user.melson user.melson mailboxes3.example.com!partition1 (for example). 4) Set the permissions back to normal after success. (I may be changing this permission switching to temporarily deny the user authentication, that method just didn't fit my environment when I started this). The script's just some perl using Cyrus::IMAP::Admin, and it should be fairly easily to replicate in anything with an IMAP library. (It's basically the same process I use to shuffle people between backends in the murder environment normally, which was handy from a laziness standpoint; the process is used for moving from 2.3.16-2.3.16 backends as well) It's been pretty successful so far, no user has noticed anything amiss (that said, most of my users are relatively light users), and delivered email just gets queued up during the transfer as near as I can tell. I've found it to be an ideal way to go about migration/upgrades - I can introduce users into the new environment slowly to catch configuration glitches, make sure I didn't misjudge the resources I would require, etc, etc. Very handy. It's been good about catching problems, even (I have a customization that extends the valid characters in mailboxes I forgot to apply on the new machines; the XFER noticed right away and left the user's mailbox unmoved and unmolested) Oh, one thing I forgot - I'm fairly sure you *do* have to set allowusermoves: 1 in your configuration if you're using this method. The gotchas I've run into are probably more quirks of the environment I'm working with or relics of a relatively old Cyrus at this point, but I'll share them in case they are useful. *) I've had some weirdness if the transfer gets interrupted in the middle (I forget the exact symptom, but the mailbox on the home spool was flagged as REMOTE (or something like that), but it hadn't actually made it over to the new machine - a simple ctl_mboxlist -m on the backend fixed it up since the frontend had the right information). *) This is probably more my environment than anything else, but I've had to throttle the moves and not run too many simultaneous moves in parallel or I basically overwhelm the server I'm migrating from - it seems to actually be the delete of the mailbox from its old home that hits the hardest. I've not looked into it much, because, old crusty server (2.4 kernel, ext3 file system; like I said, probably my environment :P). -- *Γατσής Νίκος - Gatsis Nikos* Web developer tel.: 2108256721 - 2108256722 fax: 2108256712 email: ngat...@qbit.gr http://www.qbit.gr Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Move of mailboxes
we have 2 backends in a murder environment (running cyrus 2.3.11) which works. And now for the problem: We want to replace both backends by new hardware. Therefore the mailboxes from both must be moved to the replacement machines. Is there a recommended procedure to do the move? Any pointers (even to pitfalls) are welcome. You should upgrade Cyrus on the new machines because there's a quota bug 2.3.14 (I think) where the quota on the new machine counts expunged messages or something similar so you end up needing to run quota -f to fix it. For a live migration, where downtime per mailbox is minimal - add the new hardware into the murder as machines 3 and 4. Use XFER to move the mailboxes to the new machines. ... with the consequence that you need more disk space on the new server if you're using singleinstancestore: 1 in /etc/imapd.conf (default), as you 'll loose the hardlinks. In our case this would be a facor of 1.3. Frank Richter attachment: frank_richter.vcf smime.p7s Description: S/MIME Cryptographic Signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Script to move imap mailboxes between backends and partitions ?
Hi, Anyone has a good script to move users between several backends and partitions ?? In the 'old' cyrus web page i remember that exist link for some tools and scripts in a Wiki. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Script to move imap mailboxes between backends and partitions ?
On Wed, 1 Dec 2010, Lucas Zinato Carraro wrote: Hi, Anyone has a good script to move users between several backends and partitions ?? In the 'old' cyrus web page i remember that exist link for some tools and scripts in a Wiki. I have a variety of perl scripts for managing our Cyrus Murder installation. They were in the Wiki at one time, but you can find them here now: http://people.oregonstate.edu/~morgan/cyrus/public/ In particular, look at: batch-move-mailboxes-to-new-backend.pl rename-with-backend.pl rename-with-partition.pl utils.pl Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Script to move imap mailboxes between backends and partitions ?
Thanks Andrew, This scripts could be included im cyrusimap web page. Regards Zinato On Wed, Dec 1, 2010 at 8:28 PM, Andrew Morgan mor...@orst.edu wrote: On Wed, 1 Dec 2010, Lucas Zinato Carraro wrote: Hi, Anyone has a good script to move users between several backends and partitions ?? In the 'old' cyrus web page i remember that exist link for some tools and scripts in a Wiki. I have a variety of perl scripts for managing our Cyrus Murder installation. They were in the Wiki at one time, but you can find them here now: http://people.oregonstate.edu/~morgan/cyrus/public/ In particular, look at: batch-move-mailboxes-to-new-backend.pl rename-with-backend.pl rename-with-partition.pl utils.pl Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
