Re: Recommend how to move 31GB of mail to a new server
> Would it be possible to use replication for this? > > Set up a replica of the first server, copy while both servers online. > Then take first server offline and change IPs/servernames/whatever. Yes, it's definitely possible, but potentially quite a bit of work. We actually have some modules that use replication to allow you to move one user at a time between arbitrary servers. Basically it: 1. Creates a new temporary cyrus config dir with db,quota,sieve,user sub-dirs 2. Generates on the fly a new imapd.conf file with the appropriate sync_* params 3. Creates a new mailboxes.db with just the user directories (it hacks the partition to be the same as the destination one wanted, but makes the dirs in the temp imapd.conf for that partition point to the old place) 4. symlinks the users seen,sub,quota,sieve files to the new tmp config dir 5. Runs a sync_client -u to copy the user This basically allow arbitary moving of any single user from any server to any other server on any partition. The nice thing is this can happily run while the user is logged in and doing stuff. Once it's run once, we then disable logins and email deliveries for the user and disconnect them from any imap/pop session, run the sync again (generally very quick), update the db with which server they're on, then re-enable them again. It's quite neat. Unfortunately the code isn't really a releaseable state, there's too many FM only assumptions and details in it. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
Jose Hales-Garcia schrieb: > I'm preparing a new iMAP server and have 31GB of user mail on the > current server to move. I'm looking for suggestions on doing the move > with the minimum of down time as possible. Would it be possible to use replication for this? Set up a replica of the first server, copy while both servers online. Then take first server offline and change IPs/servernames/whatever. -- Martin Schütte Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: Recommend how to move 31GB of mail to a new server
If you run one of your rsync updates just before you shutdown, the one after will be faster because of operating system caching. I would guess just a couple of minutes after stopping the services with a fast connection between the servers. So fast it will be the least of your configuration issues. We dumped and imported mailboxes.db while the rsync was running; and in our case, moved the service IP between servers. We preferred moving a secondary IP because it was faster to move it back if things weren't working as they should, and in our case with a few silly mistakes, we did quickly revert a few times. A firewall routing change would have been nicer. Just be careful what you sync to avoid overwriting updated DB files or whatever if you are updating at the same time. It took us a few tries to get the timing right and have a working system when we started the new server and began a reconstruct. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Daniel O'Connor Sent: Sunday, June 24, 2007 12:58 AM To: info-cyrus@lists.andrew.cmu.edu Subject: Re: Recommend how to move 31GB of mail to a new server On Sunday 24 June 2007 11:04, Patrick Boutilier wrote: > > That should save considerable time as I would imagine the vast > > majority of email would be unchanged between the first & second > > copy. > > But unfortunately rsync will still have to scan each file to > determine what has changed, and that can chew up a lot of time. Yes, but hopefully less time that copying the files :) By default it will compare mtime and size and skip files which match. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
Ulrich Spoerlein wrote: > On Sat, 23.06.2007 at 22:34:33 -0300, Patrick Boutilier wrote: >> Daniel O'Connor wrote: >>> On Sunday 24 June 2007 07:37, Gary Mills wrote: For only 30 gigabytes, you might be better off just copying the files over, with IMAP down. It could only take a few hours. You can copy a sample from the live system to get an idea of the timing. >>> You could rsync the mail spool while it is live, take it down and then >>> rsync again. >>> >>> That should save considerable time as I would imagine the vast majority >>> of email would be unchanged between the first & second copy. >> But unfortunately rsync will still have to scan each file to determine >> what has changed, and that can chew up a lot of time. > > Better than to speculate is to measure. > Time the rsync of the mailboxes to your new server. *Don't* shutdown > cyrus, but rsync right again. Measure the second run, too. This will > roughly be your expected downtime. I would be surprised if it is more > than 5-10 minutes. Depends on how many files are on the filesystem. I have done this kind of move before and to speed things up I did a seperate rsync for each of the letters (A-W) in the mail partition. > > You could also do: rsync (long time), rsync (short time), shutdown > cyrus, rsync (even shorter time). > > hth > Ulrich Spoerlein Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
On Sun, Jun 24, 2007 at 10:50:28AM +0200, Ulrich Spoerlein wrote: > On Sat, 23.06.2007 at 22:34:33 -0300, Patrick Boutilier wrote: > > Daniel O'Connor wrote: > > > On Sunday 24 June 2007 07:37, Gary Mills wrote: > > >> For only 30 gigabytes, you might be better off just copying the files > > >> over, with IMAP down. It could only take a few hours. You can copy > > >> a sample from the live system to get an idea of the timing. > > > > > > You could rsync the mail spool while it is live, take it down and then > > > rsync again. > > > > > > That should save considerable time as I would imagine the vast majority > > > of email would be unchanged between the first & second copy. > > > > But unfortunately rsync will still have to scan each file to determine > > what has changed, and that can chew up a lot of time. > > Better than to speculate is to measure. > Time the rsync of the mailboxes to your new server. *Don't* shutdown > cyrus, but rsync right again. Measure the second run, too. This will > roughly be your expected downtime. I would be surprised if it is more > than 5-10 minutes. > > You could also do: rsync (long time), rsync (short time), shutdown > cyrus, rsync (even shorter time). I did this a couple of years ago with some 250GB mail. Machines physically close, so good connectivity, but 250GB still takes forever. I ran rsync over a few nights, just killing it at 8am. This got us close to where we wanted to be, then on a Saturday I took everything down and ran a final rsync that, IIRC, took about 1/2 hour; switched cyrus/mail/... on the new server on and voila - migration completed. -- Alain Williams Linux Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer. +44 (0) 787 668 0256 http://www.phcomp.co.uk/ Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php #include Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
On Sat, 23.06.2007 at 22:34:33 -0300, Patrick Boutilier wrote: > Daniel O'Connor wrote: > > On Sunday 24 June 2007 07:37, Gary Mills wrote: > >> For only 30 gigabytes, you might be better off just copying the files > >> over, with IMAP down. It could only take a few hours. You can copy > >> a sample from the live system to get an idea of the timing. > > > > You could rsync the mail spool while it is live, take it down and then > > rsync again. > > > > That should save considerable time as I would imagine the vast majority > > of email would be unchanged between the first & second copy. > > But unfortunately rsync will still have to scan each file to determine > what has changed, and that can chew up a lot of time. Better than to speculate is to measure. Time the rsync of the mailboxes to your new server. *Don't* shutdown cyrus, but rsync right again. Measure the second run, too. This will roughly be your expected downtime. I would be surprised if it is more than 5-10 minutes. You could also do: rsync (long time), rsync (short time), shutdown cyrus, rsync (even shorter time). hth Ulrich Spoerlein -- "The trouble with the dictionary is you have to know how the word is spelled before you can look it up to see how it is spelled." -- Will Cuppy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
On Sunday 24 June 2007 11:04, Patrick Boutilier wrote: > > That should save considerable time as I would imagine the vast > > majority of email would be unchanged between the first & second > > copy. > > But unfortunately rsync will still have to scan each file to > determine what has changed, and that can chew up a lot of time. Yes, but hopefully less time that copying the files :) By default it will compare mtime and size and skip files which match. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C pgpdSHrdhwA3p.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
Daniel O'Connor wrote: > On Sunday 24 June 2007 07:37, Gary Mills wrote: >> For only 30 gigabytes, you might be better off just copying the files >> over, with IMAP down. It could only take a few hours. You can copy >> a sample from the live system to get an idea of the timing. > > You could rsync the mail spool while it is live, take it down and then > rsync again. > > That should save considerable time as I would imagine the vast majority > of email would be unchanged between the first & second copy. But unfortunately rsync will still have to scan each file to determine what has changed, and that can chew up a lot of time. > > > > > > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
On Sunday 24 June 2007 07:37, Gary Mills wrote: > For only 30 gigabytes, you might be better off just copying the files > over, with IMAP down. It could only take a few hours. You can copy > a sample from the live system to get an idea of the timing. You could rsync the mail spool while it is live, take it down and then rsync again. That should save considerable time as I would imagine the vast majority of email would be unchanged between the first & second copy. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C pgps8bnXgUPJp.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
On Sat, Jun 23, 2007 at 02:33:46PM -0700, Jose Hales-Garcia wrote: > >I'm preparing a new iMAP server and have 31GB of user mail on the >current server to move. I'm looking for suggestions on doing the move >with the minimum of down time as possible. I did this recently, although it was 300 gigabytes. My strategy was to convert the old server into a unified murder and configure the new server as a standard murder node. I was then able to use the XFER function of Cyrus IMAP to move mailboxes transparently from the old server to the new one. There was no downtime other than for occasional reboots. The migration took a couple of weeks. To begin, I had to upgrade the old server to cyrus-imapd-2.3.8. I recommend great caution if you follow this example. I tested all of my procedures on a pair of test servers before using them on the production servers, finding a few bugs and other pitfalls along the way. For only 30 gigabytes, you might be better off just copying the files over, with IMAP down. It could only take a few hours. You can copy a sample from the live system to get an idea of the timing. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Recommend how to move 31GB of mail to a new server
I'm preparing a new iMAP server and have 31GB of user mail on the current server to move. I'm looking for suggestions on doing the move with the minimum of down time as possible. Thank you for your suggestions, Jose ... Jose Hales-Garcia UCLA Department of Statistics [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
