Re: mailbox / user migration to a new server

[Niels Dettenbach (Syndicat IT & Internet) via Info-cyrus]

Am 6. Januar 2018 23:22:03 MEZ schrieb Scott Lambert 
>With cyrus-imapd on both ends you shouldn't have to know the
>users' passwords.  The cyrus admin user should be able to login
>to the user's mailboxes on both servers.  I believe IMAPsync,
>https://imapsync.lamiral.info , has the ability to specify the admin
>login to use when migrating any mailbox.
>
>Yes: 

>I've not used that feature as I'm always migrating from not-cyrus to
>cyrus and keeping the passwords the same. 
>
>You should just need a CSV file mapping old username to new username
>and a looping shell script calling imapsync for each user.  With that
>much mail, you may want to split the CSV into several chunks and run
>the
>script in parallel with the different files to get more performance.
>The imapsync site has example scripts.


I did this often now and have two simple scripts for such a migration which 
works even for import from other IMAP servers. it uses simple text files with 
two columns (old/new box name) with tab or some spaces between - line by line 
(could easily adapted to other CSV formats). If it help's i can send it 
tomorrow...

This makes migrations much more easy, as just a common admin user (default 
cyrus) and his password is required for imapsync.

To get - from a users view - a "one to one" copy with correct dates/timestamps 
and status a few flags are typically required (i.e. recreate timestamps from 
headers).

You can speed up the transfer a bit with disabling SSL (if your net topo allows 
this).

imapsync by default only transfers differences / deltas between source and 
target mailbox. This makes it easy to i.e. let it run days before to transfer 
the very most of the content and start it again at user migration to let just 
clearify the (tylically less) newer modifications of the mailbox tree.

In practice this allows mailbox migrations with near zero downtimes for users.


hth
good luck and
best regards

Niels.

-- 
Niels Dettenbach
Syndicat IT & Internet
http://www.Syndicat.com

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailbox / user migration to a new server

[Scott Lambert]

On Tue, Jan 02, 2018 at 02:00:59PM +0100, Kristian Rink wrote:
> Folks;
> 
> we're in the process of moving an "old" (cyrus 2.2.13, local Unix users) 
> IMAP structure to a "newer" installation (cyrus 2.4.18, ActiveDirectory 
> users attached using winbind). In total I'm about to move somewhere next 
> to 300GB of mail spool knowing that ...
> 
> - ... all this data has to be moved from one server to another and
> 
> - ... most of the users now do have different user names (as the "old" 
> Unix names differ from the "new" AD names) so data needs to be copied 
> between different mailboxes.
> 
> So just copying /var/spool/cyrus won't possibly work. Likewise, however, 
> I can't use any of the IMAP migration scripts that do these things on a 
> per-user basis as I am not supposed to know or mess with each users 
> passwords; whatever I do sync-wise should be done using cyrus / 
> administrative user accounts.

With cyrus-imapd on both ends you shouldn't have to know the
users' passwords.  The cyrus admin user should be able to login
to the user's mailboxes on both servers.  I believe IMAPsync,
https://imapsync.lamiral.info , has the ability to specify the admin
login to use when migrating any mailbox.

Yes: 

https://imapsync.lamiral.info/FAQ.d/FAQ.Admin_Authentication.txt
https://imapsync.lamiral.info/FAQ.d/FAQ.General.txt

I've not used that feature as I'm always migrating from not-cyrus to
cyrus and keeping the passwords the same. 

You should just need a CSV file mapping old username to new username
and a looping shell script calling imapsync for each user.  With that
much mail, you may want to split the CSV into several chunks and run the
script in parallel with the different files to get more performance.
The imapsync site has example scripts.

The sync process is not speedy, but it works.  You can keep repeating
the process until it completes quickly enough for you to be happy with
the final cutover to new server time.  I ran the scripts continuously
for a week when I got task switched to something else.  It was ready for
the transition when I was.  

When you throw new mail delivery to the new server, you can keep running
the process without any delete2 options to ensure you don't miss
anything before you switch client IMAP activity to the new server.

Also you don't have to reconstruct each mailbox, unlike when you're done
moving data manually.  That process was painful during the upgrade from
2.3.x to 2.4.  I only had about 100GB of mail at the time.  Of course,
that was years ago and your disks are probably faster.

-- 
Scott LambertKC5MLE   Unix SysAdmin
lamb...@lambertfam.org

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailbox / user migration to a new server

[Kristian Rink]

Hi Sebastian;

Am 02.01.2018 um 14:12 schrieb Sebastian Hagedorn:
>
> You could copy the spool directories individually. You just need to
> create all the new directories first, and your copy script needs a
> mapping from the old user name to the new one. Then your script can use
> the "mbpath" utility to find the disk path to each user's directory.

thanks bunches for your feedback. So I'll see whether I'll get things 
done that way and get back here in case of any issues. :)


Best,
Kristian


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: mailbox / user migration to a new server

[Sebastian Hagedorn]

--On 2. Januar 2018 um 14:00:59 +0100 Kristian Rink  
wrote:



we're in the process of moving an "old" (cyrus 2.2.13, local Unix users)
IMAP structure to a "newer" installation (cyrus 2.4.18, ActiveDirectory
users attached using winbind). In total I'm about to move somewhere next
to 300GB of mail spool knowing that ...

- ... all this data has to be moved from one server to another and

- ... most of the users now do have different user names (as the "old"
Unix names differ from the "new" AD names) so data needs to be copied
between different mailboxes.

So just copying /var/spool/cyrus won't possibly work. Likewise, however,
I can't use any of the IMAP migration scripts that do these things on a
per-user basis as I am not supposed to know or mess with each users
passwords; whatever I do sync-wise should be done using cyrus /
administrative user accounts.

Is something like this possible? How if so? What is the best way of doing
this?


You could copy the spool directories individually. You just need to create 
all the new directories first, and your copy script needs a mapping from 
the old user name to the new one. Then your script can use the "mbpath" 
utility to find the disk path to each user's directory.

--
   .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
.:.Regionales Rechenzentrum (RRZK).:.
  .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.

pgpp5YOSmOsPu.pgp
Description: PGP signature

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus