Re: Need guidelines on how to migrate a Cyrus-Imapd server
On Fri, 1 Feb 2013, Thibault Le Meur wrote: I'll go the rsync way then... pity I would have loved to understand what kind of file is to be fed to the sync_client -u -f command, in order to give it a try.. It is just a list of userids, one per line, in a file rather than specified on the command line. -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Need guidelines on how to migrate a Cyrus-Imapd server
Le 01/02/2013 17:03, Thibault Le Meur a écrit : Thanks for the hints, I'll go the rsync way then... pity I would have loved to understand what kind of file is to be fed to the sync_client -u -f command, in order to give it a try.. Replying to myself, According to an old thread (http://lists.andrew.cmu.edu/pipermail/info-cyrus/2007-September/026772.html), the file format is: USER $username USER $username2 It is also confirmed that the only way to have singleinstancestore preserved is to run the sync_client with the -f option so that all synchs are done in the same run. However it seems that the cache which is used to detect the duplicates is rather low (UUID cache on the server side: 1000) so that single intance deduplication may not be very efficient. Unless this has changed, I agree that the best way to initialize the replica is to use rsync and then convert the databases. I'll give it a try. Regards, Thibault Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Need guidelines on how to migrate a Cyrus-Imapd server
On Mon, 2013-02-04 at 15:01 +0100, Thibault Le Meur wrote: Le 01/02/2013 17:03, Thibault Le Meur a écrit : Thanks for the hints I'll go the rsync way then... pity I would have loved to understand what kind of file is to be fed to the sync_client -u -f command, in order to give it a try.. Replying to myself, According to an old thread (http://lists.andrew.cmu.edu/pipermail/info-cyrus/2007-September/026772.html), the file format is: USER $username USER $username2 It is also confirmed that the only way to have singleinstancestore preserved is to run the sync_client with the -f option so that all synchs are done in the same run. However it seems that the cache which is used to detect the duplicates is rather low (UUID cache on the server side: 1000) so that single intance deduplication may not be very efficient. Unless this has changed, I agree that the best way to initialize the replica is to use rsync and then convert the databases. I'll give it a try. Correct, from a thread in 2007: quote Message UUIDs are used to replicate the single instance store (see docs/text/install-replication). This won't have much effect when you first replicate a mailstore as sync_server in 2.3 only tracks the last few thousand messages that have been uploaded. It becomes much more effective when a replica has been seeded and you switch to rolling replication. /quote quote sync_server maintains a fairly modest UUID cache on the server side: 1000 messages in 2.3. A restart is negotiated after each UPLOAD command. /quote It really does seem best to seed the replica, initially, via rsync [WATCH THOSE PERMISSIONS!] then to engage rolling replication - the replica should become current. Something like - rsync \ --verbose --recursive \ --perms --owner --group --times \ --links --hard-links --delete \ $master:$root $replica:$root I also like the --numeric-ids assuming your uidNumber/gidNumber is the same between systems. That saves a lot of pointless NSS calls. -- Adam Tauno Williams GPG D95ED383 Systems Administrator, Python Developer, LPI / NCLA Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Need guidelines on how to migrate a Cyrus-Imapd server
Le 04/02/2013 15:20, Adam Tauno Williams a écrit : Something like - rsync \ --verbose --recursive \ --perms --owner --group --times \ --links --hard-links --delete \ $master:$root $replica:$root I also like the --numeric-ids assuming your uidNumber/gidNumber is the same between systems. That saves a lot of pointless NSS calls. Thanks for the hint on the --numeric-ids option. Regards, Thibault Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Need guidelines on how to migrate a Cyrus-Imapd server
Hello, I'm trying to migrate a Cyrus-Imapd serveur running on an old machine to a new virtual machine environement and I'd like to have to guidelines on how to proceed. I've first thought about using the Cyrus-Imapd replication feature, but the first import using sync_client was causing he target server to use far more space on disk that the used space ont he original serveur. My guess is that the hardlinks dedeuplication was not preserved during this transfer. I don't know if this is due to the way I'v ran the tool or if this is a limitation of sync_client itself. Can someone answer me on this ? If sync_client is able to preserve hardlinks deduplication, what would be the correct way to run it (I've not found the syntax expected for the client-list-file in the command sync_client -u -f client-list-files so I used a for loop and ran sync_client indicidually on each user). Then, I've tried the rsync solution (synching the mailbox directory and config directory), but I'm switching to a 64bits OS so I'll have to convert the databases and import them. Is it a recommended solution ? Thanks in advance for your help, Thibault Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Need guidelines on how to migrate a Cyrus-Imapd server
On Fri, 2013-02-01 at 10:43 +0100, Thibault Le Meur wrote: I'm trying to migrate a Cyrus-Imapd serveur running on an old machine to a new virtual machine environement and I'd like to have to guidelines on how to proceed. I've first thought about using the Cyrus-Imapd replication feature, but the first import using sync_client was causing he target server to use far more space on disk that the used space ont he original serveur. My guess is that the hardlinks dedeuplication was not preserved during this transfer. That is easy to check. On the target does the listing of a mailbox look the same (on the filesystem) in terms of the reference count column? ... -rw--- 1 cyrus mail 2464 Feb 1 05:00 521985. -rw--- 1 cyrus mail 2629 Feb 1 05:00 521986. -rw--- 2 cyrus mail 4441 Feb 1 05:02 521987. -rw--- 1 cyrus mail 2834 Feb 1 05:03 521988. ... The second column greater than 1 means there is hard-linking occurring, and hard-links are the backbone of duplicate suppression. Are you sure that duplicate suppression was enabled on the replica? I don't know if this is due to the way I'v ran the tool or if this is a limitation of sync_client itself. Can someone answer me on this ? If sync_client is able to preserve hardlinks deduplication, what would be the correct way to run it (I've not found the syntax expected for the client-list-file in the command sync_client -u -f client-list-files so I used a for loop and ran sync_client indicidually on each user). I'm certain that is does support it; I see lots of duplicate suppression on my replica. I certainly didn't do anything special to make it replicate that way. I have duplicatesuppression: yes set on the replica. Then, I've tried the rsync solution (synching the mailbox directory and config directory), but I'm switching to a 64bits OS so I'll have to convert the databases and import them. Is it a recommended solution ? I believe that skiplist is 32bit/64bit neutral. So convert to skiplist first, on the working server. KILL BERKELEY!! You want to do that anyway. -- Adam Tauno Williams System Administrator, OpenGroupware Developer, LPI / CNA Fingerprint 8C08 209A FBE3 C41A DD2F A270 2D17 8FA4 D95E D383 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Need guidelines on how to migrate a Cyrus-Imapd server
Hi, Thanks for your answer. Le 01/02/2013 12:17, Adam Tauno Williams a écrit : On Fri, 2013-02-01 at 10:43 +0100, Thibault Le Meur wrote: Are you sure that duplicate suppression was enabled on the replica? Yes I'm sure, and a simple find have quickly confrimed this. find . -type f -a \! -links 1 -ls ... 141637005 32 -rw--- 12 cyrusmail30678 sept. 20 18:42 ./obfustcated-username/12307. ... I have duplicatesuppression: yes set on the replica. I thought that duplicatesuppression was a different thing. From the documentation Duplicate delivery supression is a feature of the Cyrus IMAP server that allows multiple copies of an identical messages delivered to one user to be supressed, so that the user only recieves one copy. What I'm trying to achieve is to keep the Single Instance Store property on my replica (http://cyrusimap.web.cmu.edu/docs/cyrus-imapd/2.4.7/overview.php#singleinstance) Do you confirm that the single instance property of Cyrus-Imapd is saved when using sync_client ? How have you done your initial synchronization with sync_client ? using sync_client -u -f USERFILE ? If yes, what is the format of USERFILE and how to build this list ? I've tried with a USERFILE containing a login per line with no success, sot I ended up using the following loop: for i in `cat /root/USERFILE`; do echo -n Synch $i: ; /usr/lib/cyrus-imapd/sync_client -u $i; if [ $? -eq 0 ]; then echo [OK]; else echo [ERREUR]; fi; done In this case, since I use a single sync_client process per user, it is logical that hardlinks between mailboxes can't be preserved. I believe that skiplist is 32bit/64bit neutral. So convert to skiplist first, on the working server. KILL BERKELEY!! You want to do that anyway. Yes it seems so. When you say Kill Berkeley, do you mean that the DB should saty in Skiplist format even on the production server (and not converted back to Berkeley)? If yes, do you have any URL describing the procedure ? Thanks in advance, Thibault Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Need guidelines on how to migrate a Cyrus-Imapd server
On Fri, 2013-02-01 at 13:20 +0100, Thibault Le Meur wrote: On Fri, 2013-02-01 at 10:43 +0100, Thibault Le Meur wrote: Are you sure that duplicate suppression was enabled on the replica? Yes I'm sure, and a simple find have quickly confrimed this. find . -type f -a \! -links 1 -ls .. 141637005 32 -rw--- 12 cyrusmail30678 sept. 20 18:42 ./obfustcated-username/12307. ... Then something is deduplicating. I have duplicatesuppression: yes set on the replica. I thought that duplicatesuppression was a different thing. Ah, yes, it is. We are talking about single-instance-store. That is the singleinstancestore directive. What I'm trying to achieve is to keep the Single Instance Store property on my replica (http://cyrusimap.web.cmu.edu/docs/cyrus-imapd/2.4.7/overview.php#singleinstance) Yep. In this case, since I use a single sync_client process per user, it is logical that hardlinks between mailboxes can't be preserved. True, I guess that makes sense.I believe I did an initial migration using rsync and they fired up the sync-server to keep it up to date / get the last changes. But it was some time ago. It certainly dedupp'ed going forward. Perhaps sync-ing across different versions may effect it as well. I believe that skiplist is 32bit/64bit neutral. So convert to skiplist first, on the working server. KILL BERKELEY!! You want to do that anyway. Yes it seems so. When you say Kill Berkeley, do you mean that the DB should saty in Skiplist format even on the production server (and not converted back to Berkeley)? If yes, do you have any URL describing the procedure ? Yes, production 2.4.x boxes should be using Skiplist. The cvt_cyrusdb will convert databases between formats. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Need guidelines on how to migrate a Cyrus-Imapd server
Le 01/02/2013 13:45, Adam Tauno Williams a écrit : On Fri, 2013-02-01 at 13:20 +0100, Thibault Le Meur wrote: True, I guess that makes sense. I believe I did an initial migration using rsync and they fired up the sync-server to keep it up to date / get the last changes. But it was some time ago. It certainly dedupp'ed going forward. Perhaps sync-ing across different versions may effect it as well. Thanks for the hints, I'll go the rsync way then... pity I would have loved to understand what kind of file is to be fed to the sync_client -u -f command, in order to give it a try.. Thanks again, Thibault Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
