Re: Migrating 32bit to 64bit Debian Lenny
Did you install cyrus on the new server before or after the steps described below? It sounds as if it was before, just by the references to shutting it down on the new server and to /usr/lib/cyrus/cyrus-db-types.active. Also, do you have any sense of the speed and accuracy of this method as opposed to imapsync? Ross Boylan On Wed, 2009-09-02 at 09:19 -0600, Bill Cameron wrote: Hi, I've seen some questions about migrating from 32 bit to 64 bit. We migrated last weekend and these are the steps we took. Old server: - rsync the cyrus data while cyrus is running rsync -vaH --delete -e ssh /var/lib/cyrus/ new-server:/var/lib/cyrus rsync -vaH --delete -e ssh /var/spool/cyrus/ new-server:/var/spool/cyrus rsync -vaH --delete -e ssh /var/spool/sieve/ new-server:/var/spool/sieve - shut down cyrus - repeat rsyncing of the three directories to provide stable environment and databases. This will be a lot faster than the original rsync. - dump /var/lib/cyrus/mailboxes.db to a text file /usr/sbin/ctl_mboxlist -d mboxlist.txt - copy this text file to the new server New server: - make sure cyrus is shutdown - switch to user 'cyrus' su - cyrus - remove some of the databases rm /var/lib/cyrus/db/* rm /var/lib/cyrus/db.backup1/* rm /var/lib/cyrus/db.backup2/* rm /var/lib/cyrus/deliver.db rm /var/lib/cyrus/tls_sessions.db rm /var/lib/cyrus/mailboxes.db - build new mailboxes.db from mboxlist.txt file /usr/sbin/ctl_mboxlist -u mboxlist.txt - check /var/log/mail.err and /var/log/mail.info for any errors from the above command and the following commands. There should only be one error about missing timestamp file but it is automatically created. - run the following commands and check logs for errors /usr/sbin/ctl_cyrusdb -r - the above command will verify mailboxes.db and annotations.db /usr/sbin/tls_prune - the above command will create a new tls_prune database /usr/sbin/ctl_cyrusdb -c /usr/sbin/cyr_expire -E 3 - you can also run the squatter command but it doesn't really need to run until it's scheduled time. It takes a awhile to run. - start cyrus and check that it is working correctly. - you will need to reset any annotations (e.g.: expire) on mailboxes/folders. We didn't have any annotations set on mailboxes so I didn't try migrating that database. The database types are defined in /etc/imapd.conf. They don't appear in the default Lenny conf file since they use the predefined default types. The man page for imapd.conf lists those defaults: annotation_db: skiplist, duplicate_db: berkeley-nosync... The current database types are listed in /usr/lib/cyrus/cyrus-db-types.active. We had to take the server off-line to migrate some other applications on the server so I didn't use imapsync. We are using imapsync to migrate from Lotus Notes to cyrus. Bill C. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Migrating 32bit to 64bit Debian Lenny
On Mon, Sep 7, 2009 at 3:34 PM, Ross Boylanr...@biostat.ucsf.edu wrote: Did you install cyrus on the new server before or after the steps described below? It sounds as if it was before, just by the references to shutting it down on the new server and to /usr/lib/cyrus/cyrus-db-types.active. I installed the cyrus packages before transferring any data between servers. I then shut down cyrus on the new server and started rsyncing the data. Also, do you have any sense of the speed and accuracy of this method as opposed to imapsync? I've only used imapsync for moving mailboxes from Lotus Domino to cyrus. Domino is slow serving imap and it took longer to imapsync the same mailbox from Domino to cyrus then rsync it from 32 bit to 64 bit cyrus machines. I would think that imapsync would be slower because of all the imap communication overhead. The advantage of imapsync is that you don't have to shutdown either system. Bill C. Ross Boylan Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Migrating 32bit to 64bit Debian Lenny
Hi, I've seen some questions about migrating from 32 bit to 64 bit. We migrated last weekend and these are the steps we took. Old server: - rsync the cyrus data while cyrus is running rsync -vaH --delete -e ssh /var/lib/cyrus/ new-server:/var/lib/cyrus rsync -vaH --delete -e ssh /var/spool/cyrus/ new-server:/var/spool/cyrus rsync -vaH --delete -e ssh /var/spool/sieve/ new-server:/var/spool/sieve - shut down cyrus - repeat rsyncing of the three directories to provide stable environment and databases. This will be a lot faster than the original rsync. - dump /var/lib/cyrus/mailboxes.db to a text file /usr/sbin/ctl_mboxlist -d mboxlist.txt - copy this text file to the new server New server: - make sure cyrus is shutdown - switch to user 'cyrus' su - cyrus - remove some of the databases rm /var/lib/cyrus/db/* rm /var/lib/cyrus/db.backup1/* rm /var/lib/cyrus/db.backup2/* rm /var/lib/cyrus/deliver.db rm /var/lib/cyrus/tls_sessions.db rm /var/lib/cyrus/mailboxes.db - build new mailboxes.db from mboxlist.txt file /usr/sbin/ctl_mboxlist -u mboxlist.txt - check /var/log/mail.err and /var/log/mail.info for any errors from the above command and the following commands. There should only be one error about missing timestamp file but it is automatically created. - run the following commands and check logs for errors /usr/sbin/ctl_cyrusdb -r - the above command will verify mailboxes.db and annotations.db /usr/sbin/tls_prune - the above command will create a new tls_prune database /usr/sbin/ctl_cyrusdb -c /usr/sbin/cyr_expire -E 3 - you can also run the squatter command but it doesn't really need to run until it's scheduled time. It takes a awhile to run. - start cyrus and check that it is working correctly. - you will need to reset any annotations (e.g.: expire) on mailboxes/folders. We didn't have any annotations set on mailboxes so I didn't try migrating that database. The database types are defined in /etc/imapd.conf. They don't appear in the default Lenny conf file since they use the predefined default types. The man page for imapd.conf lists those defaults: annotation_db: skiplist, duplicate_db: berkeley-nosync... The current database types are listed in /usr/lib/cyrus/cyrus-db-types.active. We had to take the server off-line to migrate some other applications on the server so I didn't use imapsync. We are using imapsync to migrate from Lotus Notes to cyrus. Bill C. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Migrating 32bit to 64bit Debian Lenny
Hi, I've seen some questions about migrating from 32 bit to 64 bit. We Hi, I understand that you also migrated to a newer OS and cyrus-imapd version, right? migrated last weekend and these are the steps we took. Old server: - rsync the cyrus data while cyrus is running rsync -vaH --delete -e ssh /var/lib/cyrus/ new-server:/var/lib/cyrus rsync -vaH --delete -e ssh /var/spool/cyrus/ new-server:/var/spool/cyrus rsync -vaH --delete -e ssh /var/spool/sieve/ new-server:/var/spool/sieve - shut down cyrus - repeat rsyncing of the three directories to provide stable environment and databases. This will be a lot faster than the original rsync. - dump /var/lib/cyrus/mailboxes.db to a text file /usr/sbin/ctl_mboxlist -d mboxlist.txt - copy this text file to the new server New server: - make sure cyrus is shutdown - switch to user 'cyrus' su - cyrus - remove some of the databases rm /var/lib/cyrus/db/* rm /var/lib/cyrus/db.backup1/* rm /var/lib/cyrus/db.backup2/* rm /var/lib/cyrus/deliver.db rm /var/lib/cyrus/tls_sessions.db rm /var/lib/cyrus/mailboxes.db - build new mailboxes.db from mboxlist.txt file /usr/sbin/ctl_mboxlist -u mboxlist.txt - check /var/log/mail.err and /var/log/mail.info for any errors from the above command and the following commands. There should only be one error about missing timestamp file but it is automatically created. - run the following commands and check logs for errors /usr/sbin/ctl_cyrusdb -r - the above command will verify mailboxes.db and annotations.db /usr/sbin/tls_prune - the above command will create a new tls_prune database /usr/sbin/ctl_cyrusdb -c /usr/sbin/cyr_expire -E 3 - you can also run the squatter command but it doesn't really need to run until it's scheduled time. It takes a awhile to run. - start cyrus and check that it is working correctly. - you will need to reset any annotations (e.g.: expire) on mailboxes/folders. We didn't have any annotations set on mailboxes so I didn't try migrating that database. The database types are defined in /etc/imapd.conf. They don't appear in the default Lenny conf file since they use the predefined default types. The man page for imapd.conf lists those defaults: annotation_db: skiplist, duplicate_db: berkeley-nosync... The current database types are listed in /usr/lib/cyrus/cyrus-db-types.active. I'm wondering how much of all this was really needed for the migration from 32bit to 64bit? Are the BerkeleyDB ondisk files different on 32/64bit? Because, the last migration I did was from RHEL3/32bit to RHEL5/64bit using our own cyrus-imapd RPMs and the migration was as easy as stopping cyrus-imapd, rsyncing /var/lib/imap + /var/spool/imap to the new box and starting cyrus-imapd on the new box. Now, the RPMs do quite some magic on the database files to make sure they are all skiplist after shutdown, and switched back on startup. So my question remains, what parts of cyrus-imapd are possibly arch dependant? Regards, Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Migrating 32bit to 64bit Debian Lenny
Simon Matter wrote: I'm wondering how much of all this was really needed for the migration from 32bit to 64bit? Are the BerkeleyDB ondisk files different on 32/64bit? Yes they are. It's not the OS that matters but the architecture of the libdb4.so file. It is still a good idea not to use Berkeley DB for real important data. Here at our university's cyrus we are using Berkeley for the duplicate delivery and the tls databases - both of them are easily set to zero in case of problems without deep impact on the functionality (in case the delivery db crashes users can get some mails two times (doubling), in the latter case (tls db crash) a returning client has to re-initiate a TLS handshake including key exchange). Pascal Gienger -- Pascal Gienger University of Konstanz, IT Services Department (Rechenzentrum) Electronic Communications and Web Services Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Migrating 32bit to 64bit Debian Lenny
I'm wondering how much of all this was really needed for the migration from 32bit to 64bit? Are the BerkeleyDB ondisk files different on 32/64bit? I initially tried just using rsync with cyrus shutdown on both servers but cyrus failed to start on the new server due to db errors. As mentioned this is due to the architecture of the libdb4.so file. Further testing showed which files worked and which ones didn't. Bill C. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Migrating 32bit to 64bit Debian Lenny
On Wed, Sep 02, 2009 at 10:46:35AM -0600, Bill Cameron wrote: I'm wondering how much of all this was really needed for the migration from 32bit to 64bit? Are the BerkeleyDB ondisk files different on 32/64bit? I initially tried just using rsync with cyrus shutdown on both servers but cyrus failed to start on the new server due to db errors. As mentioned this is due to the architecture of the libdb4.so file. Further testing showed which files worked and which ones didn't. Thankfully skiplist is immune to such nonsense and just keeps working across any architecture or underlying version change (apart from a couple of bugs that are only fixed in CVS at the moment - *sigh*) We've recently switched to using skiplist even for deliver and TLS databases. It works fine. Possibly not quite as efficient, but we haven't noticed a difference in the load on the machines. Though - I do wonder how many problems we've had with BDB were caused by idled forking with an environment reference and then closing it in the parent, causing the reference count to be incorrect. There's a patch for that in CVS too. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
