Re: Various questions about databases (upgrade and migration)
--- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es www.sarenet.es [1] Antes de imprimir este correo electrónico piense si es necesario hacerlo. El 14-02-2019 15:46, Raphaël Halimi escribió: > Hi Egoitz, > > Thank you for your quick answer. > > Le 14/02/2019 à 14:46, Egoitz Aurrekoetxea a écrit : Now for the databases > themselves. In /var/lib/cyrus the global databases > were converted on-the-fly: > > # file /var/lib/cyrus/*.db > /var/lib/cyrus/annotations.db: Cyrus twoskip DB > /var/lib/cyrus/deliver.db: Cyrus twoskip DB > /var/lib/cyrus/mailboxes.db:Cyrus twoskip DB > /var/lib/cyrus/statuscache.db: Cyrus twoskip DB > /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB > /var/lib/cyrus/user_deny.db:empty > > However, the user databases were not converted: > > # file /var/lib/cyrus/user/*/* > /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB > /var/lib/cyrus/user/u/user1.sub: ASCII text > /var/lib/cyrus/user/u/user2.seen: Cyrus skiplist DB > /var/lib/cyrus/user/u/user2.sub:ASCII text > /var/lib/cyrus/user/u/user3.seen: Cyrus skiplist DB > /var/lib/cyrus/user/u/user3.sub:ASCII text > *Cyrus 2.4 converted databases on the fly. Cyrus 2.5 and newer don't. > You should launch a "reconstruct -r -V max" for that purpose.* > So my next questions are: why are the databases still in skiplist > format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they > should be twoskip ? Why didn't Cyrus convert them on-the-fly like the > global databases ? Do I have to manually do it myself ? And if I do > convert them, will it change anything (performance, reliability, etc > etc) ? > > *As said perhaps is a Debian derived config for the package. Yes you > should with the command above.* > Also, what about the various databases in the mail directories > (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file" > command only reports "data". What format are they actually in ? Do I > have to convert them too ? > > *Sure... Just launch a reconstruct -r -V max...* I just ran this command as cyrus user on my server (after reading the manual page). Unfortunately, the "seen" databases in /var/lib/cyrus/user are still reported by "file" as skiplist, and the "cyrus.cache", "cyrus.header" and "cyrus.index" in the various (sub)mailboxes, are still reported as simply "data". It did create "cyrus.annotations" databases in each subfolder, though (in twoskip format). Also,I'm a bit worried. I did see in the logs lines that said: repacking mailbox user... and reconstructing user... THOSE TWO LINES ARE PRETTY NORMAL... ...but also some more worrying lines that said: uniqueid clash with user... for - changing user... I HAVEN'T EVER SEEN IT CAN'T SAY... I SHOULD TAKE A LOT AT THE CODE FOR ANSWERING. Is it something I should worry about ? Regarding the fact that they're still not in the twoskip format, should I use cvt_cyrusdb instead ? That would be unfortunate, since I'll have to create a script fed to the "find" command to mass-convert all databases; plus, I still don't know what the input format (the "data" that file talks about) is. RECONSTRUCT -R -V MAX SHOULD HANDLE ALL CONVERSIONS... >> When I mill migrate, will I have to convert the databases through the >> flat format and back, or can I blindly copy the whole contents of >> /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to >> work out of the box ? >> >> *When doing such a migration, it would be better to setup a >> replication between the 2.4 and the new 2.5 in the hosting. You should >> encrypt that communication. You could use the own cyrus encription for >> replication or something like OpenVPN. Although it should work, I >> wouldn't copy directly (with an rsync or scp) the files.* Yes, both servers communicate through a VPN, but since both will have the same Cyrus version, I thought I could just copy the files. Why is it a bad idea ? IT'S NOT A HORRIBLE IDEA... BUT WITH THE REPLICATION YOU CREATE ALL INDEXES AND SO FROM 0, WITH THE CORRECT VERSION, WITHOUT NEEDING CONVERSIONS AND WITH THE PROPER VERSION AND, GIVES YOU CLEANER DATABASES... FOR INSTANCE... AND YOU COULD EVEN COPY CONSTANTLY CONTENT TO THE DATACENTER FOR JUST FAILING-OVER TO THE DATACENTER WHEN YOU ARE READY... MOVING TO THE NEW SERVER IS CLEANER AND EASIER... THAT'S WHAT I WOULD DO :) Regards, CHEERS! Links: -- [1] http://www.sarenet.es Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Various questions about databases (upgrade and migration)
Hi Egoitz, Thank you for your quick answer. Le 14/02/2019 à 14:46, Egoitz Aurrekoetxea a écrit : > Now for the databases themselves. In /var/lib/cyrus the global databases > were converted on-the-fly: > > # file /var/lib/cyrus/*.db > /var/lib/cyrus/annotations.db: Cyrus twoskip DB > /var/lib/cyrus/deliver.db: Cyrus twoskip DB > /var/lib/cyrus/mailboxes.db: Cyrus twoskip DB > /var/lib/cyrus/statuscache.db: Cyrus twoskip DB > /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB > /var/lib/cyrus/user_deny.db: empty > > However, the user databases were not converted: > > # file /var/lib/cyrus/user/*/* > /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB > /var/lib/cyrus/user/u/user1.sub: ASCII text > /var/lib/cyrus/user/u/user2.seen: Cyrus skiplist DB > /var/lib/cyrus/user/u/user2.sub: ASCII text > /var/lib/cyrus/user/u/user3.seen: Cyrus skiplist DB > /var/lib/cyrus/user/u/user3.sub: ASCII text > >> *Cyrus 2.4 converted databases on the fly. Cyrus 2.5 and newer don't. >> You should launch a "reconstruct -r -V max" for that purpose.* > > So my next questions are: why are the databases still in skiplist > format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they > should be twoskip ? Why didn't Cyrus convert them on-the-fly like the > global databases ? Do I have to manually do it myself ? And if I do > convert them, will it change anything (performance, reliability, etc > etc) ? > > *As said perhaps is a Debian derived config for the package. Yes you > should with the command above.* >> >> Also, what about the various databases in the mail directories >> (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file" >> command only reports "data". What format are they actually in ? Do I >> have to convert them too ? >> >> *Sure... Just launch a reconstruct -r -V max...* I just ran this command as cyrus user on my server (after reading the manual page). Unfortunately, the "seen" databases in /var/lib/cyrus/user are still reported by "file" as skiplist, and the "cyrus.cache", "cyrus.header" and "cyrus.index" in the various (sub)mailboxes, are still reported as simply "data". It did create "cyrus.annotations" databases in each subfolder, though (in twoskip format). Also,I'm a bit worried. I did see in the logs lines that said: repacking mailbox user... and reconstructing user... ...but also some more worrying lines that said: uniqueid clash with user... for - changing user... Is it something I should worry about ? Regarding the fact that they're still not in the twoskip format, should I use cvt_cyrusdb instead ? That would be unfortunate, since I'll have to create a script fed to the "find" command to mass-convert all databases; plus, I still don't know what the input format (the "data" that file talks about) is. >> When I mill migrate, will I have to convert the databases through the >> flat format and back, or can I blindly copy the whole contents of >> /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to >> work out of the box ? >> >> *When doing such a migration, it would be better to setup a >> replication between the 2.4 and the new 2.5 in the hosting. You should >> encrypt that communication. You could use the own cyrus encription for >> replication or something like OpenVPN. Although it should work, I >> wouldn't copy directly (with an rsync or scp) the files.* Yes, both servers communicate through a VPN, but since both will have the same Cyrus version, I thought I could just copy the files. Why is it a bad idea ? Regards, -- Raphaël Halimi signature.asc Description: OpenPGP digital signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Various questions about databases (upgrade and migration)
Hi Raphaël, Answering below in blue for instance.. --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnológico. Edificio 103 48170 Zamudio (Bizkaia) ego...@sarenet.es www.sarenet.es [1] Antes de imprimir este correo electrónico piense si es necesario hacerlo. El 14-02-2019 13:27, Raphaël Halimi escribió: > Hi, > > I have a small (3 users) Cyrus IMAP server that started as a very old > Debian (Woody or Sarge, I don't remember exactly) and that I upgraded > through the years, and yesterday I upgraded from Jessie to Stretch > (Cyrus 2.4.17 without caldav, to 2.5.10 with caldav). > > As I had some trouble in the past with databases migrations between > BerkeleyDB versions during some upgrades, and although Cyrus started > without any serious problem this time (after minor tuning of some > options in the configuration files), I still wanted to check the state > of the databases. > > So, the contents of /usr/lib/cyrus/cyrus-db-types.txt is: > > ANNOTATION twoskip > DBENGINE BerkeleyDB5.3 > DUPLICATE twoskip > MBOX twoskip > PTS twoskip > QUOTA quotalegacy > SEEN twoskip > STATUSCACHE twoskip > SUBS flat > TLS DEFAULT > TLS twoskip > USERDENY flat > ZONEINFO twoskip > > (note that in Debian, no format is forced in /etc/imapd.conf) > > So my first questions is: I thought that Cyrus had abandoned BerkeleyDB > a long time ago for skiplist, and then twoskip. Why is it still listed > in this file ? Is there a part of Cyrus that still uses it ? > > I SUPPOSE IT COULD SOME DEBIAN STANDARD CONFIG?. WE RUN FREEBSD AND I HAVE > NEVER SEE THAT FILE. > > Now for the databases themselves. In /var/lib/cyrus the global databases > were converted on-the-fly: > > # file /var/lib/cyrus/*.db > /var/lib/cyrus/annotations.db: Cyrus twoskip DB > /var/lib/cyrus/deliver.db: Cyrus twoskip DB > /var/lib/cyrus/mailboxes.db:Cyrus twoskip DB > /var/lib/cyrus/statuscache.db: Cyrus twoskip DB > /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB > /var/lib/cyrus/user_deny.db:empty > > However, the user databases were not converted: > > # file /var/lib/cyrus/user/*/* > /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB > /var/lib/cyrus/user/u/user1.sub: ASCII text > /var/lib/cyrus/user/u/user2.seen: Cyrus skiplist DB > /var/lib/cyrus/user/u/user2.sub:ASCII text > /var/lib/cyrus/user/u/user3.seen: Cyrus skiplist DB > /var/lib/cyrus/user/u/user3.sub:ASCII text > > CYRUS 2.4 CONVERTED DATABASES ON THE FLY. CYRUS 2.5 AND NEWER DON'T. YOU > SHOULD LAUNCH A "RECONSTRUCT -R -V MAX" FOR THAT PURPOSE. > > So my next questions are: why are the databases still in skiplist > format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they > should be twoskip ? Why didn't Cyrus convert them on-the-fly like the > global databases ? Do I have to manually do it myself ? And if I do > convert them, will it change anything (performance, reliability, etc etc) ? > > AS SAID PERHAPS IS A DEBIAN DERIVED CONFIG FOR THE PACKAGE. YES YOU SHOULD > WITH THE COMMAND ABOVE. > > Also, what about the various databases in the mail directories > (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file" > command only reports "data". What format are they actually in ? Do I > have to convert them too ? > > SURE... JUST LAUNCH A RECONSTRUCT -R -V MAX... > > My last question is about a planned migration of this home server to a > hosted private server. This old server and the new one are now both > Debian Stretch and thus, have the same Cyrus version, but they're not > the same architecture: the old one is i386, and the new one is amd64. > > When I mill migrate, will I have to convert the databases through the > flat format and back, or can I blindly copy the whole contents of > /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to > work out of the box ? > > WHEN DOING SUCH A MIGRATION, IT WOULD BE BETTER TO SETUP A REPLICATION > BETWEEN THE 2.4 AND THE NEW 2.5 IN THE HOSTING. YOU SHOULD ENCRYPT THAT > COMMUNICATION. YOU COULD USE THE OWN CYRUS ENCRIPTION FOR REPLICATION OR > SOMETHING LIKE OPENVPN. ALTHOUGH IT SHOULD WORK, I WOULDN'T COPY DIRECTLY > (WITH AN RSYNC OR SCP) THE FILES. > > Thanks a lot in advance for your answers. > > Regards, > > CHEERS! > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: -- [1] http://www.sarenet.es Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Various questions about databases (upgrade and migration)
Hi, I have a small (3 users) Cyrus IMAP server that started as a very old Debian (Woody or Sarge, I don't remember exactly) and that I upgraded through the years, and yesterday I upgraded from Jessie to Stretch (Cyrus 2.4.17 without caldav, to 2.5.10 with caldav). As I had some trouble in the past with databases migrations between BerkeleyDB versions during some upgrades, and although Cyrus started without any serious problem this time (after minor tuning of some options in the configuration files), I still wanted to check the state of the databases. So, the contents of /usr/lib/cyrus/cyrus-db-types.txt is: ANNOTATION twoskip DBENGINE BerkeleyDB5.3 DUPLICATE twoskip MBOX twoskip PTS twoskip QUOTA quotalegacy SEEN twoskip STATUSCACHE twoskip SUBS flat TLS DEFAULT TLS twoskip USERDENY flat ZONEINFO twoskip (note that in Debian, no format is forced in /etc/imapd.conf) So my first questions is: I thought that Cyrus had abandoned BerkeleyDB a long time ago for skiplist, and then twoskip. Why is it still listed in this file ? Is there a part of Cyrus that still uses it ? Now for the databases themselves. In /var/lib/cyrus the global databases were converted on-the-fly: # file /var/lib/cyrus/*.db /var/lib/cyrus/annotations.db: Cyrus twoskip DB /var/lib/cyrus/deliver.db: Cyrus twoskip DB /var/lib/cyrus/mailboxes.db:Cyrus twoskip DB /var/lib/cyrus/statuscache.db: Cyrus twoskip DB /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB /var/lib/cyrus/user_deny.db:empty However, the user databases were not converted: # file /var/lib/cyrus/user/*/* /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB /var/lib/cyrus/user/u/user1.sub: ASCII text /var/lib/cyrus/user/u/user2.seen: Cyrus skiplist DB /var/lib/cyrus/user/u/user2.sub:ASCII text /var/lib/cyrus/user/u/user3.seen: Cyrus skiplist DB /var/lib/cyrus/user/u/user3.sub:ASCII text So my next questions are: why are the databases still in skiplist format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they should be twoskip ? Why didn't Cyrus convert them on-the-fly like the global databases ? Do I have to manually do it myself ? And if I do convert them, will it change anything (performance, reliability, etc etc) ? Also, what about the various databases in the mail directories (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file" command only reports "data". What format are they actually in ? Do I have to convert them too ? My last question is about a planned migration of this home server to a hosted private server. This old server and the new one are now both Debian Stretch and thus, have the same Cyrus version, but they're not the same architecture: the old one is i386, and the new one is amd64. When I mill migrate, will I have to convert the databases through the flat format and back, or can I blindly copy the whole contents of /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to work out of the box ? Thanks a lot in advance for your answers. Regards, -- Raphaël Halimi signature.asc Description: OpenPGP digital signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Upgrade and Migration
Carson Gaspar wrote: Ben Carter wrote: Carson Gaspar wrote: Ben Carter wrote: You have to do the _exact_ same thing with imapsync, unless you want to lose email. As has already been pointed out, you are incorrect. The order is: [Pre-create inboxes with large quotas on new server] 1. Shut off mail delivery to old server 2. Shut off imapd on old server. 3. Bring up new server with imapd running 4. Start mail delivery to new server. 5. Start imapd on old server with a new IP address or bound to a nonstandard port so MUAs will not get to it. 6. Migrate data using imapsync. And your service can be down literally for only a few minutes if you plan correctly. You have chosen to have users see missing data. You have made a tradeoff You can't see missing data ;) between correctness and length of downtime by bringing up your new server in a client visible way before you sync your data. I (foolishly, it seems) assumed that you wouldn't want your users to log in to empty mailboxes, and would leave your new server non-user-visible until your sync completed. Yes, that's the trade-off, obviously. We forewarned our users a week or 2 ahead of time though, and let's face it: a lot of retained e-mail just spins around out there, never to be accessed again. This is something done on the weekend and/or at night too, remember. Also, in addition to being forewarned, each user got a start-of-migration message and an end-of-migration message so they knew exactly what was happening and they knew when their migration was complete. So, migrating like this, the users can send and receive mail with almost no interruption, and ours seemed to be quite happy with this. We didn't hear a peep, and we have a lot of users. I think that it's much more important to them to be able to send and receive e-mail without interruption than to have their old mail present immediately. And, their old mail shows up just as it was in a relatively short period of time. Ben -- Ben Carter University of Pittsburgh/CSSD b...@pitt.edu 412-624-6470 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
Carson Gaspar wrote: Ben Carter wrote: If you use rsync, you have to stop everything until that finishes, possibly reconstruct all mailboxes, maybe fix some other things before giving people their mail functionality back and allowing mail delivery to resume. That's just silly. If you're going to use rsync to migrate data, you do at least one rsync while the source data is live. More than one if the initial sync takes a long time. Then you go offline, do a final sync (which should be very fast), and bring the new data store online. The longer you wait to upgrade, the less silly it gets. In the case of a move to completely new servers, doing a migration using the protocol can be much more appealing. Also, we did what you suggest with rsync on our previous upgrade and we did an rsync every night leading up to the upgrade night, so the last rsync copied only a day's worth of changes. The last rsync took hours even though we ran multiple rsyncs concurrently. I think it may have taken as long as 6-8 hours, but I don't remember the exact timing. You have to do the _exact_ same thing with imapsync, unless you want to lose email. As has already been pointed out, you are incorrect. The order is: [Pre-create inboxes with large quotas on new server] 1. Shut off mail delivery to old server 2. Shut off imapd on old server. 3. Bring up new server with imapd running 4. Start mail delivery to new server. 5. Start imapd on old server with a new IP address or bound to a nonstandard port so MUAs will not get to it. 6. Migrate data using imapsync. And your service can be down literally for only a few minutes if you plan correctly. Also, the ACL format in the mailboxes file might be different between these 2 Cyrus versions. Might be, but I don't think it is. If you use the protocol to move the data, you don't have to worry about any data structure differences etc. You also can re-arrange your partitions and so on. Plus it re-calculates all quota usage as imapsync APPENDs the messages during the migration. All true, except to the best of my knowledge none of this (except repartitioning, which the OP didn't mention) matters for cyrus imapd - it will Just Work(tm) on your old data store. The only exceptions are database format changes (if you use bdb and you've revved the library version, for example), and sieve compiled bytecode. Yes, but that's the point: you can fix all those things that you always wanted to fix. Spin out the mailboxes evenly again across partitions, change DB formats, use fulldirhash, go to fewer, larger partitions, leave unused mailboxes behind, etc. Not to mention that the years of RENAME operations you did to move mailboxes left garbage behind that rsync would blindly copy over. And why do you care about quota re-calculation? The existing quota data should be correct. In our case, the problem was that the old code used 32-bit integers to track quota/usage so we had to have a cron job that zeroed usage on the old server for large mailboxes every once in a while. Ben -- Ben Carter University of Pittsburgh/CSSD b...@pitt.edu 412-624-6470 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
Garry wrote: Simon Matter wrote: I think an easier way is to use the 'proxyservers' option to declare a proxy user allowed to access all mailboxes. This user can then be used by imapsync. I'm quite sure the archives hold detailed information on this as I don't remember all the details. Cool! Works, at least accessing the user mail directories ... guess I will still have to figure out the best way of transferring all the access rights to the new server, but this should at least get the mails copied more easily ... -garry Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Yes, the proxy login worked for us. If by access rights you mean IMAP ACLs, the imapsync option for this is --syncacls. Ben -- Ben Carter University of Pittsburgh/CSSD b...@pitt.edu 412-624-6470 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
Ben Carter wrote: Carson Gaspar wrote: Ben Carter wrote: You have to do the _exact_ same thing with imapsync, unless you want to lose email. As has already been pointed out, you are incorrect. The order is: [Pre-create inboxes with large quotas on new server] 1. Shut off mail delivery to old server 2. Shut off imapd on old server. 3. Bring up new server with imapd running 4. Start mail delivery to new server. 5. Start imapd on old server with a new IP address or bound to a nonstandard port so MUAs will not get to it. 6. Migrate data using imapsync. And your service can be down literally for only a few minutes if you plan correctly. You have chosen to have users see missing data. You have made a tradeoff between correctness and length of downtime by bringing up your new server in a client visible way before you sync your data. I (foolishly, it seems) assumed that you wouldn't want your users to log in to empty mailboxes, and would leave your new server non-user-visible until your sync completed. -- Carson Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
Carson Gaspar wrote: Ben Carter wrote: If you use rsync, you have to stop everything until that finishes, possibly reconstruct all mailboxes, maybe fix some other things before giving people their mail functionality back and allowing mail delivery to resume. That's just silly. If you're going to use rsync to migrate data, you do at least one rsync while the source data is live. More than one if the initial sync takes a long time. Then you go offline, do a final sync (which should be very fast), and bring the new data store online. Running rsync prior to shutting down so it only has to copy incremental changes once you shut down will be faster than not doing so, but calling stat() for millions of files may not be very fast. If you're not worried about the duration of your downtime, this doesn't matter to you. You have to do the _exact_ same thing with imapsync, unless you want to lose email. Not true. Read the original response again. No downtime was incurred and no mail was lost using imapsync. Also, the ACL format in the mailboxes file might be different between these 2 Cyrus versions. Might be, but I don't think it is. As of the 2.3 code, Cyrus supports rfc4314 acls. That said, I believe the 2.3.14 code will do the right thing if it encounters the old-style acls. If you use the protocol to move the data, you don't have to worry about any data structure differences etc. You also can re-arrange your partitions and so on. Plus it re-calculates all quota usage as imapsync APPENDs the messages during the migration. All true, except to the best of my knowledge none of this (except repartitioning, which the OP didn't mention) matters for cyrus imapd - it will Just Work(tm) on your old data store. The only exceptions are database format changes (if you use bdb and you've revved the library version, for example), and sieve compiled bytecode. Standard advice, with Cyrus being a sealed-server design, is usually to use IMAP protocol to accomplish whatever it is you're trying to do and to not muck with what's in the filesystem. In this case, imapsync would do everything via protocol so you don't have to muck around in the filesystem. Regarding the original question, however, what you're proposing to do with rsync should work with some caveats. As Carson mentioned above, if you have a different version of bdb on the new machine, that could give you headaches. If your new machine is 64 bit and the old machine was 32 bit, I think that could also cause you problems. Also, check your imapd.conf to make sure you have all the correct database formats specified since you're copying over the old imapd.conf to the new server and you're changing formats on the new server. Rather than trust me or anyone else who tells you this should just work, you should test it first. If it causes you problems, try imapsync. And why do you care about quota re-calculation? The existing quota data should be correct. Technically, quota is calculated differently in 2.3.14 than it was in 2.1.16. At the very least, it now ignores things that aren't in cyrus.index when calculating quota and it didn't used to do that. hth, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
I've been following this thread, I will be doing an IMAP migration from an older to a more current version (albeit, not the newest). I was wondering - is imapsync (or IMAP for that matter) able to copy all the folders, permissions etc. by using the cyrus admin user instead of all the separate users? That would save a lot of work ... Tnx, -garry Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
On 07/04/2009 03:04 PM, Garry Glendown wrote: I've been following this thread, I will be doing an IMAP migration from an older to a more current version (albeit, not the newest). I was wondering - is imapsync (or IMAP for that matter) able to copy all the folders, permissions etc. by using the cyrus admin user instead of all the separate users? That would save a lot of work ... Yes, it is, if you first grant the admin account all rights to all mailboxes, which is kind of a pain. My preferred approach, if you can afford to have the mailstore off line for the duration, is to simply bulk-change the passwords to some known value prior to commencing the transfer. For example, if you are using unix system passwords, change the /etc/shadow (or whatever) file. If you are using LDAP change the userPassword attributes, etc. You can stash away a copy of the originals and restore them later. Then I would run a script, written in your favorite scripting language, to walk through a list of users and initiate the transfers. You can run multiple transfers at once, just keep an eye on your I/O loads to make sure you aren't just loading down the systems too much. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
On 07/04/2009 03:04 PM, Garry Glendown wrote: I've been following this thread, I will be doing an IMAP migration from an older to a more current version (albeit, not the newest). I was wondering - is imapsync (or IMAP for that matter) able to copy all the folders, permissions etc. by using the cyrus admin user instead of all the separate users? That would save a lot of work ... Yes, it is, if you first grant the admin account all rights to all mailboxes, which is kind of a pain. My preferred approach, if you can afford to have the mailstore off line for the duration, is to simply bulk-change the passwords to some known value prior to commencing the transfer. For example, if you are using unix system passwords, change the /etc/shadow (or whatever) file. If you are using LDAP change the userPassword attributes, etc. You can stash away a copy of the originals and restore them later. Then I would run a script, written in your favorite scripting language, to walk through a list of users and initiate the transfers. You can run multiple transfers at once, just keep an eye on your I/O loads to make sure you aren't just loading down the systems too much. I think an easier way is to use the 'proxyservers' option to declare a proxy user allowed to access all mailboxes. This user can then be used by imapsync. I'm quite sure the archives hold detailed information on this as I don't remember all the details. Regards, Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
Simon Matter wrote: I think an easier way is to use the 'proxyservers' option to declare a proxy user allowed to access all mailboxes. This user can then be used by imapsync. I'm quite sure the archives hold detailed information on this as I don't remember all the details. Cool! Works, at least accessing the user mail directories ... guess I will still have to figure out the best way of transferring all the access rights to the new server, but this should at least get the mails copied more easily ... -garry Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
Ben Carter wrote: If you use rsync, you have to stop everything until that finishes, possibly reconstruct all mailboxes, maybe fix some other things before giving people their mail functionality back and allowing mail delivery to resume. That's just silly. If you're going to use rsync to migrate data, you do at least one rsync while the source data is live. More than one if the initial sync takes a long time. Then you go offline, do a final sync (which should be very fast), and bring the new data store online. You have to do the _exact_ same thing with imapsync, unless you want to lose email. Also, the ACL format in the mailboxes file might be different between these 2 Cyrus versions. Might be, but I don't think it is. If you use the protocol to move the data, you don't have to worry about any data structure differences etc. You also can re-arrange your partitions and so on. Plus it re-calculates all quota usage as imapsync APPENDs the messages during the migration. All true, except to the best of my knowledge none of this (except repartitioning, which the OP didn't mention) matters for cyrus imapd - it will Just Work(tm) on your old data store. The only exceptions are database format changes (if you use bdb and you've revved the library version, for example), and sieve compiled bytecode. And why do you care about quota re-calculation? The existing quota data should be correct. -- Carson Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Upgrade and Migration
Hello,
I have an old cyrus install (2.1.16) with 30GB of data that I need to
migrate to a new server. My challenge is that something on that server
is screwed up and I have no access to PERL on that machine anymore (the
OS is past EOL, so it is long past time to move anyways). I have a new
box (2.3.14) and a bunch of trepidation about screwing this move up. I
have looked at the Cyrus Upgrade doc and seemingly this is a rather
straight forward operation.
Does this make sense?
1.rsync (pulling from the old server to the new):
config directory (/var/imap)
partition-default (/var/spool/imap)
2. /usr/cyrus/bin/cvt_cyrusdb /var/imap/mailboxes.db berkeley
/var/imap/mailboxes.db.new skiplist
3. mv /var/imap/mailboxes.db.new /var/imap/mailboxes.db
4. find /var/imap/user -name \*.seen -exec /usr/cyrus/bin/cvt_cyrusdb
\{\} flat \{\}.new skiplist \; -exec mv \{\}.new \{\} \;
5. start imapd
Is that it? I just want to make sure I am not missing anything.
Thank you,
Peter
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Upgrade and Migration
Peter Clark wrote:
Hello,
I have an old cyrus install (2.1.16) with 30GB of data that I need to
migrate to a new server. My challenge is that something on that server
is screwed up and I have no access to PERL on that machine anymore (the
OS is past EOL, so it is long past time to move anyways). I have a new
box (2.3.14) and a bunch of trepidation about screwing this move up. I
have looked at the Cyrus Upgrade doc and seemingly this is a rather
straight forward operation.
Does this make sense?
1.rsync (pulling from the old server to the new):
config directory (/var/imap)
partition-default (/var/spool/imap)
2. /usr/cyrus/bin/cvt_cyrusdb /var/imap/mailboxes.db berkeley
/var/imap/mailboxes.db.new skiplist
3. mv /var/imap/mailboxes.db.new /var/imap/mailboxes.db
4. find /var/imap/user -name \*.seen -exec /usr/cyrus/bin/cvt_cyrusdb
\{\} flat \{\}.new skiplist \; -exec mv \{\}.new \{\} \;
5. start imapd
Is that it? I just want to make sure I am not missing anything.
Thank you,
Peter
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Peter,
You might want to use imapsync to do this. It uses the IMAP protocol,
copies flags, can copy ACLs etc. What I did was to create all the
INBOXes on the new server, set up large quotas on the new server,
point mail delivery to the new server and then use imapsync to copy
the folders and messages. People had full mail functionality right
away and their folders/messages were backfilled.
At the end, I set quotas back to what I really wanted them to be.
If you use rsync, you have to stop everything until that finishes,
possibly reconstruct all mailboxes, maybe fix some other things before
giving people their mail functionality back and allowing mail delivery
to resume.
Also, the ACL format in the mailboxes file might be different between
these 2 Cyrus versions.
If you use the protocol to move the data, you don't have to worry
about any data structure differences etc. You also can re-arrange
your partitions and so on. Plus it re-calculates all quota usage as
imapsync APPENDs the messages during the migration.
You'll have to enable proxy logins on both IMAP servers to do this
administratively with imapsync.
I copied 1.2TB, ~65,000 users in ~28 hours by using up to 128
concurrent imapsync processes at a time this way.
Ben
--
Ben Carter
University of Pittsburgh/CSSD
b...@pitt.edu
412-624-6470
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
