Re: IMAP not seeing old mail present on filesystem
On 10/4/10 9:17 AM, Simon Matter wrote: Simon, I did recover /var/lib/imap (although a bit later, FWIW) and I think a bit later is your problem. But I think reconstruct should fix this. BTW, did you check the subscription status of mailboxes? Maybe the folders are there but not subscribed, and that's why some users can't see them? Simon, No, users see the folders, just not old messages. For most (all?) INBOXes but my own, new messages started arriving as 1. and continued from there. Users can see the new mail, but not the old. This makes me think it's not an internal permissions problem, because they see the mailboxes and (some) mail in them. All file permissions I checked appear correct reconstruct -rfx doesn't help. Is there anything else to try? Thanks, Chris /etc/imapd.conf. I just now compared /etc/cyrus.conf and added squatter adjusted prefork numbers -- the rest all matched. Is there other configuration I should check? Both systems are 64-bit CentOS 5, so the db4 installations should be fully compatible. If both the old and the new system are basically the same I don't think there is anything missing. Simon Thanks, Chris On 10/4/10 8:36 AM, Simon Matter wrote: All, I have run a small mail service based on Cyrus IMAP for a few years. The (CentOS 5) server I've used for the past couple years failed last week. I brought up a new CentOS 5 system on a new Linux server, installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk (actually one of 2 mdadm submirrors), and copied /var/spool/imap over to the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM. Did you also recover /var/lib/imap from the old server and make sure the configs are the same? Simon Unfortunately, for many users (all but me?), mail clients (at least Apple's Mail.app and SquirrelMail) don't show any messages from before the migration in INBOX. When I grope around in /var/spool/imap/user, I see the old message (with high numbers) and the new messages (starting a new sequence from 1). I know there are last-message counters in the cyrus.* files, so would resetting those have caused IMAP to start storing new messages from 1. and vanish the old ones? I don't know how those could have been cleared for at least 6 users simultaneously. Presumably it happened for all, and I just fixed it for myself somehow. More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of -rfx, and quota -f, but not found any way to make it show the old messages. Any suggestions? Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
Simon, I did recover /var/lib/imap (although a bit later, FWIW) and I think a bit later is your problem. But I think reconstruct should fix this. BTW, did you check the subscription status of mailboxes? Maybe the folders are there but not subscribed, and that's why some users can't see them? /etc/imapd.conf. I just now compared /etc/cyrus.conf and added squatter adjusted prefork numbers -- the rest all matched. Is there other configuration I should check? Both systems are 64-bit CentOS 5, so the db4 installations should be fully compatible. If both the old and the new system are basically the same I don't think there is anything missing. Simon Thanks, Chris On 10/4/10 8:36 AM, Simon Matter wrote: All, I have run a small mail service based on Cyrus IMAP for a few years. The (CentOS 5) server I've used for the past couple years failed last week. I brought up a new CentOS 5 system on a new Linux server, installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk (actually one of 2 mdadm submirrors), and copied /var/spool/imap over to the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM. Did you also recover /var/lib/imap from the old server and make sure the configs are the same? Simon Unfortunately, for many users (all but me?), mail clients (at least Apple's Mail.app and SquirrelMail) don't show any messages from before the migration in INBOX. When I grope around in /var/spool/imap/user, I see the old message (with high numbers) and the new messages (starting a new sequence from 1). I know there are last-message counters in the cyrus.* files, so would resetting those have caused IMAP to start storing new messages from 1. and vanish the old ones? I don't know how those could have been cleared for at least 6 users simultaneously. Presumably it happened for all, and I just fixed it for myself somehow. More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of -rfx, and quota -f, but not found any way to make it show the old messages. Any suggestions? Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
Simon, I did recover /var/lib/imap (although a bit later, FWIW) and /etc/imapd.conf. I just now compared /etc/cyrus.conf and added squatter adjusted prefork numbers -- the rest all matched. Is there other configuration I should check? Both systems are 64-bit CentOS 5, so the db4 installations should be fully compatible. Thanks, Chris On 10/4/10 8:36 AM, Simon Matter wrote: All, I have run a small mail service based on Cyrus IMAP for a few years. The (CentOS 5) server I've used for the past couple years failed last week. I brought up a new CentOS 5 system on a new Linux server, installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk (actually one of 2 mdadm submirrors), and copied /var/spool/imap over to the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM. Did you also recover /var/lib/imap from the old server and make sure the configs are the same? Simon Unfortunately, for many users (all but me?), mail clients (at least Apple's Mail.app and SquirrelMail) don't show any messages from before the migration in INBOX. When I grope around in /var/spool/imap/user, I see the old message (with high numbers) and the new messages (starting a new sequence from 1). I know there are last-message counters in the cyrus.* files, so would resetting those have caused IMAP to start storing new messages from 1. and vanish the old ones? I don't know how those could have been cleared for at least 6 users simultaneously. Presumably it happened for all, and I just fixed it for myself somehow. More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of -rfx, and quota -f, but not found any way to make it show the old messages. Any suggestions? Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
All, I have run a small mail service based on Cyrus IMAP for a few years. The (CentOS 5) server I've used for the past couple years failed last week. I brought up a new CentOS 5 system on a new Linux server, installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk (actually one of 2 mdadm submirrors), and copied /var/spool/imap over to the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM. Did you also recover /var/lib/imap from the old server and make sure the configs are the same? Simon Unfortunately, for many users (all but me?), mail clients (at least Apple's Mail.app and SquirrelMail) don't show any messages from before the migration in INBOX. When I grope around in /var/spool/imap/user, I see the old message (with high numbers) and the new messages (starting a new sequence from 1). I know there are last-message counters in the cyrus.* files, so would resetting those have caused IMAP to start storing new messages from 1. and vanish the old ones? I don't know how those could have been cleared for at least 6 users simultaneously. Presumably it happened for all, and I just fixed it for myself somehow. More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of -rfx, and quota -f, but not found any way to make it show the old messages. Any suggestions? Thanks, Chris Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
A script to find each file and IMAP append it? Just thinking outside the box here! Chris Pepper pep...@reppep.com wrote: On 10/4/10 9:17 AM, Simon Matter wrote: Simon, I did recover /var/lib/imap (although a bit later, FWIW) and I think a bit later is your problem. But I think reconstruct should fix this. BTW, did you check the subscription status of mailboxes? Maybe the folders are there but not subscribed, and that's why some users can't see them? Simon, No, users see the folders, just not old messages. For most (all?) INBOXes but my own, new messages started arriving as 1. and continued from there. Users can see the new mail, but not the old. This makes me think it's not an internal permissions problem, because they see the mailboxes and (some) mail in them. All file permissions I checked appear correct reconstruct -rfx doesn't help. Is there anything else to try? Thanks, Chris /etc/imapd.conf. I just now compared /etc/cyrus.conf and added squatter adjusted prefork numbers -- the rest all matched. Is there other configuration I should check? Both systems are 64-bit CentOS 5, so the db4 installations should be fully compatible. If both the old and the new system are basically the same I don't think there is anything missing. Simon Thanks, Chris On 10/4/10 8:36 AM, Simon Matter wrote: All, I have run a small mail service based on Cyrus IMAP for a few years. The (CentOS 5) server I've used for the past couple years failed last week. I brought up a new CentOS 5 system on a new Linux server, installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk (actually one of 2 mdadm submirrors), and copied /var/spool/imap over to the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM. Did you also recover /var/lib/imap from the old server and make sure the configs are the same? Simon Unfortunately, for many users (all but me?), mail clients (at least Apple's Mail.app and SquirrelMail) don't show any messages from before the migration in INBOX. When I grope around in /var/spool/imap/user, I see the old message (with high numbers) and the new messages (starting a new sequence from 1). I know there are last-message counters in the cyrus.* files, so would resetting those have caused IMAP to start storing new messages from 1. and vanish the old ones? I don't know how those could have been cleared for at least 6 users simultaneously. Presumably it happened for all, and I just fixed it for myself somehow. More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of -rfx, and quota -f, but not found any way to make it show the old messages. Any suggestions? Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/3/2010 6:57 AM, Chris Pepper wrote: More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of -rfx, and quota -f, but not found any way to make it show the old messages. Any suggestions? You probably need to run cyrreconstruct on each user mailbox. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/4/10 1:12 AM, Patrick Goetz wrote: On 10/3/2010 6:57 AM, Chris Pepper wrote: More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of -rfx, and quota -f, but not found any way to make it show the old messages. Any suggestions? You probably need to run cyrreconstruct on each user mailbox. On my system it's /usr/lib/cyrus-imapd/reconstruct, and I have. No joy, alas. Chris Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/04/2010 08:37 AM, Chris Pepper wrote: No, users see the folders, just not old messages. For most (all?) INBOXes but my own, new messages started arriving as 1. and continued from there. Users can see the new mail, but not the old. This makes me think it's not an internal permissions problem, because they see the mailboxes and (some) mail in them. All file permissions I checked appear correct reconstruct -rfx doesn't help. Is there anything else to try? I wasn't clear about whether the old install was completely gone or could still be booted. If you can still start cyrus on the old server, you could try imapsync to transfer mail to the new one. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/04/2010 08:47 AM, Bron Gondwana wrote: A script to find each file and IMAP append it? Just thinking outside the box here! How would you do the IMAP append? Using a Perl::IMAP function? This isn't necessarily a concern for this list, but a few days ago I upgraded a site from cyrus 2.1.16 to cyrus 2.3.16 by using imapsync to transfer mail from the old server to the new one. This worked great (i.e. all metadata seems to have been preserved), however the old server was still collecting mail for the few hours it took to imapsync all the users (note that this is very slows and probably not appropriate for larger sites). My plan was to swap the servers and then do a final imapsync from the old IMAP server to the new one. For some reason, for some users the additional set of new messages is copied to the new server, and others they aren't. I couldn't figure out what the difference is, and don't want to spend too much more time on this because of the small number of messages in play, so I need some way to transfer a couple of dozen messages by hand. In general, though imapsync seems to be a great way to clean up cyrus folders when switching servers. Had I known I would have sync problem later, I would have just taken the old server off line before syncing the messages. Before someone suggests that I should have just copied /var/lib/cyrus and the messages over to the new server, I didn't trust this because I couldn't get anyone to confirm the database files I had on the old 2.1.16 server -- the filenames were mostly not the same as this set: annotation_db duplicate_db mboxlist_db ptscache_db quota_db seenstate_db tlscache_db and the old server experienced a couple of devastating crashes which required me to cyrreconstruct all the user.user mailboxes a couple of times. It seemed pretty clear that the db files on the old server were a mess, and upgrading from anywhere from libdb2 to libdb4.3 to libdb4.8 seemed sketchy as well. The old server is such an old debian system that apt-cache depends doesn't seem to work any more because the package servers have changed. (Yeah, I could probably find the source packages somewhere, but then there are a lot of things I could do given infinite time.) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/4/10 10:23 AM, Patrick Goetz wrote: On 10/04/2010 08:37 AM, Chris Pepper wrote: No, users see the folders, just not old messages. For most (all?) INBOXes but my own, new messages started arriving as 1. and continued from there. Users can see the new mail, but not the old. This makes me think it's not an internal permissions problem, because they see the mailboxes and (some) mail in them. All file permissions I checked appear correct reconstruct -rfx doesn't help. Is there anything else to try? I wasn't clear about whether the old install was completely gone or could still be booted. If you can still start cyrus on the old server, you could try imapsync to transfer mail to the new one. Old system is not bootable, unfortunately. FYI: I have 943 directories 298,409 mail files, so manually fixing things isn't feasible. Thanks for all the suggestions! Chris -- Chris Pepper:http://cbio.mskcc.org/ http://www.extrapepperoni.com/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
I wasn't clear about whether the old install was completely gone or could still be booted. If you can still start cyrus on the old server, you could try imapsync to transfer mail to the new one. Old system is not bootable, unfortunately. FYI: I have 943 directories 298,409 mail files, so manually fixing things isn't feasible. Thanks for all the suggestions! Are the location of the old mails correct for the new system ? Means: When you deliver a new mail, does it appear in the same folder as the old ones ? Another thing: Did you make sure the reconstruct runs under the correct user account ? (Probably cyrus) and the file owner/groups are correct for the new system ? (Not root) André Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/04/2010 10:17 AM, Chris Pepper wrote: On 10/4/10 10:23 AM, Patrick Goetz wrote: I wasn't clear about whether the old install was completely gone or could still be booted. If you can still start cyrus on the old server, you could try imapsync to transfer mail to the new one. Old system is not bootable, unfortunately. FYI: I have 943 directories 298,409 mail files, so manually fixing things isn't feasible. This seems like a long shot, but could you temporarily set up another machine with the old version of CentOS, copy /var/lib/cyrus and /var/spool/cyrus to this machine as is from the old server, and then run imapsync? The other option is as Bron suggested, using some kind of IMAP function to append lost messages to the spool. I'm pretty that this will result in all the metadata being lost (i.e. replied to and forwarded flags, etc.) (And moreover, I don't know to do this, so can't advise.) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On Mon, Oct 04, 2010 at 11:17:59AM -0400, Chris Pepper wrote: On 10/4/10 10:23 AM, Patrick Goetz wrote: I wasn't clear about whether the old install was completely gone or could still be booted. If you can still start cyrus on the old server, you could try imapsync to transfer mail to the new one. Old system is not bootable, unfortunately. FYI: I have 943 directories 298,409 mail files, so manually fixing things isn't feasible. ^^ Well, of course not. It there were 5 directories and 20 mail files I wouldn't consider doing it manually - I'd write a script to automate it and then sit back and drink coffee. Manually is how you do the first one to find out how it's done. And then the second one to make sure the process you settled on after stack of trial and error is repeatable. Maybe a third one if you screwed up number 2. After that, you automate the process you've decided on and let the rest happen automatically. It always amazes me to see admins repeating themselves manually over and over for a frequent task. I'd love to see the output of your reconstruct command (including syslog with the logging level turned up) to see why it's not finding the files. And maybe an 'ls -la' of one of the imap directories with this issue as well. I'm away on a team building exercise for the next couple of days Oslo time - but I can certainly look at it afterwards. I've been travelling (from Australia) which is why I haven't been looking in on this earlier... it's a strange set of symptoms. If you have a small folder that you don't mind sharing, a tar.gz of the entire folder contents (including the metadata files) would be fantastic, because then I could check the contents of the .index and .cache files as well - and maybe even try a reconstruct on a testbed here. But file permissions are the most interesting - I'm wondering if reconstruct is unable to read the directory correctly or unable to read the old files. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/4/10 11:36 AM, Bron Gondwana wrote: On Mon, Oct 04, 2010 at 11:17:59AM -0400, Chris Pepper wrote: On 10/4/10 10:23 AM, Patrick Goetz wrote: I wasn't clear about whether the old install was completely gone or could still be booted. If you can still start cyrus on the old server, you could try imapsync to transfer mail to the new one. Old system is not bootable, unfortunately. FYI: I have 943 directories 298,409 mail files, so manually fixing things isn't feasible. ^^ Well, of course not. It there were 5 directories and 20 mail files I wouldn't consider doing it manually - I'd write a script to automate it and then sit back and drink coffee. Manually is how you do the first one to find out how it's done. And then the second one to make sure the process you settled on after stack of trial and error is repeatable. Maybe a third one if you screwed up number 2. After that, you automate the process you've decided on and let the rest happen automatically. It always amazes me to see admins repeating themselves manually over and over for a frequent task. I'd love to see the output of your reconstruct command (including syslog with the logging level turned up) to see why it's not finding the files. And maybe an 'ls -la' of one of the imap directories with this issue as well. I'm away on a team building exercise for the next couple of days Oslo time - but I can certainly look at it afterwards. I've been travelling (from Australia) which is why I haven't been looking in on this earlier... it's a strange set of symptoms. If you have a small folder that you don't mind sharing, a tar.gz of the entire folder contents (including the metadata files) would be fantastic, because then I could check the contents of the .index and .cache files as well - and maybe even try a reconstruct on a testbed here. But file permissions are the most interesting - I'm wondering if reconstruct is unable to read the directory correctly or unable to read the old files. Bron, Unfortunately I don't know how to write a script to do this. The old system was CentOS 5/x64, just like the new system. I believe it had the same cyrus-imapd db4 RPMs, as I had patched a few weeks ago, but I cannot confirm. I will send you a tarball of this directory directly. I will try reconstruct -rfx with debug logging enabled in syslog tonight. Thanks, Chris [r...@inspector ~]# ls -ltr /var/spool/imap/user/julia/ total 1756 -rw--- 1 cyrus mail 2931 May 20 12:56 158. -rw--- 1 cyrus mail 2052 Jun 21 08:33 159. -rw--- 1 cyrus mail 1536 Aug 11 17:15 162. -rw--- 1 cyrus mail 2221 Aug 13 08:15 163. -rw--- 1 cyrus mail 413395 Aug 14 18:40 164. -rw--- 1 cyrus mail 79756 Aug 21 13:57 165. -rw--- 1 cyrus mail 296759 Aug 22 15:18 166. -rw--- 1 cyrus mail 156064 Sep 4 09:43 167. -rw--- 1 cyrus mail 75003 Sep 4 10:06 168. -rw--- 1 cyrus mail 164035 Sep 6 09:51 169. -rw--- 1 cyrus mail 156853 Sep 7 12:24 170. -rw--- 1 cyrus mail 340068 Sep 16 17:44 171. drwx-- 2 cyrus mail 4096 Oct 1 21:31 Junk drwx-- 2 cyrus mail 4096 Oct 2 23:40 Drafts drwx-- 2 cyrus mail 4096 Oct 2 23:40 Sent drwx-- 2 cyrus mail 4096 Oct 2 23:40 Apple Mail To Do drwx-- 2 cyrus mail 4096 Oct 2 23:40 Deleted Messages -rw--- 1 cyrus mail 1148 Oct 2 23:44 1. drwx-- 2 cyrus mail 4096 Oct 3 00:09 Trash drwx-- 2 cyrus mail 4096 Oct 3 13:16 Sent Messages -rw--- 1 cyrus mail179 Oct 3 13:39 cyrus.header -rw--- 1 cyrus mail176 Oct 4 09:04 cyrus.index -rw--- 1 cyrus mail640 Oct 4 09:04 cyrus.cache -- Chris Pepper:http://cbio.mskcc.org/ http://www.extrapepperoni.com/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
Thanks, all, for the generous help. Bron asked about output from reconstruct, which never provided any. It turns out that I was using the wrong delimiters (., per chk_cyrus output, rather than /), and reconstruct wasn't even trying. It looks like all the missing mail is accessible again. Thanks again! Chris -- Chris Pepper:http://cbio.mskcc.org/ http://www.extrapepperoni.com/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On Mon, Oct 04, 2010 at 05:26:19PM -0400, Chris Pepper wrote: Thanks, all, for the generous help. Bron asked about output from reconstruct, which never provided any. It turns out that I was using the wrong delimiters (., per chk_cyrus output, rather than /), and reconstruct wasn't even trying. It looks like all the missing mail is accessible again. Excellent! Glad to be of help :) (and yeah, the internal vs external naming of folders is something that annoys me and I'd like to tidy up) Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
Dear Patrick, How would you do the IMAP append? Using a Perl::IMAP function? This isn't necessarily a concern for this list, but a few days ago I upgraded a site from cyrus 2.1.16 to cyrus 2.3.16 by using imapsync to transfer mail from the old server to the new one. This worked great (i.e. all metadata seems to have been preserved), however the old server was still collecting mail for the few hours it took to imapsync all the users (note that this is very slows and probably not appropriate for larger sites). My plan was to swap the servers and then do a final imapsync from the old IMAP server to the new one. For some reason, for some users the additional set of new messages is copied to the new server, and others they aren't. I couldn't figure out what the difference is, and don't want to spend too much more time on this because of the small number of messages in play, so I need some way to transfer a couple of dozen messages by hand. In general, though imapsync seems to be a great way to clean up cyrus folders when switching servers. Had I known I would have sync problem later, I would have just taken the old server off line before syncing the messages. Before someone suggests that I should have just copied /var/lib/cyrus and the messages over to the new server, I didn't trust this because I couldn't get anyone to confirm the database files I had on the old 2.1.16 server -- the filenames were mostly not the same as this set: annotation_db duplicate_db mboxlist_db ptscache_db quota_db seenstate_db tlscache_db and the old server experienced a couple of devastating crashes which required me to cyrreconstruct all the user.user mailboxes a couple of times. It seemed pretty clear that the db files on the old server were a mess, and upgrading from anywhere from libdb2 to libdb4.3 to libdb4.8 seemed sketchy as well. The old server is such an old debian system that apt-cache depends doesn't seem to work any more because the package servers have changed. (Yeah, I could probably find the source packages somewhere, but then there are a lot of things I could do given infinite time.) We use IMAPsync all the time, manually and (more often) through automated programs. One of our customers uses Gmail for their corporate email system (18,000+ user accounts, about 1000-2000 accounts of ex-employees at any point in time). We recently wrote a layer on top of our mailing product for him, to download and archive all the messages of all his ex-employees' mailboxes. We used IMAPsync in an automated mode to download all his mails. He gives us the usernames and passwords, and our server-based system does the rest, creating local mirror mail accounts and downloading the Gmail mailboxes to local Cyrus IMAP mailboxes. We've learned a few things from all this. Firstly, an older version of IMAPsync (IIRC it was v1.25x) used to hang at random and consume 100% CPU indefinitely. Upgrading to a recent release (v1.34x or later) has fixed the problem. Secondly, we have found that IMAP sessions _often_ terminate halfway for no fault of IMAPsync. Reattempting the IMAP sync succeeds. Therefore, for our automated IMAP download-and-archive solution, we wrote a daemon which fires IMAPsync for each account in an infinite loop. Whenever IMAPsync terminates with errorlevel = 0, we found that we can safely believe the session completed successfully, and complete sync has been done. So our daemon makes note of this, and doesn't try to sync this user's mailboxes any more. Therefore, our IMAPsync monitor daemon is prepared to retry sync for a recalcitrant user account an infinite number of times. Once we understood that these errors are not repeatable, and hits accounts at random, we gave up trying to debug why -- we just run IMAPsync over and over till we get an all-clear. It works well. Some users had 10+GB of messages -- all of it is pulled down eventually. Of course, these random errors are probably more a Google IMAP issue than IMAP in general. If you often need to sync mailboxes among IMAP folders, I'd suggest you write a monitor daemon and fire IMAPsync from within it in an infinite loop the way I described. Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
IMAP not seeing old mail present on filesystem
All, I have run a small mail service based on Cyrus IMAP for a few years. The (CentOS 5) server I've used for the past couple years failed last week. I brought up a new CentOS 5 system on a new Linux server, installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk (actually one of 2 mdadm submirrors), and copied /var/spool/imap over to the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM. Unfortunately, for many users (all but me?), mail clients (at least Apple's Mail.app and SquirrelMail) don't show any messages from before the migration in INBOX. When I grope around in /var/spool/imap/user, I see the old message (with high numbers) and the new messages (starting a new sequence from 1). I know there are last-message counters in the cyrus.* files, so would resetting those have caused IMAP to start storing new messages from 1. and vanish the old ones? I don't know how those could have been cleared for at least 6 users simultaneously. Presumably it happened for all, and I just fixed it for myself somehow. More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of -rfx, and quota -f, but not found any way to make it show the old messages. Any suggestions? Thanks, Chris Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
