Re: IMAP sync tool (rsync for IMAP)
--syncinternaldates stopped working at imapsync 1.200 (current is 1.204), as a side effect of the append_string-append_file change suggested by John Capo and Florin Andrei earlier in this thread, on Dec 26, 2006. That change was made to avoid memory exhaustion, see http://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg31296.html The underlying problem is that at least as of Mail::IMAPClient 2.2.9, the append_file function doesn't handle INTERNALDATE at all. A bunch of people here have talked about minor patches to Mail::IMAPClient. Has anyone added the $d argument to append_file and verified functionality? Most clients display Date: header, but some may show or use INTERNALDATE. For example, the Zimbra web interface is incapable of showing or sorting by Date:. See also http://www.zimbra.com/forums/showthread.php?p=33880#post33880 I already mailed the author of imapsync about that some days ago but without having an idea how to fix it. I also checked Mail::IMAPClient and found that it lacked the functionality needed here but it seems Mail::IMAPClient isn't maintained anymore (at least there is no development for long time now). Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
I already mailed the author of imapsync about that some days ago but without having an idea how to fix it. I also checked Mail::IMAPClient and found that it lacked the functionality needed here but it seems Mail::IMAPClient isn't maintained anymore (at least there is no development for long time now). I don't know how much work it would be, but you could look at re-implementing using Mail::IMAPTalk. It's the IMAP client I built for use at FastMail.FM after finding problems with the other ones available at the time (about 4-5 years ago now). It definitely allows capturing a literal response into a file handle, and sending a file handle as a literal response to avoid storing everything in memory. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
I'd started taking this offline because it's not entirely on-topic for info-cyrus, but I realize that it has broad impact... On Jan 3 2007, at 8:37 PM, Jo Rhett wrote: On Jan 3, 2007, at 9:20 AM, James Miller wrote: I have 2 users left on an OLD Cyrus installation I want to get off and was hoping to use imapsync. It seemed to work just fine except dates of the messages from the old server to the new were not retained. All of the messages have the date they were imported instead. You need the syncinternaldates. Did you do a sync already without that option? You need to toast the destination and resync it to get the dates fixed... --syncinternaldates stopped working at imapsync 1.200 (current is 1.204), as a side effect of the append_string-append_file change suggested by John Capo and Florin Andrei earlier in this thread, on Dec 26, 2006. That change was made to avoid memory exhaustion, see http://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg31296.html The underlying problem is that at least as of Mail::IMAPClient 2.2.9, the append_file function doesn't handle INTERNALDATE at all. A bunch of people here have talked about minor patches to Mail::IMAPClient. Has anyone added the $d argument to append_file and verified functionality? Most clients display Date: header, but some may show or use INTERNALDATE. For example, the Zimbra web interface is incapable of showing or sorting by Date:. See also http://www.zimbra.com/forums/showthread.php?p=33880#post33880 -- Rich Graves [EMAIL PROTECTED] Sr UNIX and Security Administrator Ofc 507-646-7079 Cell 952-292-6529 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Hello, This problem really annoys most people, including me. The old version kills all the system memory and even create zombie. syncinternaldates option is very important to many mail client. By missing this option, I think this is a GREAT impact. Hopely you guys find a prompt solution. Greatly thanks Patrick - Original Message - From: Rich Graves [EMAIL PROTECTED] To: info-cyrus@lists.andrew.cmu.edu Sent: Thursday, January 18, 2007 11:51 PM Subject: Re: IMAP sync tool (rsync for IMAP) I'd started taking this offline because it's not entirely on-topic for info-cyrus, but I realize that it has broad impact... On Jan 3 2007, at 8:37 PM, Jo Rhett wrote: On Jan 3, 2007, at 9:20 AM, James Miller wrote: I have 2 users left on an OLD Cyrus installation I want to get off and was hoping to use imapsync. It seemed to work just fine except dates of the messages from the old server to the new were not retained. All of the messages have the date they were imported instead. You need the syncinternaldates. Did you do a sync already without that option? You need to toast the destination and resync it to get the dates fixed... --syncinternaldates stopped working at imapsync 1.200 (current is 1.204), as a side effect of the append_string-append_file change suggested by John Capo and Florin Andrei earlier in this thread, on Dec 26, 2006. That change was made to avoid memory exhaustion, see http://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg31296.html The underlying problem is that at least as of Mail::IMAPClient 2.2.9, the append_file function doesn't handle INTERNALDATE at all. A bunch of people here have talked about minor patches to Mail::IMAPClient. Has anyone added the $d argument to append_file and verified functionality? Most clients display Date: header, but some may show or use INTERNALDATE. For example, the Zimbra web interface is incapable of showing or sorting by Date:. See also http://www.zimbra.com/forums/showthread.php?p=33880#post33880 -- Rich Graves [EMAIL PROTECTED] Sr UNIX and Security Administrator Ofc 507-646-7079 Cell 952-292-6529 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: IMAP sync tool (rsync for IMAP)
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Florin Andrei Sent: Saturday, December 23, 2006 2:14 AM To: info-cyrus@lists.andrew.cmu.edu Subject: Re: IMAP sync tool (rsync for IMAP) Florin Andrei wrote: Sort of like rsync for IMAP, if that makes sense. So far, the only tool I've found is imapsync: http://freshmeat.net/projects/imapsync/ Seems to be working fine. It's recursive, it's smart (subsequent runs don't download the old messages again, only the new messages), the command-line switches are a rich set that's pretty much self-explanatory. It's easy to do non-destructive tests. It's flexible enough. Downside: If you have very large messages on the source IMAP servers, you better run the script on a machine with lots of RAM. On my system it died repeatedly with out of memory when it was hitting a 32MB message (on-disk size), and the system has 1GB RAM. It's probably best to run it on a 3rd system, one that's separate from both the source and the destination IMAP servers, otherwise unpleasant things might happen if the script starts to gobble up RAM on the IMAP server. Again, this 3rd system is best if it has lots of RAM if you allow large messages on your source IMAP server. My apologies for this being off topic but I was hoping someone might be able to comment. I have 2 users left on an OLD Cyrus installation I want to get off and was hoping to use imapsync. It seemed to work just fine except dates of the messages from the old server to the new were not retained. All of the messages have the date they were imported instead. I ran imapsync on the destination host, I wouldn't think that would make a difference - does it matter? /usr/bin/imapsync --syncinternaldates --host1 source.host --user1 bob \ --noauthmd5 --passfile1 /tmp/secret1 --host2 destination.host --user2 bob \ --noauthmd5 --passfile2 /tmp/secret2 Any suggestions would be appreciated. --Jim Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Florin Andrei wrote: Anyway, slower is better than crashing. I'll try and contact the author to make him aware of this discussion thread. I did it and there's version 1.200 posted on Freshmeat claiming the memory issue has been fixed. I'll give it a try one of these days. http://freshmeat.net/projects/imapsync/ -- Florin Andrei http://florin.myip.org/ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
James Miller wrote: I have 2 users left on an OLD Cyrus installation I want to get off and was hoping to use imapsync. It seemed to work just fine except dates of the messages from the old server to the new were not retained. All of the messages have the date they were imported instead. I ran imapsync on the destination host, I wouldn't think that would make a difference - does it matter? /usr/bin/imapsync --syncinternaldates --host1 source.host --user1 bob \ --noauthmd5 --passfile1 /tmp/secret1 --host2 destination.host --user2 bob \ --noauthmd5 --passfile2 /tmp/secret2 I used it without --syncinternaldates and it worked fine, the dates were imported correctly. imapsync --host1 X.Y.Z.K --user1 joeblow --passfile1 pass1 \ --host2 A.B.C.D --user2 joeblow --passfile2 pass2 \ --authmech1 LOGIN --authmech2 LOGIN \ --prefix2 oldmail/ \ --subscribe The --prefix2 option was used because I did not actually migrate one account to another, but I just wanted to copy the old account under a folder on the new account. That doesn't seem to be your case. -- Florin Andrei http://florin.myip.org/ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
On Jan 3, 2007, at 9:20 AM, James Miller wrote: I have 2 users left on an OLD Cyrus installation I want to get off and was hoping to use imapsync. It seemed to work just fine except dates of the messages from the old server to the new were not retained. All of the messages have the date they were imported instead. I ran imapsync on the destination host, I wouldn't think that would make a difference - does it matter? /usr/bin/imapsync --syncinternaldates --host1 source.host --user1 bob \ --noauthmd5 --passfile1 /tmp/secret1 --host2 destination.host -- user2 bob \ --noauthmd5 --passfile2 /tmp/secret2 You need the syncinternaldates. Did you do a sync already without that option? You need to toast the destination and resync it to get the dates fixed... -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Quoting Florin Andrei ([EMAIL PROTECTED]): John Capo wrote: A 32MB message occupies a lot more than 32MB when its a Perl string. Use a temp file instead. my $message_file = /tmp/xfer-$$; unlink($message_file); $from-message_to_file($message_file, $f_msg); And further down. unless($new_id = $to-append_file($t_fold, $message_file, $flags_f, $d)) Instead of unless($new_id = $to-append_string($t_fold,$string, $flags_f, $d)) It's hard to tell without actually testing that change, but I assume it would slow down the script a little bit? I doubt that you would see a speed difference unless you were transferring to and from completely idle GigE connected machines and maybe not even then. Anyway, slower is better than crashing. I'll try and contact the author to make him aware of this discussion thread. Agreed. -- Florin Andrei http://florin.myip.org/ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Quoting Florin Andrei ([EMAIL PROTECTED]): Marten Lehmann wrote: Downside: If you have very large messages on the source IMAP servers, you better run the script on a machine with lots of RAM. On my system it died repeatedly with out of memory when it was hitting a 32MB message (on-disk size), and the system has 1GB RAM. strange, I didn't experience this while running a sync for 15.000 mailboxes some weeks ago with even larger messages partially. Yeah, and after a quick google search it turned out I'm not the only one to experience that. Its definately a problem with large messages. my $string = $from-message_string($f_msg); A 32MB message occupies a lot more than 32MB when its a Perl string. Use a temp file instead. my $message_file = /tmp/xfer-$$; unlink($message_file); $from-message_to_file($message_file, $f_msg); And further down. unless($new_id = $to-append_file($t_fold, $message_file, $flags_f, $d)) Instead of unless($new_id = $to-append_string($t_fold,$string, $flags_f, $d)) Ah well, except for just one very large message, everything else was transferred OK. -- Florin Andrei http://florin.myip.org/ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
John Capo wrote: A 32MB message occupies a lot more than 32MB when its a Perl string. Use a temp file instead. my $message_file = /tmp/xfer-$$; unlink($message_file); $from-message_to_file($message_file, $f_msg); And further down. unless($new_id = $to-append_file($t_fold, $message_file, $flags_f, $d)) Instead of unless($new_id = $to-append_string($t_fold,$string, $flags_f, $d)) It's hard to tell without actually testing that change, but I assume it would slow down the script a little bit? Anyway, slower is better than crashing. I'll try and contact the author to make him aware of this discussion thread. -- Florin Andrei http://florin.myip.org/ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
On Tue, 26 Dec 2006 14:45:43 -0800, Florin Andrei [EMAIL PROTECTED] said: John Capo wrote: A 32MB message occupies a lot more than 32MB when its a Perl string. Use a temp file instead. my $message_file = /tmp/xfer-$$; unlink($message_file); $from-message_to_file($message_file, $f_msg); And further down. unless($new_id = $to-append_file($t_fold, $message_file, $flags_f, $d)) Instead of unless($new_id = $to-append_string($t_fold,$string, $flags_f, $d)) It's hard to tell without actually testing that change, but I assume it would slow down the script a little bit? Anyway, slower is better than crashing. I'll try and contact the author to make him aware of this discussion thread. The usual way to deal with this if you still care about speed is to check the message size before hand (presumably you've already done a fetch of the UID list to work out what needs to be copied - just grab the rfc822.size field as well). Then pick some sane amount (say, 100kb) and any message bigger than that goes to disk - smaller messages can be processed directly in memory. That said, most modern filesystems are pretty good about not causing excessive IO for temporary files, and /tmp/ could very well be a memory disk anyway (we have /tmpfs/ for precisely this sort of purpose) (that and: unless your IMAP servers are really closely network connected the network IO is going to be your critical path anyway) Bron. -- Bron Gondwana [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Hello, Downside: If you have very large messages on the source IMAP servers, you better run the script on a machine with lots of RAM. On my system it died repeatedly with out of memory when it was hitting a 32MB message (on-disk size), and the system has 1GB RAM. strange, I didn't experience this while running a sync for 15.000 mailboxes some weeks ago with even larger messages partially. It's probably best to run it on a 3rd system, one that's separate from both the source and the destination IMAP servers, otherwise unpleasant things might happen if the script starts to gobble up RAM on the IMAP server. Again, this 3rd system is best if it has lots of RAM if you allow large messages on your source IMAP server. Thats surely a good advice, but I don't think it is primarily because of the memory usage, but because of the CPU usage. While syncing and checking for differences on source and target server imapsync will use _much_ CPU ressources, especially if you are running several instances of imapsync at the same time to speed up the sync (on different mailboxes of course). The source and target IMAP servers have been pretty idle while the 3rd server running imasync had a load average of 50. Regards Marten Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Marten Lehmann wrote: Downside: If you have very large messages on the source IMAP servers, you better run the script on a machine with lots of RAM. On my system it died repeatedly with out of memory when it was hitting a 32MB message (on-disk size), and the system has 1GB RAM. strange, I didn't experience this while running a sync for 15.000 mailboxes some weeks ago with even larger messages partially. Yeah, and after a quick google search it turned out I'm not the only one to experience that. Ah well, except for just one very large message, everything else was transferred OK. -- Florin Andrei http://florin.myip.org/ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
On Thu, Dec 21, 2006 at 03:47:26PM -0800, Florin Andrei wrote: Essentially, I need a tool that I can point at servers A and B and tell it get all the email from my account on server A to a specific folder on my account on server B, preserving the subfolders hierarchy. The tool needs to be smart enough to repeat the operation later on but then it must only transfer the new messages. The tool may run on one or the other IMAP servers, or even on a 3rd machine, since it should be network-based. Pretty much all systems are Linux 'round here, some Windows stragglers too. Sort of like rsync for IMAP, if that makes sense. So far, the only tool I've found is imapsync: http://freshmeat.net/projects/imapsync/ Anyone tried it with Cyrus? Good/bad experiences? Are there any other tools that work better with Cyrus? Another thing you might want to consider is offlineimap: http://software.complete.org/offlineimap/ It's a pile of multithreaded python, but don't let that put you off! I've found it quite robust for IMAP - Maildir usage (Indeed, I'm replying to this message via Mutt using local Maildirs which are kept synchronised with my FastMail (Cyrus 2.3.7+patches) IMAP account). Imap = Imap usage I found less robust in that it sometimes got confused when it had been killed in the middle of operations. If you didn't keep killing it all the time (my usage patterns were pretty strange) it was better - and also I think if you had a server which supports UID PLUS like Cyrus does then it would be safer. Give it a look though. apt-get install offlineimap works on pretty much any sane linux these days :) (I believe you can even get it with apt4rpm, though that's not my particular kink) For RedHat/Fedora (not blessed by apt/deb by default), I'm maintaining offlineimap src rpms here http://www.invoca.ch/pub/packages/offlineimap/ which can be built using rpmbuild --rebuild offlineimap-4.x.xx-x.src.rpm The same is true for imapsync here http://www.invoca.ch/pub/packages/imapsync/ Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Florin Andrei wrote: Sort of like rsync for IMAP, if that makes sense. So far, the only tool I've found is imapsync: http://freshmeat.net/projects/imapsync/ Seems to be working fine. It's recursive, it's smart (subsequent runs don't download the old messages again, only the new messages), the command-line switches are a rich set that's pretty much self-explanatory. It's easy to do non-destructive tests. It's flexible enough. Downside: If you have very large messages on the source IMAP servers, you better run the script on a machine with lots of RAM. On my system it died repeatedly with out of memory when it was hitting a 32MB message (on-disk size), and the system has 1GB RAM. It's probably best to run it on a 3rd system, one that's separate from both the source and the destination IMAP servers, otherwise unpleasant things might happen if the script starts to gobble up RAM on the IMAP server. Again, this 3rd system is best if it has lots of RAM if you allow large messages on your source IMAP server. -- Florin Andrei http://florin.myip.org/ Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Florin Andrei wrote:
I'm currently using two IMAP accounts, one on Cyrus-2.2 the other on
Cyrus-2.3.
The one on Cyrus-2.3 will get decomissioned so I need to transfer all my
email, preserving the folders/subfolders tree, under a specific folder
(oldmail/foo/bar) on the 2.2 server.
You need imapsync or similar if you are in a multimaster situation
(syncing changes made to *both* ends) or if the IMAP server software
differs, but if you have shell access to cyrus on both ends, rsync
(literally) should work fine. If the folder is quiescent and you're
using platform-independent skiplist databases, then you shouldn't even
need to run reconstruct.
A few months ago, I rsync'd 500GB of mail from Solaris/vxfs/Cyrus 2.2 to
Linux/ext3/Cyrus 2.3 with 15 minutes' downtime, reorganizing the
partitions a bit along the way.
Old server: partitions /var/spool/imap (partition-default),
/var/spool/imap1 (partition-1), /var/spool/imap2 (partition-2),
/var/spool/imap0 (partition-0).
partition-0 caused various problems with cyradmin (0 is false to
perl), so I renamed it to partition-3 on the new server. At the same
time, I renamed partition-default to partition-4.
New server: /var/spool/imap{1,2,3,4} (partition-{1,2,3,4}). New server
also has metapartition_files: squat with all squat indices stored on a
low-cost SATA LUN.
After rsyncing the mail partitions, imap/user, and imap/sieve
directories (several times over several days, with services stopped for
a final incremental sync early one morning), I just needed two minor
changes to mailboxes.db. Permissions s/lrswipcda/lrswipkxtea/g, and
prepend 0 to the the partition column (murder node, I guess).
#!/usr/bin/perl
# mbxlist.pl -- translate mailboxes list from cyrus 2.2 to 2.3 format
unlink mailboxes.db;
open OLD, ssh [EMAIL PROTECTED] /opt/cyrus/bin/ctl_mboxlist -d |;
open NEW, | /usr/lib/cyrus-imapd/ctl_mboxlist -u;
while (OLD) {
($mbox,$part,$acl) = $_ =~ m/^([^\t]+)\t([^\t]+)\t(.+)\n/;
$part =~ s/ +//g;
$part eq '0' and $part = 4;
$part eq 'default' and $part = 3;
$path = $mbox;
$path =~ s#\.#/#g;
@subdirs = split /\//, $path;
$path = /var/spool/imap${part}/$path;
if (! -e $path/cyrus.header) {
# There were some empty directories/invalid mailboxes on the
# old server, legacy of previous sysadmin errors. Skip those.
warn No $path for $mbox\n;
next;
}
$dir = shift @subdirs; # drop leading /user/
$start = /var/spool/imap-meta/$part/user;
while ($dir = shift @subdirs) {
$start = $start/$dir;
mkdir $start;
}
print join(\t,$mbox,$part,$acl) . \n;
$acl =~ 's/lrswipcda/lrswipkxtea/g';
print NEW join(\t,$mbox,0 $part,$acl) . \n;
$i++;
}
if ($i 32000) {
die FATAL: Too few mailboxes!\n;
} else {
exit 0;
}
--
Rich Graves [EMAIL PROTECTED]
Sr UNIX and Security Administrator
Ofc 507-646-7079 Cell 952-292-6529
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Sort of like rsync for IMAP, if that makes sense. So far, the only tool I've found is imapsync: http://freshmeat.net/projects/imapsync/ Anyone tried it with Cyrus? Good/bad experiences? I just used it to move from 2.1 to 2.3, there were a handful of messages it didn't like (~30 out of a few hundred thousand messages) but it appears to have worked well enough to fix the last few messages manually. most of the errors were cases of invalid headers that 2.3 wouldn't accept, but 2.1 obviously did. I used it to move from Exchange 5.5 to Cyrus 2.3. The tool works great but like David quoted, there where some messages rejected cause of broken headers (Microsoft?!). Michael Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
Florin Andrei wrote: Anyone tried it with Cyrus? Good/bad experiences? Are there any other tools that work better with Cyrus? I moved abotu 400Gig of mail from UW-imapd to cyrus a couple of months ago. Some messages where not synced because our old UW-imapd accepted broken messages. Rudy -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert [EMAIL PROTECTED] tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur ICT Department, Infrastructure office Groep SystemenSystems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
On Thu, 2006-12-21 at 15:47 -0800, Florin Andrei wrote: I'm currently using two IMAP accounts, one on Cyrus-2.2 the other on Cyrus-2.3. The one on Cyrus-2.3 will get decomissioned so I need to transfer all my email, preserving the folders/subfolders tree, under a specific folder (oldmail/foo/bar) on the 2.2 server. I need to do the bulk of the transfer sometime soon, then sync up again a few times after that, until the day the account on the 2.3 server gets nuked. Essentially, I need a tool that I can point at servers A and B and tell it get all the email from my account on server A to a specific folder on my account on server B, preserving the subfolders hierarchy. The tool needs to be smart enough to repeat the operation later on but then it must only transfer the new messages. The tool may run on one or the other IMAP servers, or even on a 3rd machine, since it should be network-based. Pretty much all systems are Linux 'round here, some Windows stragglers too. Sort of like rsync for IMAP, if that makes sense. So far, the only tool I've found is imapsync: http://freshmeat.net/projects/imapsync/ Anyone tried it with Cyrus? Good/bad experiences? Are there any other tools that work better with Cyrus? I've been using mailutil from the UW-IMAP suite, after seeing some recommendations for it on this list and with my distro. However, as far as I know, it won't automatically operate recursively over your mailboxes (it might: I haven't been looking for that feature). As some other messages I've posted attest, I've been surprised how slowly I can stuff messages into the cyrus server. My guess is the problem is with cyrus, but it might be with mailutil. Parallelizing the copies speeds things up quite a bit, and in my tests I get the best performance with ext3 partitions mounted noatime. I'll post more results when I finish trying out some other filesystems. -- Ross Boylan wk: (415) 514-8146 185 Berry St #5700 [EMAIL PROTECTED] Dept of Epidemiology and Biostatistics fax: (415) 514-8150 University of California, San Francisco San Francisco, CA 94107-1739 hm: (415) 550-1062 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
On Thu, Dec 21, 2006 at 03:47:26PM -0800, Florin Andrei wrote: Essentially, I need a tool that I can point at servers A and B and tell it get all the email from my account on server A to a specific folder on my account on server B, preserving the subfolders hierarchy. The tool needs to be smart enough to repeat the operation later on but then it must only transfer the new messages. The tool may run on one or the other IMAP servers, or even on a 3rd machine, since it should be network-based. Pretty much all systems are Linux 'round here, some Windows stragglers too. Sort of like rsync for IMAP, if that makes sense. So far, the only tool I've found is imapsync: http://freshmeat.net/projects/imapsync/ Anyone tried it with Cyrus? Good/bad experiences? Are there any other tools that work better with Cyrus? Another thing you might want to consider is offlineimap: http://software.complete.org/offlineimap/ It's a pile of multithreaded python, but don't let that put you off! I've found it quite robust for IMAP - Maildir usage (Indeed, I'm replying to this message via Mutt using local Maildirs which are kept synchronised with my FastMail (Cyrus 2.3.7+patches) IMAP account). Imap = Imap usage I found less robust in that it sometimes got confused when it had been killed in the middle of operations. If you didn't keep killing it all the time (my usage patterns were pretty strange) it was better - and also I think if you had a server which supports UID PLUS like Cyrus does then it would be safer. Give it a look though. apt-get install offlineimap works on pretty much any sane linux these days :) (I believe you can even get it with apt4rpm, though that's not my particular kink) Regards, Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
On Thu, 21 Dec 2006, Florin Andrei wrote: I'm currently using two IMAP accounts, one on Cyrus-2.2 the other on Cyrus-2.3. The one on Cyrus-2.3 will get decomissioned so I need to transfer all my email, preserving the folders/subfolders tree, under a specific folder (oldmail/foo/bar) on the 2.2 server. I need to do the bulk of the transfer sometime soon, then sync up again a few times after that, until the day the account on the 2.3 server gets nuked. Essentially, I need a tool that I can point at servers A and B and tell it get all the email from my account on server A to a specific folder on my account on server B, preserving the subfolders hierarchy. The tool needs to be smart enough to repeat the operation later on but then it must only transfer the new messages. The tool may run on one or the other IMAP servers, or even on a 3rd machine, since it should be network-based. Pretty much all systems are Linux 'round here, some Windows stragglers too. Sort of like rsync for IMAP, if that makes sense. So far, the only tool I've found is imapsync: http://freshmeat.net/projects/imapsync/ Anyone tried it with Cyrus? Good/bad experiences? I just used it to move from 2.1 to 2.3, there were a handful of messages it didn't like (~30 out of a few hundred thousand messages) but it appears to have worked well enough to fix the last few messages manually. most of the errors were cases of invalid headers that 2.3 wouldn't accept, but 2.1 obviously did. David Lang Are there any other tools that work better with Cyrus? Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
