Re: IMAP sync tool (rsync for IMAP)
--syncinternaldates stopped working at imapsync 1.200 (current is 1.204), as a side effect of the append_string-append_file change suggested by John Capo and Florin Andrei earlier in this thread, on Dec 26, 2006. That change was made to avoid memory exhaustion, see http://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg31296.html The underlying problem is that at least as of Mail::IMAPClient 2.2.9, the append_file function doesn't handle INTERNALDATE at all. A bunch of people here have talked about minor patches to Mail::IMAPClient. Has anyone added the $d argument to append_file and verified functionality? Most clients display Date: header, but some may show or use INTERNALDATE. For example, the Zimbra web interface is incapable of showing or sorting by Date:. See also http://www.zimbra.com/forums/showthread.php?p=33880#post33880 I already mailed the author of imapsync about that some days ago but without having an idea how to fix it. I also checked Mail::IMAPClient and found that it lacked the functionality needed here but it seems Mail::IMAPClient isn't maintained anymore (at least there is no development for long time now). Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Delayed expunge / undeleting messages
A bunch of people already pointed out unexpunge, so I'll point out that the delayed expunge / unexpunge functionality doesn't help if the user deletes a whole folder. See: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2871 for the problem report and a patch for a proposed solution. I also found out that delayed_expunge doesn't help if a whole folder got deleted. So if something like your proposed solution gets included that would really be a very nice feature. Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: DB type comparison
On 19.01.2007, at 01:02, Blake Hudson wrote: I know I've seen it before, but I've been having a hard time finding a good comparison of the different db types supported by cyrus and their respective strengths/weaknesses. Does anyone have a link they could share? There's a short information about all the supported backends and their features in the wiki: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/WhatDatabaseBackend Hope that's helps, Baltasar Baltasar Cevc _ FORMER 03 GmbH _ infanteriestraße 19 haus 6 eg _ D-80797 muenchen _ http://www.former03.de Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Delayed expunge / undeleting messages
https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2871 I also found out that delayed_expunge doesn't help if a whole folder got deleted. So if something like your proposed solution gets included that would really be a very nice feature. Agreed. This would also allow for complete backups of all emails and all folders. Currently there's a window of emails uploaded/delivered between the last backup and a folder being deleted that can never be restored. I know David Carter implemented a patch for this in 2.2. I'm not sure if Wesley's patch is related to it or a completely separate implementation. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
I already mailed the author of imapsync about that some days ago but without having an idea how to fix it. I also checked Mail::IMAPClient and found that it lacked the functionality needed here but it seems Mail::IMAPClient isn't maintained anymore (at least there is no development for long time now). I don't know how much work it would be, but you could look at re-implementing using Mail::IMAPTalk. It's the IMAP client I built for use at FastMail.FM after finding problems with the other ones available at the time (about 4-5 years ago now). It definitely allows capturing a literal response into a file handle, and sending a file handle as a literal response to avoid storing everything in memory. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: using skiplist for quotas.db
so I wonder, will skiplist be a better choice? obviously running quota(8) will be a very cheap operation, but I'm worried about contention on the quota database during delivery etc. (these users are for the most part not actively using the system -- they get less than one message per day each(!) -- so I probably worry too much.) The problem I saw was that accessing a skiplist db involves locking the entire file I believe. This felt like a huge point of contention, especially since every email delivered would lock it. I never did end up trying it out to test that though, so I wouldn't quote me on that, Ken could confirm. What filesystem are you using? Have you considered trying reiserfs with tails enabled. It's quite impressive how little disk space it actually uses (don't believe du output). We used to use NVRAM cards, and even with 100,000 users, it was only about 30 megabytes for the quota files if memory serves me right (300 bytes each for directory entry, stat data, and file data), and I imagine the entire thing ends up in the OS buffer cache in pretty short order... Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: using skiplist for quotas.db
Rob Mueller wrote: so I wonder, will skiplist be a better choice? obviously running quota(8) will be a very cheap operation, but I'm worried about contention on the quota database during delivery etc. (these users are for the most part not actively using the system -- they get less than one message per day each(!) -- so I probably worry too much.) The problem I saw was that accessing a skiplist db involves locking the entire file I believe. This felt like a huge point of contention, especially since every email delivered would lock it. I never did end up trying it out to test that though, so I wouldn't quote me on that, Ken could confirm. This is true. If you want to use a single database, I would consider using BDB (as painful as it sounds), since it does localized locking. I don't believe that either choice has been tested thoroughly however. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: using skiplist for quotas.db
On Fri, 2007-01-19 at 21:58 +1100, Rob Mueller wrote: What filesystem are you using? Have you considered trying reiserfs with tails enabled. It's quite impressive how little disk space it actually uses (don't believe du output). We used to use NVRAM cards, and even with 100,000 users, it was only about 30 megabytes for the quota files if memory serves me right (300 bytes each for directory entry, stat data, and file data), and I imagine the entire thing ends up in the OS buffer cache in pretty short order... we're using ext3. reiserfs is not an option, thank-you-very-much. it has hosed our spool way too many times. an FS which can't handle hardware failure (e.g. a flaky fiber cable) should be taken out and shot. I appreciate your taking the time to make suggestions, though :-) -- Kjetil T. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: using skiplist for quotas.db
On Fri, 2007-01-19 at 07:49 -0500, Ken Murchison wrote: Rob Mueller wrote: The problem I saw was that accessing a skiplist db involves locking the entire file I believe. This felt like a huge point of contention, especially since every email delivered would lock it. I never did end up trying it out to test that though, so I wouldn't quote me on that, Ken could confirm. This is true. If you want to use a single database, I would consider using BDB (as painful as it sounds), since it does localized locking. I don't believe that either choice has been tested thoroughly however. hmmm. *reads cyrusdb_skiplist.c* there's an interesting comment in mycommit about moving the unlock to before the fsync. that would be a very nice performance hack, but should preferably be settable per database, e.g. adding a skiplist_nosync to mimic berkeley_nosync. after all, the quota database isn't critical information, it's updated from an authoritative source, anyway. I'll see if I can muster the courage to try this out. the worst that can happen is that users don't have any quota limits at all, right? :) thanks for the input, Rob and Ken. -- Kjetil T. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: using skiplist for quotas.db
Or things get mangled and people end up over quota. (from my Treo) -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University -Original Message- From: Kjetil Torgrim Homme [EMAIL PROTECTED] To: Ken Murchison [EMAIL PROTECTED] Cc: Rob Mueller [EMAIL PROTECTED]; info-cyrus@lists.andrew.cmu.edu Sent: 1/19/07 9:42 AM Subject: Re: using skiplist for quotas.db On Fri, 2007-01-19 at 07:49 -0500, Ken Murchison wrote: Rob Mueller wrote: The problem I saw was that accessing a skiplist db involves locking the entire file I believe. This felt like a huge point of contention, especially since every email delivered would lock it. I never did end up trying it out to test that though, so I wouldn't quote me on that, Ken could confirm. This is true. If you want to use a single database, I would consider using BDB (as painful as it sounds), since it does localized locking. I don't believe that either choice has been tested thoroughly however. hmmm. *reads cyrusdb_skiplist.c* there's an interesting comment in mycommit about moving the unlock to before the fsync. that would be a very nice performance hack, but should preferably be settable per database, e.g. adding a skiplist_nosync to mimic berkeley_nosync. after all, the quota database isn't critical information, it's updated from an authoritative source, anyway. I'll see if I can muster the courage to try this out. the worst that can happen is that users don't have any quota limits at all, right? :) thanks for the input, Rob and Ken. -- Kjetil T. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: DB type comparison
Thanks, this is exactly what I was looking for. Now, can someone please explain what the annotations and pts databases are used for? I've had some locking issues with the deliver.db and would like to switch all of the berkeley databases over to skiplist. Once this is completed will there still be a berkeley db environment stored in db? Also, are there any foreseeable problems making this move? Thanks, -Blake FORMER 03 | Baltasar Cevc wrote: On 19.01.2007, at 01:02, Blake Hudson wrote: I know I've seen it before, but I've been having a hard time finding a good comparison of the different db types supported by cyrus and their respective strengths/weaknesses. Does anyone have a link they could share? There's a short information about all the supported backends and their features in the wiki: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/WhatDatabaseBackend Hope that's helps, Baltasar Baltasar Cevc _ FORMER 03 GmbH _ infanteriestraße 19 haus 6 eg _ D-80797 muenchen _ http://www.former03.de Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Can murder be used for IMAP server migration?
Gary Mills wrote: On Thu, Jan 04, 2007 at 07:47:54AM -0500, Ken Murchison wrote: Gary Mills wrote: On Wed, Jan 03, 2007 at 08:18:15AM -0500, Ken Murchison wrote: Gary Mills wrote: Can I use the old server as both a front end and one of the back ends for a murder configuration, with the new server as the second back end? Will that allow me to migrate mailboxes at my convenience? How do I prevent a port conflict between the IMAP server and the proxy on the old server? You'll have to run the frontend + mupdate master on a separate machine. so the frontend has to run there. Of course, the old IMAP server has to run there too. Is there not a way to have the old IMAP server listen on different ports, so that only the frontend connects to it? If not, could it listen on a secondary IP address only? It *might* make sense to upgrade the existing server to 2.3.x, since imapd can both proxy and serve local mailboxes (a unified Murder). Is there any documentation on how to configure and manage a unified Murder? I couldn't find anything in the 2.3.7 source that described Bascially build your config using both the backend and frontend instructions and set mupdate_config:unified it. In fact, the examples in the documentation still show the proxy daemons, which no longer exist. True, but we make hardlinks from proxyd - imapd, etc so that older cyrus.conf continue to work as expected. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Can murder be used for IMAP server migration?
On 18 Jan 2007, at 20:16, Gary Mills wrote: Is there any documentation on how to configure and manage a unified Murder? See the man page for imapd.conf: mupdate_config: standard The configuration of the mupdate servers in the Cyrus Murder. The standard config is one in which there are discreet frontend (proxy) and backend servers. The unified config is one in which a server can be both a frontend and backend. The replicated config is one in which multiple backend servers all share the same mailspool, but each have their own replicated copy of mail- boxes.db. Allowed values: standard, unified, replicated AFAIK, that's all there is to it. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Delayed expunge / undeleting messages
On 19 Jan 2007, at 05:50, Rob Mueller wrote: I know David Carter implemented a patch for this in 2.2. I'm not sure if Wesley's patch is related to it or a completely separate implementation. Completely separate, tho my is based on comment from David suggesting how he would do it if he were to do it again, i.e., very simply. I like simple, it's easier to write :) :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
