Re: Cyrus murder ( IMAP Aggregator )
Hi Wesley, Thank you for your info. It gives me a clearer idea how it works. The multiple front-ends with the imap aggregator would give a huge advantage for expanding when more resources are needed, so this is an option i would like to use. The only issue i can think of is backups of the mailboxes. The planning is to have all the mailbox storage on a SAN. My idea is to have 1 replication slave which does the replication of all the other backend nodes. This replication slave uses an other SAN as storage. This way it should be easy to handle if the SAN dies or the filesystem gets corrupted from one of the backends. Each backend will replicate it's data to the replication slave on an own 'disk'. So that if the 'disk' on the main SAN is dead i can connect the backend to the other SAN. I hope this still makes any sense :-) Is this even a good idea? Thanks for your time. Best regards, Richard On Thu, 11 Feb 2010 10:25:26 -0500, Wesley Craig w...@umich.edu wrote: On 10 Feb 2010, at 16:54, Richard Pijnenburg wrote: How do the backends deceide who services which mailbox? The admin decides, more or less, as you provision mailboxes. And what happens if one of those backends dies? The mailboxes on that backend are inaccessible. The rest of the cluster continues to operate as normal, tho your mail queues might be negatively impacted. You might want to investigate replication, for how to recover from a failed backend. If I understand also correct I’ll need 1 Murder master and then 1 or multiple Murder Front-end and back-end servers ? More or less. It's pretty reasonable to have one of the frontends act as the murder master, particularly in a low-load environment. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Sieve and encoded Headers
Bron Gondwana wrote: On Mon, Feb 08, 2010 at 11:29:22AM -0500, Ken Murchison wrote: Bron Gondwana wrote: On Mon, Feb 08, 2010 at 07:53:21AM +0100, Garry wrote: Hi, after wondering for a while why occasionally my sieve script rules wouldn't work, I just found the reason (I guess) - Sieve doesn't (correctly?) decode utf encoded header lines which e.g. are in a format like this: Subject: =?utf-8?B?W0xPR10gV2F0Y2hsaXN0OiBzbWVhZ29sIGZvdW5kIEdydcOfIGF1cyBkZXIgVW50ZXJ3ZWx0IChVbmtub3duIENhY2hlKQ==?= I'm using version: Cyrus timsieved v2.2.13-Debian-2.2.13-19 ... is it something that is fixed in a newer version? I tried finding something on the net, but at least the first couple of results pages didn't yield any insight ... No - it's not fixed in any released version. It is fixed in the FastMail Cyrus patches - but it's a very invasive change to the charset encoding, so it's been kept out of the stable line for now. I've CC'd Ken on this - I wonder if it's worth going back and doing a minimal still compatible set of patches that fixes charset encoding in sieve without actually changing the on disk format of the cyrus.cache It might be worth doing this for 2.3. Done! I've put lots of testing in to it too :) Added to the cyrus-imapd-2_3-tail branch. Are you planning on re-comitting your charset changes to 2.4 soon? -- Kenneth Murchison Systems Programmer Carnegie Mellon University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Setting TCP keepalive for Cyrus daemons
I've been noticing idle pop3d processes on our Cyrus front end server for some time. These should be transient. One that was several days old had an established TCP connection to a wireless client that had disappeared. Presumably the client never closed the connection. Setting TCP keepalive on the file descriptor should permit the kernel to close the connection in this situation. Does this sound reasonable? Perhaps it's already been addressed in a later Cyrus version. We're running cyrus-imapd-2.3.8. I'm willing to add a `keepalive' option to Cyrus master along with the setsockopt() system call to enable that setting. This option could be added to the cyrus.conf file for any services that could benefit from it. Would this be a reasonable addition to Cyrus? -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Delays in collecting messages with Mac Mail
On Fri, 2010-02-05 at 09:54 +, Simon Fraser wrote: On Thu, 2010-02-04 at 16:27 -0500, Brian Awood wrote: We've seen somewhat similar behavior with the Apple Mail client in the past. My guess is it's IDLE related. If your using Apple Mail that came with 10.5, it's IDLE implementation appears to work at first. However it has a bug in it so that it stops working after a while. It seemed related to how it managed different connections to the server, using a different connection for each folder you opened. It is more reliable in the version that comes with OS X 10.6. We do mostly have 10.5 on our Macs, so I will ask a few people to disable the use IDLE option and see if that makes a difference. We've had no further reports of delayed email, so this does indeed seem to be the problem. Thanks again, Simon. -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Sieve and encoded Headers
On Fri, Feb 12, 2010 at 10:18:02AM -0500, Ken Murchison wrote: Bron Gondwana wrote: On Mon, Feb 08, 2010 at 11:29:22AM -0500, Ken Murchison wrote: Bron Gondwana wrote: On Mon, Feb 08, 2010 at 07:53:21AM +0100, Garry wrote: Hi, after wondering for a while why occasionally my sieve script rules wouldn't work, I just found the reason (I guess) - Sieve doesn't (correctly?) decode utf encoded header lines which e.g. are in a format like this: Subject: =?utf-8?B?W0xPR10gV2F0Y2hsaXN0OiBzbWVhZ29sIGZvdW5kIEdydcOfIGF1cyBkZXIgVW50ZXJ3ZWx0IChVbmtub3duIENhY2hlKQ==?= I'm using version: Cyrus timsieved v2.2.13-Debian-2.2.13-19 ... is it something that is fixed in a newer version? I tried finding something on the net, but at least the first couple of results pages didn't yield any insight ... No - it's not fixed in any released version. It is fixed in the FastMail Cyrus patches - but it's a very invasive change to the charset encoding, so it's been kept out of the stable line for now. I've CC'd Ken on this - I wonder if it's worth going back and doing a minimal still compatible set of patches that fixes charset encoding in sieve without actually changing the on disk format of the cyrus.cache It might be worth doing this for 2.3. Done! I've put lots of testing in to it too :) Added to the cyrus-imapd-2_3-tail branch. Are you planning on re-comitting your charset changes to 2.4 soon? Yes - I'll do that too :) I've finally worked out how to keep the different branches functioning in git, so I'm going to pull a 2.2 branch into git and do some work on that too. In particular I'd like to backport the skiplist bugfixes to 2.2 just so people stuck with it can at least have stable databases! I'm still planning to have the bulk of my changes ready by April - it's sneaking up faster than I thought, but deadlines are good for keeping us honest! I'd like to do a beta release in April anyway. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Setting TCP keepalive for Cyrus daemons
On Fri, Feb 12, 2010 at 09:45:02AM -0600, Gary Mills wrote: I've been noticing idle pop3d processes on our Cyrus front end server for some time. These should be transient. One that was several days old had an established TCP connection to a wireless client that had disappeared. Presumably the client never closed the connection. Setting TCP keepalive on the file descriptor should permit the kernel to close the connection in this situation. Does this sound reasonable? Perhaps it's already been addressed in a later Cyrus version. We're running cyrus-imapd-2.3.8. I'm willing to add a `keepalive' option to Cyrus master along with the setsockopt() system call to enable that setting. This option could be added to the cyrus.conf file for any services that could benefit from it. Would this be a reasonable addition to Cyrus? This is something I've been wanting to look in to as well. If a machine crashes, it can leave sync_clients hanging for ever, thinking they are still talking to the replica - meaning that they hold the lock and replication doesn't start back up. Quite annoying. Is there any reason to make it an option rather than just always having it on? Or at least not to make it the default. If it's a good idea, it SHOULD be the default. I'm strongly against having hundreds of lines of config file required to get the sanest defaults! Hmm... oh: http://tools.ietf.org/html/rfc1122#page-101 Implementors MAY include keep-alives in their TCP implementations, although this practice is not universally accepted. If keep-alives are included, the application MUST be able to turn them on or off for each TCP connection, and they MUST default to off. Keep-alive packets MUST only be sent when no data or acknowledgement packets have been received for the connection within an interval. This interval MUST be configurable and MUST default to no less than two hours. Sounds like it does need to be configurable! Also, the defaults are crazy high amounts for a local reliable network. I think I'd prefer about 5 minutes than 2 hours! Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Setting TCP keepalive for Cyrus daemons
Shouldn't these client connections already be handled by the poptimeout timeout options? unless you have it set to zero... We have had problems within the murder (old code had several spots where murder front - back communications could deadlock).. On Sat, 13 Feb 2010, Bron Gondwana wrote: On Fri, Feb 12, 2010 at 09:45:02AM -0600, Gary Mills wrote: I've been noticing idle pop3d processes on our Cyrus front end server for some time. These should be transient. One that was several days old had an established TCP connection to a wireless client that had disappeared. Presumably the client never closed the connection. Setting TCP keepalive on the file descriptor should permit the kernel to close the connection in this situation. Does this sound reasonable? Perhaps it's already been addressed in a later Cyrus version. We're running cyrus-imapd-2.3.8. I'm willing to add a `keepalive' option to Cyrus master along with the setsockopt() system call to enable that setting. This option could be added to the cyrus.conf file for any services that could benefit from it. Would this be a reasonable addition to Cyrus? This is something I've been wanting to look in to as well. If a machine crashes, it can leave sync_clients hanging for ever, thinking they are still talking to the replica - meaning that they hold the lock and replication doesn't start back up. Quite annoying. Is there any reason to make it an option rather than just always having it on? Or at least not to make it the default. If it's a good idea, it SHOULD be the default. I'm strongly against having hundreds of lines of config file required to get the sanest defaults! Hmm... oh: http://tools.ietf.org/html/rfc1122#page-101 Implementors MAY include keep-alives in their TCP implementations, although this practice is not universally accepted. If keep-alives are included, the application MUST be able to turn them on or off for each TCP connection, and they MUST default to off. Keep-alive packets MUST only be sent when no data or acknowledgement packets have been received for the connection within an interval. This interval MUST be configurable and MUST default to no less than two hours. Sounds like it does need to be configurable! Also, the defaults are crazy high amounts for a local reliable network. I think I'd prefer about 5 minutes than 2 hours! Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Setting TCP keepalive for Cyrus daemons
On Fri, Feb 12, 2010 at 03:59:31PM -0600, Paul M Fleming wrote: Shouldn't these client connections already be handled by the poptimeout timeout options? unless you have it set to zero... They don't seem to be. We're using the default timeout setting. It seems to have no effect on front end daemons. Here's a stack trace on one that's several days old: # pstack 12708 12708: pop3d -s feb1a5c5 read (0, 817faf0, b) fec2dfaf sock_read () + 3f We have had problems within the murder (old code had several spots where murder front - back communications could deadlock).. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
