Re: IMAP Date-Time format for Append
We're pretty sure this is related to the timezone being represented as a short name, as opposed to in numeric format (+0400, etc.). The IMAP spec is vague on whether or not this format should be accepted. I believe that this has to do with the way the function from the C Library converts the string, but am not sure. The spec isn't vague at all: http://www.faqs.org/rfcs/rfc3501.html append = APPEND SP mailbox [SP flag-list] [SP date-time] SP literal date-time = DQUOTE date-day-fixed - date-month - date-year SP time SP zone DQUOTE zone= (+ / -) 4DIGIT ; Signed four-digit value of hhmm representing ; hours and minutes east of Greenwich (that is, ; the amount that the given time differs from ; Universal Time). Subtracting the timezone ; from the given time will give the UT form. ; The Universal Time zone is +. Seems pretty clear that what the client is doing is broken. Having said that, the parsing code is in cyrus, so changing the C library won't help. You'll need to dig into the cyrus source.Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
We have a moderately sized Cyrus installation with 2 TB of storage and a few thousand simultaneous IMAP sessions. When one of the backup processes is running during the day, there's a noticable slowdown in IMAP client performance. When I start my `mutt' mail reader, it pauses for several seconds at `Selecting INBOX'. That behavior disappears when the backup finishes. Where's the first place to look for this problem? I/O statistics show a higher read bandwidth while the backup is running, but writes still dominate. The backup would typically read all of the files in a single Cyrus partition. Some more information about your setup would be helpful. 1. What OS? 32 or 64bit? 2. How much RAM do you have? How much is used by cache? 3. What sort of drive system is the data stored on? 4. Do you have split data/meta data? 5. What filesystem are you using? 6. What mount options to you have on your volumnes? 7. What IO scheduler are you using? 8. What version of cyrus are you using? 9. Do you have the cyrus status cache enabled? Some things to try if you haven't: 1. Make sure you have noatime (and nodiratime if applicable) set on your FS 2. If on linux, use the deadline IO scheduler 3. If you can, split your data and meta data onto separate volumes (probably a lot of work) 4. Install more RAM and use a 64-bit OS. x64_64 linux can cache many, many more inodes and dentries in memory than x86 linux. 5. Upgrade to the latest cyrus and enable the status cache Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Sat, 27 Sep 2008 13:46:39 +1000, Rob Mueller [EMAIL PROTECTED] said: Some things to try if you haven't: 1. Make sure you have noatime (and nodiratime if applicable) set on your FS 2. If on linux, use the deadline IO scheduler 3. If you can, split your data and meta data onto separate volumes (probably a lot of work) 4. Install more RAM and use a 64-bit OS. x64_64 linux can cache many, many more inodes and dentries in memory than x86 linux. 5. Upgrade to the latest cyrus and enable the status cache 6. Don't fetch every single file from your cyrus server every backup run. One nice property of cyrus's on-disk format is that all the data files are immutable once created: b/user/brong/798. is always going to be the same file, no matter what happens to the mailbox. Only the cyrus.* files and new data file actually need to be fetched. (our system also stats the cyrus.* files, and uses some other funky tricks to ensure consistent backups with minimum IO) -- Bron Gondwana [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Fri, Sep 26, 2008 at 09:19:25AM -0700, Rob Banz wrote: On Sep 26, 2008, at 06:44, Gary Mills wrote: We have a moderately sized Cyrus installation with 2 TB of storage and a few thousand simultaneous IMAP sessions. When one of the backup processes is running during the day, there's a noticable slowdown in IMAP client performance. When I start my `mutt' mail reader, it pauses for several seconds at `Selecting INBOX'. That behavior disappears when the backup finishes. As always, the answer is probably complicated. Of course. I just wanted to know what the usual suspects might be. What's your storage backend look like? Some sort of RAID with cache? The backup is very probably blowing out your cache while running, which can manifest itself in ways that will make it look like the performance profile of your storage has taken on a different personality. Especially with respect to writes, since you might usually be blessed with an abundance of write-back cache that's now fighting with all the reads for the backup. Do you have your cyrus partitions mounted with noatime? Your backups could be causing fs write operations that you really don't want. Our storage backend is four Iscsi LUNs from our Netapp filer. It will indeed be RAID with cache. Yes, the backup certainly would run the cache, and that may well be the problem. There will also be a large memory cache on the IMAP server. That could be upset as well. I'll see if I can find some cache statistics for both places. `noatime' is also certainly a good idea. I haven't yet tried it. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Fri, Sep 26, 2008 at 01:37:30PM -0400, Wesley Craig wrote: I'd probably use imtest to connect, get the PID of the server process that I'm connected to, and then attach to that process with ktrace (or whatever) with timestamps enabled. Then I'd select the mailbox -- this is assuming that mutt is only issuing a select when it says Selecting INBOX. Obviously it could be doing any number of things. You can get positive confirmation of which command is taking a long time by enabling telemetry, of course. Thanks for the suggestions. I believe I'll start with the telemetry, to get an idea what mutt is doing when it pauses. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Sat, Sep 27, 2008 at 01:46:39PM +1000, Rob Mueller wrote: Where's the first place to look for this problem? I/O statistics show a higher read bandwidth while the backup is running, but writes still dominate. The backup would typically read all of the files in a single Cyrus partition. Some more information about your setup would be helpful. I deliberately omitted that information, hoping that somebody would know what happens on the server side when mutt says `selecting INBOX'. I don't want to redesign the server at this stage, at least not until I've isolated the problem. 1. Make sure you have noatime (and nodiratime if applicable) set on your FS 2. If on linux, use the deadline IO scheduler 3. If you can, split your data and meta data onto separate volumes (probably a lot of work) 4. Install more RAM and use a 64-bit OS. x64_64 linux can cache many, many more inodes and dentries in memory than x86 linux. 5. Upgrade to the latest cyrus and enable the status cache Those are all good suggestions. I'll investigate some of them. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Sun, Sep 28, 2008 at 09:42:27AM +1000, Bron Gondwana wrote: On Sat, 27 Sep 2008 13:46:39 +1000, Rob Mueller [EMAIL PROTECTED] said: Some things to try if you haven't: 6. Don't fetch every single file from your cyrus server every backup run. One nice property of cyrus's on-disk format is that all the data files are immutable once created: b/user/brong/798. is always going to be the same file, no matter what happens to the mailbox. Only the cyrus.* files and new data file actually need to be fetched. (our system also stats the cyrus.* files, and uses some other funky tricks to ensure consistent backups with minimum IO) Most of our restores are done from ZFS snapshots. We keep two weeks of them. That's worked out extremely well. We use Networker with tapes for longer-term backups. It's that that seems to cause the problem. Certainly our Networker backup system has no knowledge of the structure and properties of the Cyrus IMAP spool. It would just read every file for a full backup, or look at timestamps for an incremental backup. I don't know if much can be changed in the way it operates. However, we may be able to change the way it takes backups if contention is really the problem. I'm not convinced that it is just now. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
