Re: Stuck on 97% changing mailboxes
FYI, since we didn't have a bug in trac for this problem, I went ahead and created one: http://dev.mutt.org/trac/ticket/3459 me
Re: Stuck on 97% changing mailboxes
On 23 Sep 2010, at 23:13, Michael Elkins wrote: Yes, this is the problem. Mutt expects to see a FETCH response for each message the server says EXISTS. The IMAP standard requires that no holes exist in the message sequence numbers, and mutt is not prepared to handle them. I hope it's as simple as that, since that does seem like something mutt could work around (at least in principle). I'm a little sceptical, however, since not only do Apple Mail and Thunderbird not hang, but they actually show all the messages in these problem mailboxes. There are no missing messages, as evidenced by the message totals and the message #s shown in their GUIs matching expectations (and being large!). I've done a bit more testing on this Exchange 2010 server and established based on mutt debug logs that the problem is with mailboxes with greater than 2046 messages. If an IMAP mailbox contains 2047 or more messages, the list of FETCH numbers in .muttdebug0 has missing entries, often several dozen (e.g. there were about 90 missing in a mailbox of ~4000 messages, about 40 in a mailbox of ~2000). Curiously, FETCH 1 is always missing. On the other hand, if a mailbox contains 2046 or fewer messages, the the list of FETCH numbers is contiguous, it includes FETCH 1, and the last number matches the number given by EXISTS. Hope that helps. Please let me know if there's any more testing I can do. In the meantime, I've used Thunderbird to divide all my mailboxes up into smaller ones!
Re: Stuck on 97% changing mailboxes
Hi Michael!
On Do, 23 Sep 2010, Michael Williams wrote:
4 * 3700 FETCH (UID 17146 FLAGS (\Seen) INTERNALDATE 23-Sep-2010 09:14:57
+0100 RFC822.SIZE 2612 BODY[HEADER.FIELDS (DATE FROM SUBJECT TO CC
MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO REPLY-TO
LINES LIST-POST X-LABEL)] {404}
4 * 3702 FETCH (UID 17148 FLAGS (\Seen) INTERNALDATE 23-Sep-2010 14:35:54
+0100 RFC822.SIZE 20548 BODY[HEADER.FIELDS (DATE FROM SUBJECT TO CC
MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO REPLY-TO
LINES LIST-POST X-LABEL)] {408}
Here is one missing.
regards,
Christian
Re: Stuck on 97% changing mailboxes
On 23 Sep 2010, at 19:17, Christian Brabandt wrote:
4 * 3700 FETCH (UID 17146 FLAGS (\Seen) INTERNALDATE 23-Sep-2010 09:14:57
+0100 RFC822.SIZE 2612 BODY[HEADER.FIELDS (DATE FROM SUBJECT TO CC
MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO REPLY-TO
LINES LIST-POST X-LABEL)] {404}
4 * 3702 FETCH (UID 17148 FLAGS (\Seen) INTERNALDATE 23-Sep-2010 14:35:54
+0100 RFC822.SIZE 20548 BODY[HEADER.FIELDS (DATE FROM SUBJECT TO CC
MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO REPLY-TO
LINES LIST-POST X-LABEL)] {408}
Here is one missing.
Is this the problem? Is there a workaround or is the conclusion that mutt is
not compatible with Exchange 2010 IMAP's support (or Exchange 2010 is not
compatible with mutt)?
Re: Stuck on 97% changing mailboxes
On Thu, Sep 23, 2010 at 10:19:28PM +0200, Michael Williams wrote:
On 23 Sep 2010, at 19:17, Christian Brabandt wrote:
4 * 3700 FETCH (UID 17146 FLAGS (\Seen) INTERNALDATE 23-Sep-2010 09:14:57
+0100 RFC822.SIZE 2612 BODY[HEADER.FIELDS (DATE FROM SUBJECT TO CC MESSAGE-ID
REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO REPLY-TO LINES LIST-POST X-LABEL)]
{404}
4 * 3702 FETCH (UID 17148 FLAGS (\Seen) INTERNALDATE 23-Sep-2010 14:35:54
+0100 RFC822.SIZE 20548 BODY[HEADER.FIELDS (DATE FROM SUBJECT TO CC MESSAGE-ID
REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO REPLY-TO LINES LIST-POST X-LABEL)]
{408}
Here is one missing.
Is this the problem? Is there a workaround or is the conclusion that mutt is
not compatible with Exchange 2010 IMAP's support (or Exchange 2010 is not
compatible with mutt)?
Yes, this is the problem. Mutt expects to see a FETCH response for each
message the server says EXISTS. The IMAP standard requires that no holes
exist in the message sequence numbers, and mutt is not prepared to handle
them.
me
Re: Stuck on 97% changing mailboxes
On Thursday, 23 September 2010 at 14:13, Michael Elkins wrote:
On Thu, Sep 23, 2010 at 10:19:28PM +0200, Michael Williams wrote:
On 23 Sep 2010, at 19:17, Christian Brabandt wrote:
4 * 3700 FETCH (UID 17146 FLAGS (\Seen) INTERNALDATE 23-Sep-2010
09:14:57 +0100 RFC822.SIZE 2612 BODY[HEADER.FIELDS (DATE FROM SUBJECT TO
CC MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO
REPLY-TO LINES LIST-POST X-LABEL)] {404}
4 * 3702 FETCH (UID 17148 FLAGS (\Seen) INTERNALDATE 23-Sep-2010
14:35:54 +0100 RFC822.SIZE 20548 BODY[HEADER.FIELDS (DATE FROM SUBJECT TO
CC MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO
REPLY-TO LINES LIST-POST X-LABEL)] {408}
Here is one missing.
Is this the problem? Is there a workaround or is the conclusion that mutt is
not compatible with Exchange 2010 IMAP's support (or Exchange 2010 is not
compatible with mutt)?
Yes, this is the problem. Mutt expects to see a FETCH response for
each message the server says EXISTS. The IMAP standard requires that
no holes exist in the message sequence numbers, and mutt is not
prepared to handle them.
We might be able to work around this by creating an empty message
marked as expunged for the holes, then running the expunge compactor
after the command finishes. But it does seem like there's probably
something wrong with your mailbox on the server side. It might go away
if the exchange server administrator runs some kind of maintenance
tool on the problematic mailbox.
Re: Stuck on 97% changing mailboxes
On Thu, Sep 23, 2010 at 02:13:05PM -0700, Michael Elkins wrote: Yes, this is the problem. Mutt expects to see a FETCH response for each message the server says EXISTS. The IMAP standard requires that no holes exist in the message sequence numbers, and mutt is not prepared to handle them. Mutt could be a lot more asynchronous though. Mutt could display a page-ful of messages, filling in slots (re-drawing as necessary) as the FETCH responses come in. Right now mutt wants to enumerate a mailbox when opening it. It has to for mbox/mh/maildir folders, given their nature. But IMAP is designed to allow for listing page-fuls of messages at a time instead of enumerating folders -- the fact that mutt doesn't do this has me sad. IMAP is also designed to allow multiple concurrent requests, that is, it's designed to allow for asynchronous operation. I've looked at mutt source code, and ISTM that it'd require massive surgery to add support for using IMAP this way. Nico --
Re: Stuck on 97% changing mailboxes
On Thu, Sep 23, 2010 at 02:32:54PM -0700, Brendan Cully wrote:
On Thursday, 23 September 2010 at 14:13, Michael Elkins wrote:
On Thu, Sep 23, 2010 at 10:19:28PM +0200, Michael Williams wrote:
On 23 Sep 2010, at 19:17, Christian Brabandt wrote:
4 * 3700 FETCH (UID 17146 FLAGS (\Seen) INTERNALDATE 23-Sep-2010
09:14:57 +0100 RFC822.SIZE 2612 BODY[HEADER.FIELDS (DATE FROM SUBJECT
TO CC MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO
REPLY-TO LINES LIST-POST X-LABEL)] {404}
4 * 3702 FETCH (UID 17148 FLAGS (\Seen) INTERNALDATE 23-Sep-2010
14:35:54 +0100 RFC822.SIZE 20548 BODY[HEADER.FIELDS (DATE FROM SUBJECT
TO CC MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO
REPLY-TO LINES LIST-POST X-LABEL)] {408}
Here is one missing.
Is this the problem? Is there a workaround or is the conclusion that mutt
is not compatible with Exchange 2010 IMAP's support (or Exchange 2010 is
not compatible with mutt)?
Yes, this is the problem. Mutt expects to see a FETCH response for
each message the server says EXISTS. The IMAP standard requires that
no holes exist in the message sequence numbers, and mutt is not
prepared to handle them.
We might be able to work around this by creating an empty message
marked as expunged for the holes, then running the expunge compactor
after the command finishes. But it does seem like there's probably
something wrong with your mailbox on the server side. It might go away
if the exchange server administrator runs some kind of maintenance
tool on the problematic mailbox.
That's a good idea -- we know how many messages there are, so pretend we
know them all and fill in the details as the responses come in. The
only other part of the puzzle: how to handle curses input and IMAP async
I/O? I think doing this with minimal surgery might require a thread to
poll for curses input and another to handle IMAP responses.
Nico
--
Re: Stuck on 97% changing mailboxes
On Thu, Sep 23, 2010 at 04:38:33PM -0500, Nicolas Williams wrote: On Thu, Sep 23, 2010 at 02:13:05PM -0700, Michael Elkins wrote: Yes, this is the problem. Mutt expects to see a FETCH response for each message the server says EXISTS. The IMAP standard requires that no holes exist in the message sequence numbers, and mutt is not prepared to handle them. Mutt could be a lot more asynchronous though. Mutt could display a page-ful of messages, filling in slots (re-drawing as necessary) as the FETCH responses come in. Right now mutt wants to enumerate a mailbox when opening it. It has to for mbox/mh/maildir folders, given their nature. But IMAP is designed to allow for listing page-fuls of messages at a time instead of enumerating folders -- the fact that mutt doesn't do this has me sad. IMAP is also designed to allow multiple concurrent requests, that is, it's designed to allow for asynchronous operation. I've looked at mutt source code, and ISTM that it'd require massive surgery to add support for using IMAP this way. The main issue is that some popular IMAP servers (gmail, exchange), do not support the SORT extensions, so you wouldn't be able to do the pageful-at-a-time and still have all of Mutt's current threading capabilities. You are correct that major surgery would be required to change Mutt. A few years back I wrote a patch to lazily load the message headers, but the lack of threading support made it mostly useless. me
Re: Stuck on 97% changing mailboxes
On Thu, Sep 23, 2010 at 02:45:47PM -0700, Michael Elkins wrote: The main issue is that some popular IMAP servers (gmail, exchange), do not support the SORT extensions, so you wouldn't be able to do the pageful-at-a-time and still have all of Mutt's current threading capabilities. Interesting. I'd still like to open the folder and see it in the one order quickly, with the enumeration happening in the background, so that eventually I could sort it any way I like. You are correct that major surgery would be required to change Mutt. A few years back I wrote a patch to lazily load the message headers, but the lack of threading support made it mostly useless. Are you referring to pthreads or mail therading? Nico --
Re: Stuck on 97% changing mailboxes
On Thu, Sep 23, 2010 at 04:52:37PM -0500, Nicolas Williams wrote: Are you referring to pthreads or mail therading? Mail threading. me
Re: Stuck on 97% changing mailboxes
On Thu, Sep 23, 2010 at 02:58:56PM -0700, Michael Elkins wrote: On Thu, Sep 23, 2010 at 04:52:37PM -0500, Nicolas Williams wrote: Are you referring to pthreads or mail therading? Mail threading. That's OK. I'd be happy to live with that, since eventually the folder does get fully enumerated. And if you have a server that doesn't answer some FETCHes, as in this case, it'd be a perfect workaround.
Re: Stuck on 97% changing mailboxes
On Thu, Sep 23, 2010 at 19:09:43 +0200, Michael Williams wrote:
$ grep 'FETCH' .muttdebug0 | tail
[...]
4 * 3709 FETCH (UID 17155 FLAGS (\Seen) INTERNALDATE 20-Sep-2010 17:19:34
+0100 RFC822.SIZE 126689 BODY[HEADER.FIELDS (DATE FROM SUBJECT TO CC
MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO REPLY-TO
LINES LIST-POST X-LABEL)] {451}
4 a0011 OK FETCH completed.
With mutt itself stuck at Fetching message headers... 3630/3709 (97%).
On Thu, Sep 23, 2010 at 14:13:05 -0700, Michael Elkins wrote:
Yes, this is the problem. Mutt expects to see a FETCH response for each
message the server says EXISTS. The IMAP standard requires that no holes
exist in the message sequence numbers, and mutt is not prepared to handle
them.
As someone who doesn't know any details about the IMAP standard or
Mutt's internal workings, I'm curious to know what happens once Mutt
sees the OK message from the server in this situation.
Is there any way it could use that message to at least abort the fetch
operation with some error message, rather than hanging?
Nathan
Re: Stuck on 97% changing mailboxes
On 20 Sep 2010, at 12:30, Michael Williams wrote: Hi, following on from problems getting a new Exchange 2010 server to play nice with mutt's SMTP support (http://marc.info/?l=mutt-usersm=128493280217503w=2), I have now encountered a problem with IMAP access to the same server. This problem is unique to mutt and is not displayed by other mail clients (which proves that I haven't missed something completely basic, but does not necessarily mean the problem is mutt and not the server!) I can connect fine, and read the INBOX fine, and I seem to be able to change into small mailboxes 2000 messages. By change I mean, I can select the mailbox in mutt and after a delay while it downloads the message headers, the mutt list view shows the contents of the folder. However, for larger mailboxes, when I attempt to change, mutt only gets as far as saying Fetching message headers... 3320/3391 (97%) which remains unchanged for ~5 minutes before I give up. Curiously, the .muttdebug log looks normal. The last few lines are Date: Mon, 20 Sep 2010 10:18:35 +0100 4 ) parse_parameters: `charset=US-ASCII; format=flowed; delsp=yes' parse_parameter: `charset' = `US-ASCII' parse_parameter: `format' = `flowed' parse_parameter: `delsp' = `yes' 4 a0011 OK FETCH completed. IMAP queue drained (where that Date is the date of the most recent email in that mailbox). The other data point is the fact that Thunderbird has no problem reading these mailboxes: it downloads the headers, displays the full message list, and reads the contents of the emails to generate its search index, all without any unusual delay. The problem seems to be 97% every time. And there does seem to be something significant about mailbox size -- I haven't found a mailbox with fewer than 2000 messages with this problem, and I haven't found a mailbox larger than that without it. But this is obviously based on limited statistics and it's difficult to test. I have disabled header and body caching in mutt. No change. Any ideas? Is there a switch I can throw in mutt? No suggestions?
Re: Stuck on 97% changing mailboxes
On Mon, Sep 20, 2010 at 12:30:18PM +0200, Michael Williams wrote: following on from problems getting a new Exchange 2010 server to play nice with mutt's SMTP support (http://marc.info/?l=mutt-usersm=128493280217503w=2), I have now encountered a problem with IMAP access to the same server. This problem is unique to mutt and is not displayed by other mail clients (which proves that I haven't missed something completely basic, but does not necessarily mean the problem is mutt and not the server!) I can connect fine, and read the INBOX fine, and I seem to be able to change into small mailboxes 2000 messages. By change I mean, I can select the mailbox in mutt and after a delay while it downloads the message headers, the mutt list view shows the contents of the folder. However, for larger mailboxes, when I attempt to change, mutt only gets as far as saying Fetching message headers... 3320/3391 (97%) which remains unchanged for ~5 minutes before I give up. Curiously, the .muttdebug log looks normal. The last few lines are Date: Mon, 20 Sep 2010 10:18:35 +0100 4 ) parse_parameters: `charset=US-ASCII; format=flowed; delsp=yes' parse_parameter: `charset' = `US-ASCII' parse_parameter: `format' = `flowed' parse_parameter: `delsp' = `yes' 4 a0011 OK FETCH completed. IMAP queue drained Can you do a spot check and see if there are missing FETCH respones? You should see lines like: * 1 FETCH ... * 2 FETCH ... * 3 FETCH ... We've seen some servers which seem to lie about how many messages there are in the EXISTS response, and Mutt is currently incapabable of handing the case where it doesn't get all FETCH responses. I'm not sure how other MUAs handle this. me
Stuck on 97% changing mailboxes
Hi, following on from problems getting a new Exchange 2010 server to play nice with mutt's SMTP support (http://marc.info/?l=mutt-usersm=128493280217503w=2), I have now encountered a problem with IMAP access to the same server. This problem is unique to mutt and is not displayed by other mail clients (which proves that I haven't missed something completely basic, but does not necessarily mean the problem is mutt and not the server!) I can connect fine, and read the INBOX fine, and I seem to be able to change into small mailboxes 2000 messages. By change I mean, I can select the mailbox in mutt and after a delay while it downloads the message headers, the mutt list view shows the contents of the folder. However, for larger mailboxes, when I attempt to change, mutt only gets as far as saying Fetching message headers... 3320/3391 (97%) which remains unchanged for ~5 minutes before I give up. Curiously, the .muttdebug log looks normal. The last few lines are Date: Mon, 20 Sep 2010 10:18:35 +0100 4 ) parse_parameters: `charset=US-ASCII; format=flowed; delsp=yes' parse_parameter: `charset' = `US-ASCII' parse_parameter: `format' = `flowed' parse_parameter: `delsp' = `yes' 4 a0011 OK FETCH completed. IMAP queue drained (where that Date is the date of the most recent email in that mailbox). The other data point is the fact that Thunderbird has no problem reading these mailboxes: it downloads the headers, displays the full message list, and reads the contents of the emails to generate its search index, all without any unusual delay. The problem seems to be 97% every time. And there does seem to be something significant about mailbox size -- I haven't found a mailbox with fewer than 2000 messages with this problem, and I haven't found a mailbox larger than that without it. But this is obviously based on limited statistics and it's difficult to test. I have disabled header and body caching in mutt. No change. Any ideas? Is there a switch I can throw in mutt? -- Mike
