Re: [Dovecot] Corrupted index cache files
On 8/13/2008, Mike Abbott ([EMAIL PROTECTED]) wrote: Post your dovecot -n output? Here's some of it. Not very enlightening. Why just some of it? Its not like its that much, and what you may *think* is irrelevant, may in fact be *very* relevant... -- Best regards, Charles
Re: [Dovecot] Corrupted index cache files
I re-ran imaptest on an empty mail store, single client, multiple users, using your dovecot-crlf input file, for a couple hours. Here's the distribution of errors that imaptest reports: 100 Error: user%d[%d]: ...: Header DELIVERED-TO changed 167 Error: user%d[%d]: ...: Header CC changed 417 Error: user%d[%d]: ...: Header TO changed 438 Error: user%d[%d]: ...: Header IN-REPLY-TO changed 445 Error: user%d[%d]: ...: Header REFERENCES changed 485 Error: user%d[%d]: ...: Header FROM changed 503 Error: user%d[%d]: ...: Header SUBJECT changed 544 Error: user%d[%d]: ...: Header MESSAGE-ID changed 764 Error: user%d[%d]: UID=%d INTERNALDATE changed 823 Error: user%d[%d]: UID %d changed 1040 Error: user%d[%d]: uid=%d ...: BODY changed 1052 Error: user%d[%d]: uid=%d ...: BODYSTRUCTURE changed 1546 Error: user%d[%d]: uid=%d ...: ENVELOPE changed 1833 Error: user%d[%d]: uid=%d ...: RFC822.SIZE size changed 3493 Error: user%d[%d]: uid=%d ...: BODY[] size changed 3691 Error: user%d[%d]: uid=%d ...: BODY[HEADER] size changed 3695 Error: user%d[%d]: uid=%d ...: BODY[TEXT] size changed 3880 Error: user%d[%d]: uid=%d ...: BODY[%d] size changed 376365 Error: UIVALIDITY changed: %d - %d During that time dovecot reported no corruption errors. What does it mean that imaptest is having such trouble?
Re: [Dovecot] Corrupted index cache files
On Aug 14, 2008, at 9:56 AM, Mike Abbott wrote: I re-ran imaptest on an empty mail store, single client, multiple users, using your dovecot-crlf input file, for a couple hours. Here's the distribution of errors that imaptest reports: 100 Error: user%d[%d]: ...: Header DELIVERED-TO changed .. During that time dovecot reported no corruption errors. What do you need to do to get corruption errors? Is multiple clients enough? What does it mean that imaptest is having such trouble? I think it just means that imaptest's support for multiple users is broken. I always test it only using a single test user. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
One or more users? Many different users. Post your dovecot -n output? Here's some of it. Not very enlightening. # 1.1.2: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ verbose_proctitle: yes first_valid_uid: 6 last_valid_uid: 5 first_valid_gid: 6 last_valid_gid: 5 mail_access_groups: mail mail_privileged_group: mail mail_location: maildir:~/mail Can you reproduce these easily with my imaptest tool? http://imapwiki.org/ImapTest Some of them. When running imaptest I see these dovecot errors: Corrupted index cache file %s: record continues outside its allocated size Corrupted index cache file %s: record points outside file Corrupted index cache file %s: used_file_size too large Plus I see LOTS of errors from imaptest itself: Error: UIVALIDITY changed: %d - %d Error: user%d[%d]: ...: Header CC changed Error: user%d[%d]: ...: Header DELIVERED-TO changed Error: user%d[%d]: ...: Header FROM changed Error: user%d[%d]: ...: Header IN-REPLY-TO changed Error: user%d[%d]: ...: Header MESSAGE-ID changed Error: user%d[%d]: ...: Header REFERENCES changed Error: user%d[%d]: ...: Header SUBJECT changed Error: user%d[%d]: ...: Header SUBJECT changed Error: user%d[%d]: ...: Header TO changed Error: user%d[%d]: UID %d changed Message-Id Error: user%d[%d]: UID=%d INTERNALDATE changed Error: user%d[%d]: uid=%d ...: BODY changed Error: user%d[%d]: uid=%d ...: BODYSTRUCTURE changed Error: user%d[%d]: uid=%d ...: BODY[%d] size changed Error: user%d[%d]: uid=%d ...: BODY[HEADER] size changed Error: user%d[%d]: uid=%d ...: BODY[TEXT] size changed Error: user%d[%d]: uid=%d ...: BODY[] size changed Error: user%d[%d]: uid=%d ...: ENVELOPE changed Error: user%d[%d]: uid=%d ...: RFC822.SIZE size changed One problem with HFS+ is that hard links are more or less buggy. But v1.1's default settings should include dotlock_use_excl=yes. You maybe should set maildir_copy_with_hardlinks=no, but that shouldn't cause this bug. Changing maildir_copy_with_hardlinks makes no difference. You say that the dovecot errors are harmless because dovecot fixes them, but still there must be some downside, if only a performance hit. Any other info I can provide to help you figure this out?
Re: [Dovecot] Corrupted index cache files
On Aug 13, 2008, at 6:09 PM, Mike Abbott wrote: mail_access_groups: mail mail_privileged_group: mail You probably won't need either of these. And there's no point in setting them to the same value. Can you reproduce these easily with my imaptest tool? http://imapwiki.org/ImapTest Some of them. When running imaptest I see these dovecot errors: Corrupted index cache file %s: record continues outside its allocated size Corrupted index cache file %s: record points outside file Corrupted index cache file %s: used_file_size too large How soon? With what kind of imaptest parameters? I can't reproduce this on my Macbook (OS X 10.5.4, HFS+). Plus I see LOTS of errors from imaptest itself: Error: UIVALIDITY changed: %d - %d Did you test with a single user or multiple users? imaptest's support for testing multiple users is currently a bit broken, but this shouldn't happen with a single user.. Error: user%d[%d]: ...: Header CC changed Error: user%d[%d]: ...: Header DELIVERED-TO changed Error: user%d[%d]: ...: Header FROM changed These shouldn't be happening as long as you use a test mbox file that has unique Message-ID headers. Were you using the recommended dovecot- crlf as the input? You say that the dovecot errors are harmless because dovecot fixes them, but still there must be some downside, if only a performance hit. Any other info I can provide to help you figure this out? Yes, there's a performance hit since all the cached data is lost. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
On Aug 13, 2008, at 6:09 PM, Mike Abbott wrote: Can you reproduce these easily with my imaptest tool? http://imapwiki.org/ImapTest Some of them. When running imaptest I see these dovecot errors: Corrupted index cache file %s: record continues outside its allocated size Corrupted index cache file %s: record points outside file Corrupted index cache file %s: used_file_size too large You could try if it makes a difference to set mmap_disable=yes or lock_method=flock. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
How soon? With what kind of imaptest parameters? I can't reproduce this on my Macbook (OS X 10.5.4, HFS+). I ran imaptest for just a few minutes and saw all these errors in that time. Default imaptest parameters except for user/host names etc. Error: UIVALIDITY changed: %d - %d Did you test with a single user or multiple users? imaptest's support for testing multiple users is currently a bit broken, but this shouldn't happen with a single user.. This error happens with clients=1, clients=10 and clients=100. Error: user%d[%d]: ...: Header CC changed Error: user%d[%d]: ...: Header DELIVERED-TO changed Error: user%d[%d]: ...: Header FROM changed These shouldn't be happening as long as you use a test mbox file that has unique Message-ID headers. Were you using the recommended dovecot-crlf as the input? I passed in your dovecot-crlf file to imaptest, but the message store was cloned from elsewhere first. So you're saying that dovecot only works with special hand-crafted input, and not with real-world mail? Hmmm :) :) Actually I do see plenty of duplicate Message-ID headers in the mail store (not from your input file). I'll investigate that further. Thanks for the tip.
Re: [Dovecot] Corrupted index cache files
On Aug 13, 2008, at 7:12 PM, Mike Abbott wrote: Error: UIVALIDITY changed: %d - %d Did you test with a single user or multiple users? imaptest's support for testing multiple users is currently a bit broken, but this shouldn't happen with a single user.. This error happens with clients=1, clients=10 and clients=100. I mean the user parameter, do you use user=something or user=something %d or similar to specify one/multiple different users where imaptest logs on to? Error: user%d[%d]: ...: Header CC changed Error: user%d[%d]: ...: Header DELIVERED-TO changed Error: user%d[%d]: ...: Header FROM changed These shouldn't be happening as long as you use a test mbox file that has unique Message-ID headers. Were you using the recommended dovecot-crlf as the input? I passed in your dovecot-crlf file to imaptest, but the message store was cloned from elsewhere first. So you're saying that dovecot only works with special hand-crafted input, and not with real-world mail? Hmmm :) :) Actually I do see plenty of duplicate Message-ID headers in the mail store (not from your input file). I'll investigate that further. Thanks for the tip. Dovecot doesn't care about any of that, but imaptest's state tracking requires that Message-ID header uniquely identifies a mail. So the imaptest run was done with a mailbox that already contained other data? Can you break if if you try with an empty mailbox where only imaptest starts appending new mails? PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
On Aug 8, 2008, at 9:59 PM, Mike Abbott wrote: I see these errors more often than I'd like from Dovecot-1.1.2 on Mac OS X 10.5.4 (names and numbers elided): One or more users? Corrupted index cache file %s: Corrupted virtual size for uid=%d: %d != %d Corrupted index cache file %s: Broken virtual size for mail UID %d Corrupted index cache file %s: used_file_size too large How bad are these? As long as Dovecot detects the corruption, the errors are transparent to users since Dovecot fixes them automatically. What should I look for to find out why they happen? Other details: File system is HFS+ Mail store is maildir Hardware is Intel (Mac Pro) Post your dovecot -n output? Can you reproduce these easily with my imaptest tool? http://imapwiki.org/ImapTest One problem with HFS+ is that hard links are more or less buggy. But v1.1's default settings should include dotlock_use_excl=yes. You maybe should set maildir_copy_with_hardlinks=no, but that shouldn't cause this bug. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
On 8/8/2008 9:59 PM, Mike Abbott wrote: I see these errors more often than I'd like from Dovecot-1.1.2 on Mac OS X 10.5.4 (names and numbers elided): Corrupted index cache file %s: Corrupted virtual size for uid=%d: %d != %d Corrupted index cache file %s: Broken virtual size for mail UID %d Corrupted index cache file %s: used_file_size too large How bad are these? What should I look for to find out why they happen? Other details: File system is HFS+ Mail store is maildir Hardware is Intel (Mac Pro) Any reason you left out your config? -- Best regards, Charles
[Dovecot] Corrupted index cache files
I see these errors more often than I'd like from Dovecot-1.1.2 on Mac OS X 10.5.4 (names and numbers elided): Corrupted index cache file %s: Corrupted virtual size for uid=%d: %d ! = %d Corrupted index cache file %s: Broken virtual size for mail UID %d Corrupted index cache file %s: used_file_size too large How bad are these? What should I look for to find out why they happen? Other details: File system is HFS+ Mail store is maildir Hardware is Intel (Mac Pro) Thanks.
