Re: [Dovecot] Corrupted index cache file and high CPU usage
On 12/23/2011 11:08 AM, hydra wrote: Selinux wasn't running. I can reproduce the problem with this setup: - restore the maildir with the broken index.cache - run `/usr/bin/doveadm -D expunge -A mailbox Trash savedbefore 1w` in an infinite loop - use horde webmail to access the mail with IMAP - randomly display messages I could reproduce the problem on: Linux-2.6.37-hardened-r7 Linux-2.6.39-hardened-r8 Linux-3.0.4-hardened-r5 Linux-3.0.6-gentoo Linux-3.1.4-gentoo However I failed to reproduce the problem on: Linux-2.6.36-hardened-r9 So it seems it was introduced in 2.6.37. I'm going to inspect it later on. By wishing you a God blessed merry Christmas I thank you for the support ;) Aha, cool. Glad you're making some progress, and that you now know it's kernel related in some way, just as Timo suspected. Keep in mind it may not necessarily be a code change problem in 2.6.37 onward. Could be a compiler/linker issue. It could also be the default EXT4 journal behavior changed (I haven't checked), or something similar. Merry Christmas to you as well hydra. I don't really deserve much credit for support here. I've just nudged you in the direction you already knew you should be heading. :) Best regards, -- Stan
Re: [Dovecot] Corrupted index cache file and high CPU usage
Yes, it is reproducible. About the file, I have tarred up the original maildir before I deleted the dovecot.index.cache, so when I try to bring in the original file, a 'cat' does no harm. Moreover, I've tried to do this will all files on the virtual machine, same results - no harm. On Thu, Dec 22, 2011 at 5:13 PM, Timo Sirainen t...@iki.fi wrote: On 22.12.2011, at 18.06, hydra wrote: After the virtual machine reboot, the CPU usage is normal again, but just until doveadm is launched again (it was run from cron). Sorry for the partition/drive terminology mess up. So it's reproducible? You could see if you can reproduce it by simply reading the file(s) that are causing problems. Like with zsh: cat **/* /dev/null
Re: [Dovecot] Corrupted index cache file and high CPU usage
On 12/23/2011 4:12 AM, hydra wrote: Yes, it is reproducible. About the file, I have tarred up the original maildir before I deleted the dovecot.index.cache, so when I try to bring in the original file, a 'cat' does no harm. Moreover, I've tried to do this will all files on the virtual machine, same results - no harm. Is selinux running? If so, turn it off and see if that helps. And turn off anything/everything else on this Linux VM that's not running on the other Dovecot VMs you mentioned are working properly. Let's do some process of elimination here. -- Stan
Re: [Dovecot] Corrupted index cache file and high CPU usage
Selinux wasn't running. I can reproduce the problem with this setup: - restore the maildir with the broken index.cache - run `/usr/bin/doveadm -D expunge -A mailbox Trash savedbefore 1w` in an infinite loop - use horde webmail to access the mail with IMAP - randomly display messages I could reproduce the problem on: Linux-2.6.37-hardened-r7 Linux-2.6.39-hardened-r8 Linux-3.0.4-hardened-r5 Linux-3.0.6-gentoo Linux-3.1.4-gentoo However I failed to reproduce the problem on: Linux-2.6.36-hardened-r9 So it seems it was introduced in 2.6.37. I'm going to inspect it later on. By wishing you a God blessed merry Christmas I thank you for the support ;) On Fri, Dec 23, 2011 at 11:36 AM, Stan Hoeppner s...@hardwarefreak.comwrote: On 12/23/2011 4:12 AM, hydra wrote: Yes, it is reproducible. About the file, I have tarred up the original maildir before I deleted the dovecot.index.cache, so when I try to bring in the original file, a 'cat' does no harm. Moreover, I've tried to do this will all files on the virtual machine, same results - no harm. Is selinux running? If so, turn it off and see if that helps. And turn off anything/everything else on this Linux VM that's not running on the other Dovecot VMs you mentioned are working properly. Let's do some process of elimination here. -- Stan
Re: [Dovecot] Corrupted index cache file and high CPU usage
Hello Timo, thank you for the reply. I was suspecting the same. However: - the machine runs under Vmware, - I've tried 3 different kernel versions, - I've tried 3 different SCSI controllers. All same results. On Wed, Dec 21, 2011 at 6:16 PM, Timo Sirainen t...@iki.fi wrote: On 21.12.2011, at 18.38, hydra wrote: I suspect, that dovecot v2.0.16 under Linux (tested on Gentoo 3.0.4 Hardened, Gentoo 3.0.6, Gentoo 3.1.4) causes 100% cpu utilization when index.cache is broken and doveadm is lanched. Dovecot uses LDAP as the userdb and provides IMAP and LDA. Because each user has a quota, the doveadm's expunge functionality is used to get rid of old trashed mail. The mail log: Dec 12 07:38:59 www dovecot: imap(user): Error: Corrupted index cache file /var/data/mail/domain.tld/user/mail/dovecot.index.cache: invalid record size OK.. Mail is stored on ext4 and flush will use all of the CPU capacity like this: load average: 4.00, 4.01, 4.05 ps: root 26401 98.2 0.0 0 0 ?RDec20 1369:49 [flush-253:1] That's a kernel process.. I suspect, that this is something to do with Dovecot, because after deleting the dovecot.index.cache file, everything went back to normal. When this happens, I cannot unmount the drive nor a system reboot works. That's a kernel bug.. I think you're thinking it the wrong way: Dovecot isn't causing your system to break. Your system is causing Dovecot to break. Faulty hardware or faulty kernel.
Re: [Dovecot] Corrupted index cache file and high CPU usage
On 12/22/2011 8:08 AM, hydra wrote: Hello Timo, thank you for the reply. I was suspecting the same. However: - the machine runs under Vmware, - I've tried 3 different kernel versions, - I've tried 3 different SCSI controllers. All same results. dmesg output? Log errors? Is your EXT4 filesystem on a VMFS volume or an RDM (SAN LUN)? On Wed, Dec 21, 2011 at 6:16 PM, Timo Sirainen t...@iki.fi wrote: On 21.12.2011, at 18.38, hydra wrote: That's a kernel process.. I suspect, that this is something to do with Dovecot, because after deleting the dovecot.index.cache file, everything went back to normal. When this happens, I cannot unmount the drive nor a system reboot works. System (host machine) reboot, or virtual machine reboot doesn't fix the problem? FYI, Linux doesn't unmount drives, it unmounts filesystems. I'd say you may have a problem with your VMFS volume or RDM, or maybe just your EXT4 filesystem. Have you run an fsck on it? What result? Or, as Timo suggests, could be a kernel bug. Or an interaction of these low level layers causing a problem. If you can't unmount a filesystem, that has nothing to do with Dovecot, and points to a much larger, more critical, problem. Do you have this problem when booting an older kernel? Say 2.6.32? 2.6.37? That's a kernel bug.. I think you're thinking it the wrong way: Dovecot isn't causing your system to break. Your system is causing Dovecot to break. Faulty hardware or faulty kernel. -- Stan
Re: [Dovecot] Corrupted index cache file and high CPU usage
On Thu, Dec 22, 2011 at 3:49 PM, Stan Hoeppner s...@hardwarefreak.comwrote: On 12/22/2011 8:08 AM, hydra wrote: Hello Timo, thank you for the reply. I was suspecting the same. However: - the machine runs under Vmware, - I've tried 3 different kernel versions, - I've tried 3 different SCSI controllers. All same results. dmesg output? Log errors? Nothing there Is your EXT4 filesystem on a VMFS volume or an RDM (SAN LUN)? VMFS On Wed, Dec 21, 2011 at 6:16 PM, Timo Sirainen t...@iki.fi wrote: On 21.12.2011, at 18.38, hydra wrote: That's a kernel process.. I suspect, that this is something to do with Dovecot, because after deleting the dovecot.index.cache file, everything went back to normal. When this happens, I cannot unmount the drive nor a system reboot works. System (host machine) reboot, or virtual machine reboot doesn't fix the problem? FYI, Linux doesn't unmount drives, it unmounts filesystems. After the virtual machine reboot, the CPU usage is normal again, but just until doveadm is launched again (it was run from cron). Sorry for the partition/drive terminology mess up. I'd say you may have a problem with your VMFS volume or RDM, or maybe just your EXT4 filesystem. Have you run an fsck on it? What result? A normal system reboot wasn't possible, because the ext4 fs wasn't unmounted (and it wasn't possible to unmount the fs nor run sync - both locked up) and thus I had to reboot from the vSphere Client. After the reboot, fsck placed the fs to a consistent state, however the problem occurred the next morning, when doveadm from the cron was run again. So a fsck didn't help. Or, as Timo suggests, could be a kernel bug. Or an interaction of these low level layers causing a problem. If you can't unmount a filesystem, that has nothing to do with Dovecot, and points to a much larger, more critical, problem. Do you have this problem when booting an older kernel? Say 2.6.32? 2.6.37? The oldest available kernel is 2.6.32 so I'm going to test it. Thank you :) That's a kernel bug.. I think you're thinking it the wrong way: Dovecot isn't causing your system to break. Your system is causing Dovecot to break. Faulty hardware or faulty kernel. -- Stan
Re: [Dovecot] Corrupted index cache file and high CPU usage
On 22.12.2011, at 18.06, hydra wrote: After the virtual machine reboot, the CPU usage is normal again, but just until doveadm is launched again (it was run from cron). Sorry for the partition/drive terminology mess up. So it's reproducible? You could see if you can reproduce it by simply reading the file(s) that are causing problems. Like with zsh: cat **/* /dev/null
[Dovecot] Corrupted index cache file and high CPU usage
I suspect, that dovecot v2.0.16 under Linux (tested on Gentoo 3.0.4
Hardened, Gentoo 3.0.6, Gentoo 3.1.4) causes 100% cpu utilization when
index.cache is broken and doveadm is lanched. Dovecot uses LDAP as the
userdb and provides IMAP and LDA. Because each user has a quota, the
doveadm's expunge functionality is used to get rid of old trashed mail.
The mail log:
Dec 12 07:38:59 www dovecot: imap(user): Error: Corrupted index cache file
/var/data/mail/domain.tld/user/mail/dovecot.index.cache: invalid record size
Doveadm is launched as:
doveadm -D expunge -A mailbox Trash savedbefore 1w
Mail is stored on ext4 and flush will use all of the CPU capacity like this:
load average: 4.00, 4.01, 4.05
ps:
root 26401 98.2 0.0 0 0 ?RDec20 1369:49
[flush-253:1]
lsblk:
vg_data-mail (dm-1) 253:10 500M 0 /var/data/mail
dovecot -n:
# 2.0.16: /etc/dovecot/dovecot.conf
# OS: Linux 3.0.4-hardened-r5 x86_64 Gentoo Base System release 2.0.3
auth_debug = yes
auth_debug_passwords = yes
auth_verbose = yes
hostname = mail.domain.tld
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
listen = *
mail_debug = yes
mail_gid = vmail
mail_home = /var/data/mail/domain.tld/%n
mail_location = maildir:~/mail
mail_plugins = mail_log notify zlib quota
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
autocreate = Sent
autocreate2 = Trash
autocreate3 = Spam
autosubscribe = Sent
autosubscribe2 = Trash
autosubscribe3 = Spam
fts = squat
fts_squat = partial=4 full=10
quota = maildir:User quota
quota_rule = *:storage=100M
quota_rule2 = Trash:ignore
quota_warning = storage=95%% quota-warning 95 %u
quota_warning2 = storage=80%% quota-warning 80 %u
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_quota_max_scripts = 5
sieve_quota_max_storage = 5M
}
postmaster_address = postmas...@domain.tld
protocols = imap sieve
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = vmail
mode = 0600
user = vmail
}
}
service quota-warning {
executable = script /etc/dovecot/quota-warning.sh
unix_listener quota-warning {
user = vmail
}
user = vmail
}
ssl_cert = /etc/postfix/mail.domain.tld-cert.pem
ssl_key = /etc/postfix/mail.domain.tld-key.pem
userdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
verbose_proctitle = yes
verbose_ssl = yes
protocol lda {
info_log_path =
log_path =
mail_plugins = mail_log notify zlib quota sieve
}
protocol imap {
mail_plugins = mail_log notify zlib quota autocreate fts fts_squat
imap_zlib imap_quota
}
dovecot-ldap.conf.ext:
hosts = localhost
dn = user
dnpass = pass
base = base
auth_bind = yes
user_attrs =
homeDirectory=home,uidNumber=uid,gidNumber=gid,mailboxQuota=quota_rule=*:bytes=%$
user_filter = ((objectClass=inetOrgPerson)(uid=%u))
pass_attrs = uid=user
pass_filter = ((objectClass=inetOrgPerson)(uid=%u))
iterate_attrs = uid=user
iterate_filter = (objectClass=inetOrgPerson)
I suspect, that this is something to do with Dovecot, because after
deleting the dovecot.index.cache file, everything went back to normal. When
this happens, I cannot unmount the drive nor a system reboot works.
Thank you !
Re: [Dovecot] Corrupted index cache file and high CPU usage
On 21.12.2011, at 18.38, hydra wrote: I suspect, that dovecot v2.0.16 under Linux (tested on Gentoo 3.0.4 Hardened, Gentoo 3.0.6, Gentoo 3.1.4) causes 100% cpu utilization when index.cache is broken and doveadm is lanched. Dovecot uses LDAP as the userdb and provides IMAP and LDA. Because each user has a quota, the doveadm's expunge functionality is used to get rid of old trashed mail. The mail log: Dec 12 07:38:59 www dovecot: imap(user): Error: Corrupted index cache file /var/data/mail/domain.tld/user/mail/dovecot.index.cache: invalid record size OK.. Mail is stored on ext4 and flush will use all of the CPU capacity like this: load average: 4.00, 4.01, 4.05 ps: root 26401 98.2 0.0 0 0 ?RDec20 1369:49 [flush-253:1] That's a kernel process.. I suspect, that this is something to do with Dovecot, because after deleting the dovecot.index.cache file, everything went back to normal. When this happens, I cannot unmount the drive nor a system reboot works. That's a kernel bug.. I think you're thinking it the wrong way: Dovecot isn't causing your system to break. Your system is causing Dovecot to break. Faulty hardware or faulty kernel.
Re: [Dovecot] Corrupted index cache file
On Fri, 2011-04-15 at 08:32 +0200, Frank Bonnet wrote: Apr 14 19:24:37 imap(xxx): Error: FETCH [] for mailbox Drafts (Hugues at ESIEE) UID 17836 got too little data: 16384 vs 18012 Apr 14 19:24:37 imap(xxx): Error: Corrupted index cache file /var/cache/dovecot/index/xxx/.imap/Drafts (Hugues at ESIEE)/dovecot.index.cache: Broken virtual size for mail UID 17836 mboxes are a bit annoying things. I guess there's a bug somewhere in Dovecot code, but unless you can show me how to reproduce it I don't think I can do anything about it. (I've earlier spent days looking at the mbox code without seeing any bugs in it.) Maybe once everything else is working perfectly I'll just try once more to rewrite the mbox code to be simpler to understand and become perfect. :)
Re: [Dovecot] Corrupted index cache file
For what it's worth.. I've seen the FETCH error with got too little data also with Maildir setup. So, not only limited to mbox. This was quite some time ago with (I think) 1.0.15. Back then, I just removed the dovecot.index and cache files (leaving only the uidlist file) for that maildir folder. Worked fine after that (except I had to convince Outlook to resync.. Mail bodies in Outlook cache sometimes didn't match with mail headers...) Currently running 1.2.15, no further corruption seen. -- Maarten On Mon, 9 May 2011, Timo Sirainen wrote: On Fri, 2011-04-15 at 08:32 +0200, Frank Bonnet wrote: Apr 14 19:24:37 imap(xxx): Error: FETCH [] for mailbox Drafts (Hugues at ESIEE) UID 17836 got too little data: 16384 vs 18012 Apr 14 19:24:37 imap(xxx): Error: Corrupted index cache file /var/cache/dovecot/index/xxx/.imap/Drafts (Hugues at ESIEE)/dovecot.index.cache: Broken virtual size for mail UID 17836 mboxes are a bit annoying things. I guess there's a bug somewhere in Dovecot code, but unless you can show me how to reproduce it I don't think I can do anything about it. (I've earlier spent days looking at the mbox code without seeing any bugs in it.) Maybe once everything else is working perfectly I'll just try once more to rewrite the mbox code to be simpler to understand and become perfect. :)
[Dovecot] Corrupted index cache file
Hello I got this error message yesterday evening for a user I've checked LDA error log it happened 6 times in 3 days for 5 differents users, not much but I wonder what's going on as actually the server is moderately loaded ( Easter holidays ) we have currently 250/300 IMAP processes , in normal production time we have approx 1000 IMAP processes running. Apr 14 19:24:37 imap(xxx): Error: FETCH [] for mailbox Drafts (Hugues at ESIEE) UID 17836 got too little data: 16384 vs 18012 Apr 14 19:24:37 imap(xxx): Error: Corrupted index cache file /var/cache/dovecot/index/xxx/.imap/Drafts (Hugues at ESIEE)/dovecot.index.cache: Broken virtual size for mail UID 17836 The machine is supposed to be HUDGE for our 4000 mailboxes HP Proliant DL380 G7 , 12 Gb RAM 24 cores !!! I've located index/cache files on another disk than users mailboxes mailboxes are living on a 2Tb RAID array ( 50% full ) index/cache are living on another single disk ( 10% full ) I run Dovecot 2.0.11 + dovecot2-pigeonhole compiled from FreeBSD 8.2 ports Dovecot or hardware problem ??? Thanks a lot for any ideas/infos
Re: [Dovecot] Corrupted index cache file
On 2011-04-15 2:32 AM, Frank Bonnet wrote: Dovecot or hardware problem ??? Thanks a lot for any ideas/infos Sorry, crystal ball is broken, rather than making us guess, maybe you could provide your config details (dovecot -n output for starters)? -- Best regards, Charles
Re: [Dovecot] Corrupted index cache file
On 04/15/2011 01:38 PM, Charles Marcus wrote:
On 2011-04-15 2:32 AM, Frank Bonnet wrote:
Dovecot or hardware problem ???
Thanks a lot for any ideas/infos
Sorry, crystal ball is broken, rather than making us guess, maybe you
could provide your config details (dovecot -n output for starters)?
yes of course here it is
# 2.0.11: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 8.2-RELEASE amd64
auth_worker_max_count = 100
debug_log_path = /var/log/dovecot/debug.log
default_client_limit = 6144
default_process_limit = 1024
disable_plaintext_auth = no
info_log_path = /var/log/dovecot/info.log
log_path = /var/log/dovecot/dovecot.log
mail_location = mbox:%h/:INDEX=/var/cache/dovecot/index/%u:INBOX=/var/mail/%u
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date
mbox_write_locks = fcntl
passdb {
driver = passwd
}
plugin/sieve = ~/.dovecot.sieve
plugin/sieve_dir = ~/sieve
postmaster_address = postmas...@esiee.fr
protocols = imap pop3 sieve sieve
sendmail_path = /usr/local/sbin/sendmail
service auth {
client_limit = 7169
}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_min_avail = 10
service_count = 1
vsz_limit = 1 G
}
service imap {
process_limit = 2048
vsz_limit = 1 G
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
inet_listener sieve_deprecated {
port = 2000
}
}
service pop3-login {
inet_listener pop3 {
port = 110
}
inet_listener pop3s {
port = 995
ssl = yes
}
}
ssl_ca = /etc/ssl/cacert.pem
ssl_cert = /etc/ssl/cert.pem
ssl_key = /etc/ssl/server.key
userdb {
driver = passwd
}
verbose_proctitle = yes
protocol lda {
info_log_path = /var/log/dovecot/dovecot-lda.log
log_path = /var/log/dovecot/dovecot-lda-errors.log
mail_plugins = sieve
}
protocol imap {
imap_client_workarounds = tb-extra-mailbox-sep
}
protocol sieve {
managesieve_implementation_string = Dovecot Pigeonhole
}
Re: [Dovecot] Corrupted index cache file and Maildir filename has wrong W value
On Tuesday 25 May 2010 16:26:15 Timo Sirainen wrote: On Tue, 2010-05-04 at 18:21 +0100, Barry Scott wrote: May 4 15:25:03 f12barry dovecot: IMAP(bscott): Maildir filename has wrong W value: /home/bscott/Maildir/._NewMail-com/cur/1271683097.M615087P3538.f12barry.office.onelan.co.uk,S=3154,W=3278:2,S S=3154 means the file's size should be 3154 bytes. W=3278 means the file's virtual size = size if linefeeds were CRLF should be 3278 bytes. Easy way to check these values is with e.g.: # wc 1274725794.M935807P27627.kurkku,S=1901,W=1951:2, 50 212 1901 1274725794.M935807P27627.kurkku,S=1901,W=1951:2, # echo 1901+50|bc 1951 If W value is wrong, it's not currently fixed automatically. Because it just shouldn't happen. How are you delivering new mails? That file looks like it's created by Dovecot. So the only possibilities are: a) Something modified the file after it was already written. b) The program writing the maildir file is buggy (I'm not aware of any Dovecot bugs related to this.) c) Filesystem corrupted the mail file. Thanks for the info. I'll write some tools to check the W values and see if I can track down the corruption source. Barry
Re: [Dovecot] Corrupted index cache file and Maildir filename has wrong W value
On Tuesday 25 May 2010 16:26:15 Timo Sirainen wrote: On Tue, 2010-05-04 at 18:21 +0100, Barry Scott wrote: May 4 15:25:03 f12barry dovecot: IMAP(bscott): Maildir filename has wrong W value: /home/bscott/Maildir/._NewMail-com/cur/1271683097.M615087P3538.f12barry.o ffice.onelan.co.uk,S=3154,W=3278:2,S S=3154 means the file's size should be 3154 bytes. W=3278 means the file's virtual size = size if linefeeds were CRLF should be 3278 bytes. Easy way to check these values is with e.g.: # wc 1274725794.M935807P27627.kurkku,S=1901,W=1951:2, 50 212 1901 1274725794.M935807P27627.kurkku,S=1901,W=1951:2, # echo 1901+50|bc 1951 If W value is wrong, it's not currently fixed automatically. Because it just shouldn't happen. How are you delivering new mails? That file looks like it's created by Dovecot. So the only possibilities are: a) Something modified the file after it was already written. This was the cause. I had miss configured spambayes and it was updating the files. I had figure out to fix up the S= values but had not fixed the W= values. Again, Thanks for the info. Barry
[Dovecot] Corrupted index cache file and Maildir filename has wrong W value
kmail stopped working and when I check the dovecot log file I see: May 4 11:40:38 f12barry dovecot: IMAP(bscott): Corrupted index cache file /home/bscott/Maildir/._NewMail-com/dovecot.index.cache: Corrupted virtual size for uid=234: 633278 != 633251 May 4 11:40:38 f12barry dovecot: IMAP(bscott): Corrupted index cache file /home/bscott/Maildir/._NewMail-com/dovecot.index.cache: Corrupted virtual size for uid=234: 633278 != 633251 May 4 12:34:44 f12barry dovecot: IMAP(bscott): Corrupted index cache file /home/bscott/Maildir/.Mailing Lists.Xorg Devel 2010-04/dovecot.index.cache: Corrupted virtual size for uid=312: 8884 != 8858 May 4 12:34:44 f12barry dovecot: IMAP(bscott): Corrupted index cache file /home/bscott/Maildir/.Mailing Lists.Xorg Devel 2010-04/dovecot.index.cache: Corrupted virtual size for uid=312: 8884 != 8858 ... more of the same... May 4 13:02:54 f12barry dovecot: IMAP(bscott): Corrupted index cache file /home/bscott/Maildir/.Mailing Lists.Xorg Users 2010-04/dovecot.index.cache: Corrupted virtual size for uid=38: 30392 != 30366 May 4 13:02:58 f12barry dovecot: IMAP(bscott): Corrupted index cache file /home/bscott/Maildir/.Mailing Lists.Xorg Users 2010-04/dovecot.index.cache: Corrupted virtual size for uid=43: 8796 != 8770 May 4 15:25:03 f12barry dovecot: IMAP(bscott): FETCH [] for mailbox _NewMail-com UID 48 got too little data: 3230 vs 3278 May 4 15:25:03 f12barry dovecot: IMAP(bscott): Maildir filename has wrong W value: /home/bscott/Maildir/._NewMail-com/cur/1271683097.M615087P3538.f12barry.office.onelan.co.uk,S=3154,W=3278:2,S May 4 15:25:03 f12barry dovecot: IMAP(bscott): Corrupted index cache file /home/bscott/Maildir/._NewMail-com/dovecot.index.cache: Broken virtual size for mail UID 48 May 4 15:25:03 f12barry dovecot: IMAP(bscott): Disconnected: FETCH failed bytes=15437134/33737060 May 4 17:45:41 f12barry dovecot: dovecot: Killed with signal 15 (by pid=7300 uid=0 code=kill) At this point I had only tail'ed the log and seen the issue with UID 48. I then: 1. stopped dovecot 2. deleted the ofending file 3. deleted the dovecot* files 4. started dovecot May 4 17:49:58 f12barry dovecot: Dovecot v1.2.11 starting up (core dumps disabled) May 4 17:52:17 f12barry dovecot: Killed with signal 15 (by pid=7556 uid=0 code=kill) May 4 17:53:21 f12barry dovecot: Dovecot v1.2.11 starting up (core dumps disabled) May 4 17:53:21 f12barry sendmail[2067]: starting daemon (8.14.3): smtp+queue...@01:00:00 May 4 17:53:21 f12barry sm-msp-queue[2075]: starting daemon (8.14.3): queue...@01:00:00 May 4 17:55:26 f12barry dovecot: imap-login: Login: user=bscott, method=PLAIN, rip=::1, lip=::1, secured May 4 17:55:29 f12barry dovecot: IMAP(bscott): FETCH [] for mailbox _NewMail-com UID 48 got too little data: 3878 vs 3920 May 4 17:55:29 f12barry dovecot: IMAP(bscott): Maildir filename has wrong W value: /home/bscott/Maildir/._NewMail-com/cur/1271683097.M733325P3538.f12barry.office.onelan.co.uk,S=3780,W=3920:2,S May 4 17:55:29 f12barry dovecot: IMAP(bscott): Corrupted index cache file /home/bscott/Maildir/._NewMail-com/dovecot.index.cache: Broken virtual size for mail UID 48 May 4 17:55:29 f12barry dovecot: IMAP(bscott): Disconnected: FETCH failed bytes=3955/48628 May 4 18:05:55 f12barry dovecot: imap-login: Login: user=bscott, method=PLAIN, rip=::1, lip=::1, secured Now another file is being complained about. What would cause these problems? What is the W value and how could I check it? Here is the details of the dovecot I'm using and the kmail version I'm using.ƀ $ sudo dovecot -n # 1.2.11: /etc/dovecot.conf # OS: Linux 2.6.32.11-99.fc12.x86_64 x86_64 Fedora release 12 (Constantine) login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 auth default: passdb: driver: pam userdb: driver: passwd Client is kmail from kdepim-4.4.2-1.fc12.x86_64 Barry
[Dovecot] corrupted index cache file, physcial size, in migration to snow leopard
Hi, I am in the middle of migrating a server from Apple's leopard to snow leopard. Having big problems with migrating mail to dovecot. The version is: # dovecotd --version 1.1.17apple0.5 and I use the migration script provide by Apple, /usr/libexec/dovecot/migrate_mail_data.pl The current problems concern migrating imap accounts. The old data typically shows up to mail clients as only partially migrated, with some issues there, like folders change from being listed on the server to being listed locally, but the biggest problem is that many many messages and folders have disappeared from view. But on the server, they seem to be there, in their correctly migrated place. The log files are full of messages like the ones below, which I'm guessing has something to do with the problem. The number of files with this problem is huge. Are there any scripts to rebuild the index cache files with correct physical sizes? Thanks in advance for any help! Best, Norman Apr 21 08:05:47 newhot dovecot[31163]: IMAP(*): Corrupted index cache file /Volumes/xserveHD2/var/spool/imap/dovecot/mail/A1F36563-F978-4F99- A1FF-EC5D223F18D6/dovecot.index.cache: Corrupted physical size for uid=8142: 4325 != 4214 Apr 21 08:05:50 newhot dovecot[31163]: IMAP(*): Corrupted index cache file /Volumes/xserveHD2/var/spool/imap/dovecot/mail/A1F36563-F978-4F99- A1FF-EC5D223F18D6/dovecot.index.cache: Corrupted physical size for uid=8145: 4625 != 4510 Apr 21 08:05:53 newhot dovecot[31163]: IMAP(*): Corrupted index cache file /Volumes/xserveHD2/var/spool/imap/dovecot/mail/A1F36563-F978-4F99- A1FF-EC5D223F18D6/dovecot.index.cache: Corrupted physical size for uid=8148: 4037 != 3956 Apr 21 08:06:04 newhot dovecot[31163]: IMAP(*): Corrupted index cache file /Volumes/xserveHD2/var/spool/imap/dovecot/mail/A1F36563-F978-4F99- A1FF-EC5D223F18D6/dovecot.index.cache: Corrupted physical size for uid=8158: 4673 != 4571
Re: [Dovecot] Corrupted index cache file still occurs with Dovecot 1.2.9
Timo Sirainen wrote, on 19/01/10 13:09: On Mon, 2010-01-11 at 11:07 +, Stuart Rowan wrote: Timo Sirainen wrote, on 28/07/09 19:24: On Tue, 2009-07-28 at 19:17 +0100, Stuart Rowan wrote: dovecot: 2009-07-28 10:23:16 Error: IMAP(strr): Corrupted index cache file /home/local/strr/Maildir/dovecot.index.cache: record continues outside its allocated size dovecot: 2009-07-28 10:23:56 Error: IMAP(strr): Corrupted index cache file /home/local/strr/Maildir/dovecot.index.cache: record continues outside its allocated size .. The index corruption issues continues to dog us though. Attached text file shows this corruption (I've masked the differing user names and full paths) as listed by the log file produced since 06:00 this morning. What filesystem are you using? NFS or any other remote filesystem? XFS as a local filesystem. Cheers, Stu. tigris:~# xfs_info /home meta-data=/dev/mapper/data-home--xfs isize=256agcount=33, agsize=6553536 blks = sectsz=512 attr=0 data = bsize=1024 blocks=209715200, imaxpct=25 = sunit=64 swidth=128 blks naming =version 2 bsize=4096 log =internal bsize=1024 blocks=51200, version=2 = sectsz=512 sunit=64 blks, lazy-count=0 realtime =none extsz=65536 blocks=0, rtextents=0 tigris:~# mount | grep /home /dev/mapper/data-home--xfs on /home type xfs (rw,noatime,logbufs=8) tigris:~# uname -a Linux tigris 2.6.26-2-amd64 #1 SMP Thu Nov 5 02:23:12 UTC 2009 x86_64 GNU/Linux tigris:~# cat /etc/debian_version 5.0.3
Re: [Dovecot] Corrupted index cache file still occurs with Dovecot 1.2.9
On Wed, 2010-01-20 at 10:33 +, Stuart Rowan wrote:
dovecot: 2009-07-28 10:23:16 Error: IMAP(strr): Corrupted index cache
file /home/local/strr/Maildir/dovecot.index.cache: record continues
outside its allocated size
dovecot: 2009-07-28 10:23:56 Error: IMAP(strr): Corrupted index cache
file /home/local/strr/Maildir/dovecot.index.cache: record continues
outside its allocated size
..
What filesystem are you using? NFS or any other remote filesystem?
XFS as a local filesystem.
Weird. See what it logs with the attached patch?
diff -r a1177c6cf8c7 src/lib-index/mail-cache-lookup.c
--- a/src/lib-index/mail-cache-lookup.c Wed Jan 20 11:44:06 2010 +0200
+++ b/src/lib-index/mail-cache-lookup.c Wed Jan 20 15:35:00 2010 +0200
@@ -263,7 +263,8 @@
if (ctx-rec-size - ctx-pos data_size) {
mail_cache_set_corrupted(cache,
- record continues outside its allocated size);
+ record continues outside its allocated size %u-%u%u, field=%s,
+ ctx-rec-size, ctx-pos, data_size, cache-fields[field_idx].field.name);
return -1;
}
signature.asc
Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache file still occurs with Dovecot 1.2.9
On Mon, 2010-01-11 at 11:07 +, Stuart Rowan wrote: Timo Sirainen wrote, on 28/07/09 19:24: On Tue, 2009-07-28 at 19:17 +0100, Stuart Rowan wrote: dovecot: 2009-07-28 10:23:16 Error: IMAP(strr): Corrupted index cache file /home/local/strr/Maildir/dovecot.index.cache: record continues outside its allocated size dovecot: 2009-07-28 10:23:56 Error: IMAP(strr): Corrupted index cache file /home/local/strr/Maildir/dovecot.index.cache: record continues outside its allocated size .. The index corruption issues continues to dog us though. Attached text file shows this corruption (I've masked the differing user names and full paths) as listed by the log file produced since 06:00 this morning. What filesystem are you using? NFS or any other remote filesystem? signature.asc Description: This is a digitally signed message part
[Dovecot] Corrupted index cache file still occurs with Dovecot 1.2.9
Timo Sirainen wrote, on 28/07/09 19:24: On Tue, 2009-07-28 at 19:17 +0100, Stuart Rowan wrote: dovecot: 2009-07-28 10:23:16 Error: IMAP(strr): Corrupted index cache file /home/local/strr/Maildir/dovecot.index.cache: record continues outside its allocated size dovecot: 2009-07-28 10:23:56 Error: IMAP(strr): Corrupted index cache file /home/local/strr/Maildir/dovecot.index.cache: record continues outside its allocated size These are kind of annoying. Apparently they still happens in some conditions, but I've really no idea how to reproduce/fix them. Anyway I'm going to simplify the code in a few months, maybe after that there will be no more cache corruption. :) Hi Timo, Dovecot continues to serve us extremely well: I turned on squat based FTS last week and I am amazed how much snappier Thunderbird's Search is at finding messages now :) The index corruption issues continues to dog us though. Attached text file shows this corruption (I've masked the differing user names and full paths) as listed by the log file produced since 06:00 this morning. Cheers, Stu. tigris:~# dovecot --version ; dovecot --build-options ; perl -pe '$u = 'user'; s/(?=IMAP\()(\w+)\)/$u)/;s@(?= /home/local/)(.*?)(?=/dove)@$u/ ... @' /var/log/dovecot/error.log 1.2.9 Build options: ioloop=epoll notify=inotify ipv6 openssl Mail storages: cydir dbox maildir mbox raw shared SQL drivers: mysql postgresql sqlite Passdb: checkpassword ldap pam passwd passwd-file shadow sql Userdb: nss ldap passwd passwd-file prefetch sql static 2010-01-11 08:23:26 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 08:31:51 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: record continues outside its allocated size 2010-01-11 09:13:49 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 09:15:08 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 09:17:01 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 09:17:58 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 09:49:05 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: record continues outside its allocated size 2010-01-11 09:50:21 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 09:50:22 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: record continues outside its allocated size 2010-01-11 09:51:01 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 09:51:10 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 09:57:20 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 09:58:00 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 10:02:14 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 10:04:20 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: record continues outside its allocated size 2010-01-11 10:05:38 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 10:06:23 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 10:06:33 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 10:06:34 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 10:08:46 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 10:10:16 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 10:15:07 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size 2010-01-11 10:18:34 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid field header size 2010-01-11 10:18:42 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: field header names corrupted 2010-01-11 10:21:40 IMAP(user): Error: Corrupted index cache file /home/local/user/ ... /dovecot.index.cache: invalid record size
Re: [Dovecot] Corrupted index cache file issues (Corrupted physical size)
Dovecot never modifies the S= field, even when it notices that it's wrong. So it's your LDA that writes them wrong. Is it ok to rename all files deleting S= and W= fields if no keywords was used?
Re: [Dovecot] Corrupted index cache file issues (Corrupted physical size)
On Jun 8, 2009, at 7:06 AM, Brent Bloxam wrote: Hello, I'm running Dovecot 1.1.13 on FreeBSD 7.0-RELEASE and I'm receiving a large number of errors much like the following: dovecot: Jun 08 09:27:04 Error: IMAP(x...@x.x): Corrupted index cache file /nfs/x.x/x/Maildir/dovecot.index.cache: Corrupted physical size for uid=2284: 5312 != 5217 How are you saving messages to the maildir? Do these happen for new mails also or only old ones?
Re: [Dovecot] Corrupted index cache file issues (Corrupted physical size)
Timo Sirainen wrote: On Jun 8, 2009, at 7:06 AM, Brent Bloxam wrote: Hello, I'm running Dovecot 1.1.13 on FreeBSD 7.0-RELEASE and I'm receiving a large number of errors much like the following: dovecot: Jun 08 09:27:04 Error: IMAP(x...@x.x): Corrupted index cache file /nfs/x.x/x/Maildir/dovecot.index.cache: Corrupted physical size for uid=2284: 5312 != 5217 How are you saving messages to the maildir? Do these happen for new mails also or only old ones? At the moment messages are being saved to the maildir through maildrop (called from qmail), as we're migrating from an old system. I'm not sure how I could go about pinpointing whether this affects new or old emails as I'm not aware of what actually generates this error. Any suggestions?
Re: [Dovecot] Corrupted index cache file issues (Corrupted physical size)
On Jun 9, 2009, at 11:49 AM, Brent Bloxam wrote: Timo Sirainen wrote: On Jun 8, 2009, at 7:06 AM, Brent Bloxam wrote: Hello, I'm running Dovecot 1.1.13 on FreeBSD 7.0-RELEASE and I'm receiving a large number of errors much like the following: dovecot: Jun 08 09:27:04 Error: IMAP(x...@x.x): Corrupted index cache file /nfs/x.x/x/Maildir/dovecot.index.cache: Corrupted physical size for uid=2284: 5312 != 5217 How are you saving messages to the maildir? Do these happen for new mails also or only old ones? At the moment messages are being saved to the maildir through maildrop (called from qmail), as we're migrating from an old system. I'm not sure how I could go about pinpointing whether this affects new or old emails as I'm not aware of what actually generates this error. Any suggestions? You'll see the uid=2284 above. Find the filename for that UID from dovecot-uidlist file and see how new the file is. Also does that filename contain ,S=5312? And what is the real size of the file?
Re: [Dovecot] Corrupted index cache file issues (Corrupted physical size)
Timo Sirainen wrote: On Jun 9, 2009, at 11:49 AM, Brent Bloxam wrote: Timo Sirainen wrote: On Jun 8, 2009, at 7:06 AM, Brent Bloxam wrote: Hello, I'm running Dovecot 1.1.13 on FreeBSD 7.0-RELEASE and I'm receiving a large number of errors much like the following: dovecot: Jun 08 09:27:04 Error: IMAP(x...@x.x): Corrupted index cache file /nfs/x.x/x/Maildir/dovecot.index.cache: Corrupted physical size for uid=2284: 5312 != 5217 How are you saving messages to the maildir? Do these happen for new mails also or only old ones? At the moment messages are being saved to the maildir through maildrop (called from qmail), as we're migrating from an old system. I'm not sure how I could go about pinpointing whether this affects new or old emails as I'm not aware of what actually generates this error. Any suggestions? You'll see the uid=2284 above. Find the filename for that UID from dovecot-uidlist file and see how new the file is. Also does that filename contain ,S=5312? And what is the real size of the file? Thanks. The errors I've checked are occurring for recent files, looks like the S= field doesn't match the real size, for example: 33568 Jun 8 23:46 1235105170.42208.abc.xyz.com,S=33498:2,RS Looking at older e-mails though I am seeing S != real size for some e-mail. Would dovecot ever be modifying the S field, or is this actually pointing to an issue with the LDA?
Re: [Dovecot] Corrupted index cache file issues (Corrupted physical size)
On Jun 9, 2009, at 12:26 PM, Brent Bloxam wrote: Thanks. The errors I've checked are occurring for recent files, looks like the S= field doesn't match the real size, for example: 33568 Jun 8 23:46 1235105170.42208.abc.xyz.com,S=33498:2,RS That's the problem then. Looking at older e-mails though I am seeing S != real size for some e-mail. Would dovecot ever be modifying the S field, or is this actually pointing to an issue with the LDA? Dovecot never modifies the S= field, even when it notices that it's wrong. So it's your LDA that writes them wrong.
[Dovecot] Corrupted index cache file issues (Corrupted physical size)
Hello, I'm running Dovecot 1.1.13 on FreeBSD 7.0-RELEASE and I'm receiving a large number of errors much like the following: dovecot: Jun 08 09:27:04 Error: IMAP(x...@x.x): Corrupted index cache file /nfs/x.x/x/Maildir/dovecot.index.cache: Corrupted physical size for uid=2284: 5312 != 5217 dovecot: Jun 08 09:27:04 Error: IMAP(x...@x.x): Corrupted index cache file /nfs/x.x/x/Maildir/dovecot.index.cache: Corrupted physical size for uid=2281: 6992 != 6894 dovecot: Jun 08 09:31:17 Error: IMAP(y...@y.y): Corrupted index cache file /nfs/y.y/y/Maildir/dovecot.index.cache: Corrupted physical size for uid=9241: 14545 != 14453 dovecot: Jun 08 09:37:05 Error: IMAP(z...@z.z): Corrupted index cache file /nfs/z.z/z/Maildir/dovecot.index.cache: Corrupted physical size for uid=22741: 12150 != 12066 These are not occurring for all users, but it occurs frequently for the addresses that do show up in the logs. The maildirs are migrated from a previous courier server. I've tried searching the archives and can't find anything related to this error. Could someone shed some light on it? Other than the error messages, I'm not aware of anything being affected, but I'd like to be sure of it Some info about the configuration: Maildirs are accessed via NFS, related settings are mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes lock_method: dotlock Regards, Brent
[Dovecot] Corrupted Index Cache
I am starting to use Squirrelmail a little bit more to read my mail during the day and I seem to be getting this error more frequently while using Squirrelmail. Just a coincidence or do you think using Squirrelmail might have something to do with it? Corrupted index cache file /home/vmail/jeff/dovecot.index.cache: record points outside file I am using Squirrelmail, IMAPProxy, and Dovecot 1.2RC3. Here is a copy of my dovecot -n. # 1.2.rc3: /usr/local/etc/dovecot.conf # OS: Linux 2.6.29-1-amd64 x86_64 Debian squeeze/sid ext3 base_dir: /var/run/dovecot/ protocols: imap imaps pop3 pop3s ssl_cert_file: /etc/ssl/certs/stikman-godaddy.crt ssl_key_file: /etc/ssl/private/stikman-godaddy.key ssl_cipher_list: ALL:!LOW:!SSLv2 disable_plaintext_auth: no login_dir: /var/run/dovecot//login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login mail_location: maildir:/home/vmail/%u mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 auth default: mechanisms: plain login username_format: %Lu passdb: driver: pam userdb: driver: passwd args: uid=vmail gid=vmail home=/home/vmail/%u socket: type: listen master: path: /var/run/dovecot/auth-master mode: 384 user: vmail group: vmail plugin: sieve: sieve
[Dovecot] Corrupted index cache file
Hello all. I run on a problems? dovecot: Feb 13 08:36:13 Error: IMAP(logis...@fxclub.org): Maildir filename has wrong W value: /var/spool/dovecot/domains/fxclub.org/logistic/.fxclub/1135926892.Re1a4ad89Q1436.newmail.fxclub.org,S=24630,W=2 4849:2, dovecot: Feb 13 08:36:13 Error: IMAP(logis...@fxclub.org): Corrupted index cache file /var/spool/dovecot/domains/fxclub.org/logistic/.fxclub/dovecot.index.cache: Broken virtual size for mail UID 332 dovecot: Feb 13 08:36:13 Info: IMAP(logis...@fxclub.org): Disconnected: Disconnected bytes=435/135596 dovecot: Feb 13 08:36:14 Info: imap-login: Login: user=logis...@fxclub.org, method=PLAIN, rip=194.87.53.162, lip=78.129.148.74, TLS dovecot: Feb 13 08:36:15 Info: IMAP(ch...@omsk.fxclub.org): Disconnected: Logged out bytes=349/5517 dovecot: Feb 13 08:36:15 Info: IMAP(ch...@omsk.fxclub.org): Disconnected: Logged out bytes=63/280 dovecot: Feb 13 08:36:15 Error: IMAP(logis...@fxclub.org): FETCH for mailbox fxclub UID 332 got too little data: 318 vs 319 dovecot: Feb 13 08:36:15 Error: IMAP(logis...@fxclub.org): Maildir filename has wrong W value: /var/spool/dovecot/domains/fxclub.org/logistic/.fxclub/1135926892.Re1a4ad89Q1436.newmail.fxclub.org,S=24630,W=2 4849:2, dovecot: Feb 13 08:36:15 Error: IMAP(logis...@fxclub.org): Corrupted index cache file /var/spool/dovecot/domains/fxclub.org/logistic/.fxclub/dovecot.index.cache: Broken virtual size for mail UID 332 dovecot: Feb 13 08:36:15 Info: IMAP(logis...@fxclub.org): Disconnected: Disconnected bytes=395/135319 It got then reconfigure MS Outlook 2007 SP1 to make a full download of email - not only headers. Then I reconfig it back to headers only - all work fine. How can I fix it? It is safe to delete index files? Self healing seems to not work here. #dovecot -n mail# dovecot -n # 1.1.8: /usr/local/etc/dovecot.conf # OS: FreeBSD 7.0-RELEASE-p9 i386 ufs base_dir: /var/run/dovecot/ log_path: /var/log/dovecot/dovecot.log info_log_path: /var/log/dovecot/dovecot.log protocols: imap imaps pop3 pop3s managesieve listen(default): * listen(imap): * listen(pop3): * listen(managesieve): *:2000 ssl_ca_file: /usr/local/etc/certs/GlobalSignCA.crt ssl_cert_file: /usr/local/etc/certs/fxclub_org.crt ssl_key_file: /usr/local/etc/certs/fxclub_org.key ssl_cipher_list: ALL:!LOW:!SSLv2 login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login login_executable(managesieve): /usr/local/libexec/dovecot/managesieve-login login_greeting_capability(default): yes login_greeting_capability(imap): yes login_greeting_capability(pop3): no login_greeting_capability(managesieve): no login_process_size: 128 login_processes_count: 20 login_max_processes_count: 512 mail_max_userip_connections(default): 256 mail_max_userip_connections(imap): 256 mail_max_userip_connections(pop3): 20 mail_max_userip_connections(managesieve): 10 verbose_proctitle: yes first_valid_gid: 0 mail_privileged_group: mail mail_uid: dmail mail_gid: dmail mail_location(default): maildir:/var/spool/dovecot/domains/%d/%n/ mail_location(imap): maildir:/var/spool/dovecot/domains/%d/%n/ mail_location(pop3): maildir:/var/spool/dovecot/domains/%d/%n/ mail_location(managesieve): mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_executable(managesieve): /usr/local/libexec/dovecot/managesieve mail_process_size: 512 mail_plugins(default): quota imap_quota autocreate mail_plugins(imap): quota imap_quota autocreate mail_plugins(pop3): quota mail_plugins(managesieve): mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 mail_plugin_dir(managesieve): /usr/local/lib/dovecot/managesieve imap_client_workarounds(default): delay-newmail tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail tb-extra-mailbox-sep imap_client_workarounds(pop3): imap_client_workarounds(managesieve): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_client_workarounds(managesieve): sieve_storage(default): sieve_storage(imap): sieve_storage(pop3): sieve_storage(managesieve): % sieve(default): sieve(imap): sieve(pop3): sieve(managesieve): ~/.dovecot.sieve auth default: mechanisms: plain login user: dovecot-auth username_format: %Lu verbose: yes worker_max_count: 100 passdb: driver: ldap args: /usr/local/etc/dovecot-ldap.conf userdb: driver: ldap args: /usr/local/etc/dovecot-ldap.conf socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 user: exim master: path: /var/run/dovecot/auth-master mode: 384 user: dmail plugin: quota:
Re: [Dovecot] Corrupted index cache file
On Fri, Feb 13, 2009 at 12:32 AM, Timo Sirainen t...@iki.fi wrote: On Fri, 2009-02-13 at 00:10 +0100, Diego Liziero wrote: dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): FETCH for mailbox Sent UID 7139 got too little data: 2 vs 11160 .. When this error appears, the last mail attachment is actually truncated. Any idea if the saved data simply isn't written / is truncated, instead of the following message overwriting it? i.e. has this happened to the last message in the mbox file (with uid = next_uid-1 to make sure the last message wasn't actually expunged)? Sorry, no idea. There is always a delay between when the error is written in dovecot.log and when I and ask the user what happened. The truncated message is always one of the latest in the mbox, but I can't tell if it happens when the message is the last one. Maybe a debug message that logs the latest mbox uid together with the FETCH for mailbox Sent UID 7139 got too little data : 2 vs 11160 message could help. Or maybe a debug extra check that reads every newly written mbox mail if it's actually written up to the last byte... Diego.
Re: [Dovecot] Corrupted index cache file
Words by Timo Sirainen [Fri, Feb 13, 2009 at 12:27:07PM -0500]: On Fri, 2009-02-13 at 11:46 +0300, Proskurin Kirill wrote: Hello all. I run on a problems? dovecot: Feb 13 08:36:13 Error: IMAP(logis...@fxclub.org): Maildir filename has wrong W value: The problem is what it says. It has wrong W value: /var/spool/dovecot/domains/fxclub.org/logistic/.fxclub/1135926892.Re1a4ad89Q1436.newmail.fxclub.org,S=24630,W=2 4849:2, Timo, just for the record, what is a wrong W value ? -- Jose Celestino
Re: [Dovecot] Corrupted index cache file
On Fri, 2009-02-13 at 17:47 +, Jose Celestino wrote: Words by Timo Sirainen [Fri, Feb 13, 2009 at 12:27:07PM -0500]: On Fri, 2009-02-13 at 11:46 +0300, Proskurin Kirill wrote: Hello all. I run on a problems? dovecot: Feb 13 08:36:13 Error: IMAP(logis...@fxclub.org): Maildir filename has wrong W value: The problem is what it says. It has wrong W value: /var/spool/dovecot/domains/fxclub.org/logistic/.fxclub/1135926892.Re1a4ad89Q1436.newmail.fxclub.org,S=24630,W=2 4849:2, Timo, just for the record, what is a wrong W value ? S=n contains the file size. W=m contains the message's virtual size, meaning what the file size would be if all line feeds were CRLFs instead of just LFs. Typically you get the correct value from: file size + number of lines in file. signature.asc Description: This is a digitally signed message part
[Dovecot] Corrupted index cache file
Hello I got this error yesterday dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): FETCH for mailbox Sent UID 7139 got too little data: 2 vs 11160 dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): Corrupted index cache file /user/dumontj/.imap/Sent/dovecot.index.cache: Broken virtual size for mail UID 7139 Does remove the cache will cure the problem ? Thanks
Re: [Dovecot] Corrupted index cache file
On Thu, 2009-02-12 at 09:31 +0100, Frank Bonnet wrote: Hello I got this error yesterday dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): FETCH for mailbox Sent UID 7139 got too little data: 2 vs 11160 dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): Corrupted index cache file /user/dumontj/.imap/Sent/dovecot.index.cache: Broken virtual size for mail UID 7139 This still seems to be happening sometimes with mbox. If you ever figure out a way to reproduce it I'd like to know. Does remove the cache will cure the problem ? It should cure the problem itself automatically. Didn't it? signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache file
On 2/12/2009 3:31 AM, Frank Bonnet wrote: dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): FETCH for mailbox Sent UID 7139 got too little data: 2 vs 11160 dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): Corrupted index cache file /user/dumontj/.imap/Sent/dovecot.index.cache: Broken virtual size for mail UID 7139 Does remove the cache will cure the problem ? Version? There have been lots of mbox fixes in recent versions... -- Best regards, Charles
Re: [Dovecot] Corrupted index cache file
On Thu, Feb 12, 2009 at 5:39 PM, Charles Marcus cmar...@media-brokers.com wrote: On 2/12/2009 3:31 AM, Frank Bonnet wrote: dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): FETCH for mailbox Sent UID 7139 got too little data: 2 vs 11160 dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): Corrupted index cache file /user/dumontj/.imap/Sent/dovecot.index.cache: Broken virtual size for mail UID 7139 Does remove the cache will cure the problem ? Version? There have been lots of mbox fixes in recent versions... I'm still getting this error in the 1.1.x hg tree (using a post 1.1.11 snapshot taken 6 days ago), happy to see I'm not the only one. It's really _rare_ and difficult to reproduce. Usually, it happens when a thunderbird user tells dovecot to store a mail with multipart mime attachments in a mbox folder (mostly Sent or Trash folder). When this error appears, the last mail attachment is actually truncated. Filesystem here is ext3 on a two node drbd active/passive cluster (Centos 5.2) Regards, Diego.
Re: [Dovecot] Corrupted index cache file
On Fri, 2009-02-13 at 00:10 +0100, Diego Liziero wrote: dovecot: Feb 11 16:07:27 Error: IMAP(dumontj): FETCH for mailbox Sent UID 7139 got too little data: 2 vs 11160 .. It's really _rare_ and difficult to reproduce. Usually, it happens when a thunderbird user tells dovecot to store a mail with multipart mime attachments in a mbox folder (mostly Sent or Trash folder). When this error appears, the last mail attachment is actually truncated. Any idea if the saved data simply isn't written / is truncated, instead of the following message overwriting it? i.e. has this happened to the last message in the mbox file (with uid = next_uid-1 to make sure the last message wasn't actually expunged)? signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
On 8/13/2008, Mike Abbott ([EMAIL PROTECTED]) wrote: Post your dovecot -n output? Here's some of it. Not very enlightening. Why just some of it? Its not like its that much, and what you may *think* is irrelevant, may in fact be *very* relevant... -- Best regards, Charles
Re: [Dovecot] Corrupted index cache files
I re-ran imaptest on an empty mail store, single client, multiple users, using your dovecot-crlf input file, for a couple hours. Here's the distribution of errors that imaptest reports: 100 Error: user%d[%d]: ...: Header DELIVERED-TO changed 167 Error: user%d[%d]: ...: Header CC changed 417 Error: user%d[%d]: ...: Header TO changed 438 Error: user%d[%d]: ...: Header IN-REPLY-TO changed 445 Error: user%d[%d]: ...: Header REFERENCES changed 485 Error: user%d[%d]: ...: Header FROM changed 503 Error: user%d[%d]: ...: Header SUBJECT changed 544 Error: user%d[%d]: ...: Header MESSAGE-ID changed 764 Error: user%d[%d]: UID=%d INTERNALDATE changed 823 Error: user%d[%d]: UID %d changed 1040 Error: user%d[%d]: uid=%d ...: BODY changed 1052 Error: user%d[%d]: uid=%d ...: BODYSTRUCTURE changed 1546 Error: user%d[%d]: uid=%d ...: ENVELOPE changed 1833 Error: user%d[%d]: uid=%d ...: RFC822.SIZE size changed 3493 Error: user%d[%d]: uid=%d ...: BODY[] size changed 3691 Error: user%d[%d]: uid=%d ...: BODY[HEADER] size changed 3695 Error: user%d[%d]: uid=%d ...: BODY[TEXT] size changed 3880 Error: user%d[%d]: uid=%d ...: BODY[%d] size changed 376365 Error: UIVALIDITY changed: %d - %d During that time dovecot reported no corruption errors. What does it mean that imaptest is having such trouble?
Re: [Dovecot] Corrupted index cache files
On Aug 14, 2008, at 9:56 AM, Mike Abbott wrote: I re-ran imaptest on an empty mail store, single client, multiple users, using your dovecot-crlf input file, for a couple hours. Here's the distribution of errors that imaptest reports: 100 Error: user%d[%d]: ...: Header DELIVERED-TO changed .. During that time dovecot reported no corruption errors. What do you need to do to get corruption errors? Is multiple clients enough? What does it mean that imaptest is having such trouble? I think it just means that imaptest's support for multiple users is broken. I always test it only using a single test user. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
One or more users? Many different users. Post your dovecot -n output? Here's some of it. Not very enlightening. # 1.1.2: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ verbose_proctitle: yes first_valid_uid: 6 last_valid_uid: 5 first_valid_gid: 6 last_valid_gid: 5 mail_access_groups: mail mail_privileged_group: mail mail_location: maildir:~/mail Can you reproduce these easily with my imaptest tool? http://imapwiki.org/ImapTest Some of them. When running imaptest I see these dovecot errors: Corrupted index cache file %s: record continues outside its allocated size Corrupted index cache file %s: record points outside file Corrupted index cache file %s: used_file_size too large Plus I see LOTS of errors from imaptest itself: Error: UIVALIDITY changed: %d - %d Error: user%d[%d]: ...: Header CC changed Error: user%d[%d]: ...: Header DELIVERED-TO changed Error: user%d[%d]: ...: Header FROM changed Error: user%d[%d]: ...: Header IN-REPLY-TO changed Error: user%d[%d]: ...: Header MESSAGE-ID changed Error: user%d[%d]: ...: Header REFERENCES changed Error: user%d[%d]: ...: Header SUBJECT changed Error: user%d[%d]: ...: Header SUBJECT changed Error: user%d[%d]: ...: Header TO changed Error: user%d[%d]: UID %d changed Message-Id Error: user%d[%d]: UID=%d INTERNALDATE changed Error: user%d[%d]: uid=%d ...: BODY changed Error: user%d[%d]: uid=%d ...: BODYSTRUCTURE changed Error: user%d[%d]: uid=%d ...: BODY[%d] size changed Error: user%d[%d]: uid=%d ...: BODY[HEADER] size changed Error: user%d[%d]: uid=%d ...: BODY[TEXT] size changed Error: user%d[%d]: uid=%d ...: BODY[] size changed Error: user%d[%d]: uid=%d ...: ENVELOPE changed Error: user%d[%d]: uid=%d ...: RFC822.SIZE size changed One problem with HFS+ is that hard links are more or less buggy. But v1.1's default settings should include dotlock_use_excl=yes. You maybe should set maildir_copy_with_hardlinks=no, but that shouldn't cause this bug. Changing maildir_copy_with_hardlinks makes no difference. You say that the dovecot errors are harmless because dovecot fixes them, but still there must be some downside, if only a performance hit. Any other info I can provide to help you figure this out?
Re: [Dovecot] Corrupted index cache files
On Aug 13, 2008, at 6:09 PM, Mike Abbott wrote: mail_access_groups: mail mail_privileged_group: mail You probably won't need either of these. And there's no point in setting them to the same value. Can you reproduce these easily with my imaptest tool? http://imapwiki.org/ImapTest Some of them. When running imaptest I see these dovecot errors: Corrupted index cache file %s: record continues outside its allocated size Corrupted index cache file %s: record points outside file Corrupted index cache file %s: used_file_size too large How soon? With what kind of imaptest parameters? I can't reproduce this on my Macbook (OS X 10.5.4, HFS+). Plus I see LOTS of errors from imaptest itself: Error: UIVALIDITY changed: %d - %d Did you test with a single user or multiple users? imaptest's support for testing multiple users is currently a bit broken, but this shouldn't happen with a single user.. Error: user%d[%d]: ...: Header CC changed Error: user%d[%d]: ...: Header DELIVERED-TO changed Error: user%d[%d]: ...: Header FROM changed These shouldn't be happening as long as you use a test mbox file that has unique Message-ID headers. Were you using the recommended dovecot- crlf as the input? You say that the dovecot errors are harmless because dovecot fixes them, but still there must be some downside, if only a performance hit. Any other info I can provide to help you figure this out? Yes, there's a performance hit since all the cached data is lost. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
On Aug 13, 2008, at 6:09 PM, Mike Abbott wrote: Can you reproduce these easily with my imaptest tool? http://imapwiki.org/ImapTest Some of them. When running imaptest I see these dovecot errors: Corrupted index cache file %s: record continues outside its allocated size Corrupted index cache file %s: record points outside file Corrupted index cache file %s: used_file_size too large You could try if it makes a difference to set mmap_disable=yes or lock_method=flock. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
How soon? With what kind of imaptest parameters? I can't reproduce this on my Macbook (OS X 10.5.4, HFS+). I ran imaptest for just a few minutes and saw all these errors in that time. Default imaptest parameters except for user/host names etc. Error: UIVALIDITY changed: %d - %d Did you test with a single user or multiple users? imaptest's support for testing multiple users is currently a bit broken, but this shouldn't happen with a single user.. This error happens with clients=1, clients=10 and clients=100. Error: user%d[%d]: ...: Header CC changed Error: user%d[%d]: ...: Header DELIVERED-TO changed Error: user%d[%d]: ...: Header FROM changed These shouldn't be happening as long as you use a test mbox file that has unique Message-ID headers. Were you using the recommended dovecot-crlf as the input? I passed in your dovecot-crlf file to imaptest, but the message store was cloned from elsewhere first. So you're saying that dovecot only works with special hand-crafted input, and not with real-world mail? Hmmm :) :) Actually I do see plenty of duplicate Message-ID headers in the mail store (not from your input file). I'll investigate that further. Thanks for the tip.
Re: [Dovecot] Corrupted index cache files
On Aug 13, 2008, at 7:12 PM, Mike Abbott wrote: Error: UIVALIDITY changed: %d - %d Did you test with a single user or multiple users? imaptest's support for testing multiple users is currently a bit broken, but this shouldn't happen with a single user.. This error happens with clients=1, clients=10 and clients=100. I mean the user parameter, do you use user=something or user=something %d or similar to specify one/multiple different users where imaptest logs on to? Error: user%d[%d]: ...: Header CC changed Error: user%d[%d]: ...: Header DELIVERED-TO changed Error: user%d[%d]: ...: Header FROM changed These shouldn't be happening as long as you use a test mbox file that has unique Message-ID headers. Were you using the recommended dovecot-crlf as the input? I passed in your dovecot-crlf file to imaptest, but the message store was cloned from elsewhere first. So you're saying that dovecot only works with special hand-crafted input, and not with real-world mail? Hmmm :) :) Actually I do see plenty of duplicate Message-ID headers in the mail store (not from your input file). I'll investigate that further. Thanks for the tip. Dovecot doesn't care about any of that, but imaptest's state tracking requires that Message-ID header uniquely identifies a mail. So the imaptest run was done with a mailbox that already contained other data? Can you break if if you try with an empty mailbox where only imaptest starts appending new mails? PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
On Aug 8, 2008, at 9:59 PM, Mike Abbott wrote: I see these errors more often than I'd like from Dovecot-1.1.2 on Mac OS X 10.5.4 (names and numbers elided): One or more users? Corrupted index cache file %s: Corrupted virtual size for uid=%d: %d != %d Corrupted index cache file %s: Broken virtual size for mail UID %d Corrupted index cache file %s: used_file_size too large How bad are these? As long as Dovecot detects the corruption, the errors are transparent to users since Dovecot fixes them automatically. What should I look for to find out why they happen? Other details: File system is HFS+ Mail store is maildir Hardware is Intel (Mac Pro) Post your dovecot -n output? Can you reproduce these easily with my imaptest tool? http://imapwiki.org/ImapTest One problem with HFS+ is that hard links are more or less buggy. But v1.1's default settings should include dotlock_use_excl=yes. You maybe should set maildir_copy_with_hardlinks=no, but that shouldn't cause this bug. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] Corrupted index cache files
On 8/8/2008 9:59 PM, Mike Abbott wrote: I see these errors more often than I'd like from Dovecot-1.1.2 on Mac OS X 10.5.4 (names and numbers elided): Corrupted index cache file %s: Corrupted virtual size for uid=%d: %d != %d Corrupted index cache file %s: Broken virtual size for mail UID %d Corrupted index cache file %s: used_file_size too large How bad are these? What should I look for to find out why they happen? Other details: File system is HFS+ Mail store is maildir Hardware is Intel (Mac Pro) Any reason you left out your config? -- Best regards, Charles
[Dovecot] Corrupted index cache files
I see these errors more often than I'd like from Dovecot-1.1.2 on Mac OS X 10.5.4 (names and numbers elided): Corrupted index cache file %s: Corrupted virtual size for uid=%d: %d ! = %d Corrupted index cache file %s: Broken virtual size for mail UID %d Corrupted index cache file %s: used_file_size too large How bad are these? What should I look for to find out why they happen? Other details: File system is HFS+ Mail store is maildir Hardware is Intel (Mac Pro) Thanks.
Re: [Dovecot] Dovecot corrupted index cache
On Wed, 2008-06-25 at 09:40 -0700, Michael D. Godfrey wrote: A guess would be that this is likely due to the endianess of the multiple architectures that the index is being accessed with. We have the same issue here across i686/x86_64/sparc. I'm about to post to an older email thread about this as well. This is a good guess. We use a mixture of i386 and x86_64. This is not an endedness conflict, but could be the problem for other reasons. So you use NFS? signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Dovecot corrupted index cache
On Wed, 2008-06-25 at 09:40 -0700, Michael D. Godfrey wrote: A guess would be that this is likely due to the endianess of the multiple architectures that the index is being accessed with. We have the same issue here across i686/x86_64/sparc. I'm about to post to an older email thread about this as well. This is a good guess. We use a mixture of i386 and x86_64. This is not an endedness conflict, but could be the problem for other reasons. So you use NFS? Clients mount off of the server, but dovecot (IMAP) connections are made directly. Home directories are not mounted, so I do not think NFS can be affecting dovecot. I think the problem is due to 2 users using the same account making simultaneous access. It could be that the users need to be on machines with differing architectures (i386 and x86_64 in our case). Anything more I can tell you? Michael
Re: [Dovecot] Dovecot corrupted index cache
On Sat, 2008-06-28 at 10:00 -0700, Michael D Godfrey wrote: On Wed, 2008-06-25 at 09:40 -0700, Michael D. Godfrey wrote: A guess would be that this is likely due to the endianess of the multiple architectures that the index is being accessed with. We have the same issue here across i686/x86_64/sparc. I'm about to post to an older email thread about this as well. This is a good guess. We use a mixture of i386 and x86_64. This is not an endedness conflict, but could be the problem for other reasons. So you use NFS? Clients mount off of the server, but dovecot (IMAP) connections are made directly. Home directories are not mounted, so I do not think NFS can be affecting dovecot. I think the problem is due to 2 users using the same account making simultaneous access. It could be that the users need to be on machines with differing architectures (i386 and x86_64 in our case). I don't really understand. If there is no NFS, then that means you have only one Dovecot server. So how can one Dovecot server be both i386 and x86_64? Or if you mean the client machines are i386/x86-64, Dovecot doesn't even know about them so that doesn't matter. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Dovecot corrupted index cache
A guess would be that this is likely due to the endianess of the multiple architectures that the index is being accessed with. We have the same issue here across i686/x86_64/sparc. I'm about to post to an older email thread about this as well. Michael D. Godfrey wrote: System: Linux 2.6.25.6-55.fc9.x86_64 #1 SMP Tue Jun 10 16:05:21 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux Dovecot: dovecot-1.0.14-8.fc9.x86_64 We have used Dovecot for several years. The error message below appears in the system log quite frequently. I have run a test which shows that this message is due to 2 or more users accessing the same mail (i.e connecting using the same userid). This happens for various reasons including the fact that some users have their client machine set up (typically using Thunderbird) with more than one account within a single Thunderbird configuration. Thus, more than one person may be reading or sending mail from a single Dovecot account. The Dovecot error is: Dovecot disconnects: Inactivity: 13 Time(s) Logged out: 2 Time(s) in IDLE: 12 Time(s) **Unmatched Entries** dovecot: IMAP(godfrey): Connection closed: 4 Time(s) dovecot: IMAP(godfrey): Corrupted index cache file /home/godfrey/Maildir/dovecot.index.cache: invalid record size: 1 Time(s) dovecot: IMAP(igodfrey): Connection closed: 4 Time(s) === or: Dovecot disconnects: Inactivity: 23 Time(s) Logged out: 1 Time(s) in IDLE: 6 Time(s) **Unmatched Entries** dovecot: IMAP(godfrey): Connection closed: 8 Time(s) dovecot: IMAP(godfrey): Corrupted index cache file /home/godfrey/Maildir/.Sent/dovecot.index.cache: invalid record size: 1 Time(s) dovecot: IMAP(godfrey): Corrupted index cache file /home/godfrey/Maildir/dovecot.index.cache: invalid record size: 1 Time(s) dovecot: IMAP(igodfrey): Connection closed: 6 Time(s) -- Dovecot End - = These errors do not seem to cause any serious problem, but it would be good if the problem could be fixed. I have seen other corrupted error reports in the mail archive, but they did not seem to be exactly like this one. -- David Halik System Administrator OIT-CSS Rutgers University [EMAIL PROTECTED]
Re: [Dovecot] Dovecot corrupted index cache
A guess would be that this is likely due to the endianess of the multiple architectures that the index is being accessed with. We have the same issue here across i686/x86_64/sparc. I'm about to post to an older email thread about this as well. This is a good guess. We use a mixture of i386 and x86_64. This is not an endedness conflict, but could be the problem for other reasons. Michael
[Dovecot] Dovecot corrupted index cache
System: Linux 2.6.25.6-55.fc9.x86_64 #1 SMP Tue Jun 10 16:05:21 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux Dovecot: dovecot-1.0.14-8.fc9.x86_64 We have used Dovecot for several years. The error message below appears in the system log quite frequently. I have run a test which shows that this message is due to 2 or more users accessing the same mail (i.e connecting using the same userid). This happens for various reasons including the fact that some users have their client machine set up (typically using Thunderbird) with more than one account within a single Thunderbird configuration. Thus, more than one person may be reading or sending mail from a single Dovecot account. The Dovecot error is: Dovecot disconnects: Inactivity: 13 Time(s) Logged out: 2 Time(s) in IDLE: 12 Time(s) **Unmatched Entries** dovecot: IMAP(godfrey): Connection closed: 4 Time(s) dovecot: IMAP(godfrey): Corrupted index cache file /home/godfrey/Maildir/dovecot.index.cache: invalid record size: 1 Time(s) dovecot: IMAP(igodfrey): Connection closed: 4 Time(s) === or: Dovecot disconnects: Inactivity: 23 Time(s) Logged out: 1 Time(s) in IDLE: 6 Time(s) **Unmatched Entries** dovecot: IMAP(godfrey): Connection closed: 8 Time(s) dovecot: IMAP(godfrey): Corrupted index cache file /home/godfrey/Maildir/.Sent/dovecot.index.cache: invalid record size: 1 Time(s) dovecot: IMAP(godfrey): Corrupted index cache file /home/godfrey/Maildir/dovecot.index.cache: invalid record size: 1 Time(s) dovecot: IMAP(igodfrey): Connection closed: 6 Time(s) -- Dovecot End - = These errors do not seem to cause any serious problem, but it would be good if the problem could be fixed. I have seen other corrupted error reports in the mail archive, but they did not seem to be exactly like this one.
[Dovecot] Corrupted index cache file error (Dovecot 1.0.3)
Greetings - Now that users are beginning to pile up more on our new Dovecot-based IMAP service I'm seeing a small number of entries like this in the logfiles: Corrupted index cache file /mailstore/index/o/ozw100/.INBOX/ dovecot.index.cache : invalid record size We are using Dovecot 1.0.3 with Maildir folders served over NFS from NetApp filers, but the index files are stored on local disk. By a small number I mean 4 or 5 on each of our two IMAP servers since midnight. * Should I be worried? * Is there anything I need to do fix the files? (I'm guessing not as no problem reports have come in, and the logs show the people affected continuing to work throughout the day.) * Would upgrading to 1.0.5 help? (I couldn't see anything in the Release Notes for either 1.0.4 or 1.0.5 about this) Cheers, Mike B-) -- The Computing Service, University of York, Heslington, York Yo10 5DD, UK Tel:+44-1904-433811 FAX:+44-1904-433740 * Unsolicited commercial e-mail is NOT welcome at this e-mail address. *
Re: [Dovecot] Corrupted index cache file error (Dovecot 1.0.3)
Mike Brudenell wrote: Greetings - Now that users are beginning to pile up more on our new Dovecot-based IMAP service I'm seeing a small number of entries like this in the logfiles: Corrupted index cache file /mailstore/index/o/ozw100/.INBOX/dovecot.index.cache : invalid record size We are using Dovecot 1.0.3 with Maildir folders served over NFS from NetApp filers, but the index files are stored on local disk. By a small number I mean 4 or 5 on each of our two IMAP servers since midnight. * Should I be worried? Probably not. * Is there anything I need to do fix the files? (I'm guessing not as no problem reports have come in, and the logs show the people affected continuing to work throughout the day.) I think they probably fix themselves. You could check to see if the same file occurs more than once. If so, I'd try deleting the cache files while the user is disconnected (or at least not having that folder open). * Would upgrading to 1.0.5 help? (I couldn't see anything in the Release Notes for either 1.0.4 or 1.0.5 about this) Possibly, but we still get occassional index errors too (just one today, plus four assertion failures in mail_index_view_sync_end). We're on 1.0.5 with mboxes. Chris -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, [EMAIL PROTECTED] IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Re: [Dovecot] Corrupted index cache file error (Dovecot 1.0.3)
On Fri, 2007-10-05 at 17:40 +0100, Mike Brudenell wrote: Corrupted index cache file /mailstore/index/o/ozw100/.INBOX/ dovecot.index.cache : invalid record size All cache file errors are completely transparent to clients. The cache file gets deleted and Dovecot creates the wanted data from scratch. We are using Dovecot 1.0.3 with Maildir folders served over NFS from NetApp filers, but the index files are stored on local disk. I haven't seen these in my stress tests for a long time. So unless you can figure out a way to reproduce this, I don't think I can do anything about these for now. Maybe v1.1 has already fixed them. :) signature.asc Description: This is a digitally signed message part
