Bug#661008: dovecot-imapd: Corrupted index cache file
Following up on my previous email, we fixed the issue by enabling maildir_broken_filename_sizes = yes Kind Regards Dominic
Bug#661008: dovecot-imapd: Corrupted index cache file
After upgrading from dovecot-core 2.2.13-12~deb8u4 (jessie) to 2.2.27-3+deb9u2
(stretch),
we are seeing this issue appearing again on a few of our customer mailboxes
with lots of
entries inside mail.log. It seems that in our case especially large mailboxes
containing older
emails dating back a few years are affected.
We already tried to rebuild dovecot mailbox cache by stopping the dovecot
service
and deleting the dovecot.index.cache without success:
find /var/mail -name 'dovecot.index.cache' -exec rm {} \;
Any sugesstions on how to debug this issue?
Regards
Dominic
Bug#661008: dovecot-imapd: Corrupted index cache file
The parts with the supposedly unread emails is now solved. Here is what happened: - I created public folders with per-user seen flags, granted delete permission but forgot to add expunge rights to the users. - Email clients were configured to delete mails (including expunge) or move mails to trash folders instead of just marking them as deleted. For Thunderbird, this means that mails that are marked as deleted are not shown. - Now when some clients deleted mails in public folders, the mails were only marked as deleted. This was not a problem for the user who deleted the mail, because for him the message was both read and deleted. However, other users of the same public mailbox had individual seen flags, and for them the mails were both UNSEEN and DELETED. - The default setting in Thunderbird retrieves a list of folders from the mail server which includes the number of unseen mails per folder. This number is displayed initially, but as soon as the user clicks on the folder, messages that are deleted are not counted as unseen and the unseen message indicator disappears. Important: The errors reported in the initial bug report still occur (corrupted index.cache files). Sorry for misleading you with this configuration error, but it took me a long time to figure this out... Regards Thomas -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#661008: dovecot-imapd: Corrupted index cache file
Some more input on this: Roundcube shows unread messages in some folders in the folder pane, but when I click on it and inspect the folder contents, no new messages can be found. The folder pane continuously shows that the folder contains some unread messages. So this is definitely not a client issue. Maybe this is what happens (just a guess): - a new mail is placed in a public folder - my mail client sees that there is a new mail, and dovecot stores that information in my index file for this public folder - another user moves that mail out of the folder - after a restart of my mail client, dovecot checks my index file and sees that there is an unread message, but dovecot does not test if that message is still in that folder Regards Thomas -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#661008: dovecot-imapd: Corrupted index cache file
On 03/06/2012 09:26 PM, Jaldhar H. Vyas wrote:
> On Fri, 24 Feb 2012, Thomas Herrmann wrote:
>> On 02/24/2012 12:42 AM, Timo Sirainen wrote:
>>> Anyway, I don't have time to try to debug bugs that may have been
>>> fixed already. If this keeps happening with v2.0.18 or v2.1.1 I'm
>>> happy to help.
Sorry for the long delay; but I have finally found some time to upgrade
to 2.1.7 (current debian testing packages). The problem still occurrs
exactly as described in the first posts. After the upgrade, I did this
in the mail folder:
find . -name dovecot.index.cache -exec rm {} \;
and today, this came up in the mail.err log file:
Jul 23 10:20:08 mailsrv2 dovecot: imap(xx...@adwin.de): Error:
Corrupted index cache file
/home/vmail/adwin.de/XX/Maildir/.SomeFolder/dovecot.index.cache:
record continues outside its allocated size
The folder in question is a regular mail folder that is symlinked into
the public folder structure. Mails are placed in that folder by the
users sieve script. The folder is then accessed as a public folder by
multiple clients.
My mail client still shows new mails in some folders, and when you click
on the folder, there are no new folders. I assume that the problems
are connecte
d, but I don't know how.
I hope that we can track this down now, let me know what you need.
Regards
Thomas
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#661008: dovecot-imapd: Corrupted index cache file
I found some time to further debug my problem, and I suspect there is a
problem with my namespace configuration (full dovecot -n output is in
the original bug report):
namespace:
type: private
separator: /
inbox: yes
list: yes
subscriptions: yes
namespace:
type: public
separator: /
prefix: ADwin/
location: maildir:/home/vmail/public:INDEX=/home/vmail/adwin.de/%n/public
list: children
Since the problem with "fake" unread emails occurs only in public
folders, I would like to know if that is a valid namespace config or if
I should move "private" to a third namespace with a non-empty prefix.
When I traced the IMAP dialogue with wireshark, I saw that the public
folders are returned for the commands
5 lsub "" "*"
and
6 lsub "" "ADwin/*"
Is that the expected result?
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
STARTTLS AUTH=PLAIN] Dovecot ready.
1 authenticate plain
+
secret.==
1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT
IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE
QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS ACL
RIGHTS=texk] Logged in
2 namespace
* NAMESPACE (("" "/")) NIL (("ADwin/" "/"))
2 OK Namespace completed.
3 ENABLE CONDSTORE
* ENABLED CONDSTORE
3 OK Enabled.
4 ID ("name" "Icedove" "version" "8.0")
* ID NIL
4 OK ID completed.
5 lsub "" "*"
* LSUB () "/" "Junk"
* LSUB () "/" "Drafts"
* LSUB () "/" "INBOX"
* LSUB () "/" "Listen/Arpwatch"
* LSUB () "/" "Listen/Buildmaster"
* LSUB () "/" "Listen/CSTA"
[...] snipped about 50 other folders
* LSUB () "/" "Sent"
* LSUB () "/" "Trash"
* LSUB () "/" "ADwin/Faxe_JPG"
* LSUB () "/" "ADwin/Info"
* LSUB () "/" "ADwin/Virus"
* LSUB () "/" "ADwin/Outgoing_Support"
5 OK Lsub completed.
6 lsub "" "ADwin/*"
* LSUB () "/" "ADwin/Faxe_JPG"
* LSUB () "/" "ADwin/Info"
* LSUB () "/" "ADwin/Virus"
* LSUB () "/" "ADwin/Outgoing_Support"
6 OK Lsub completed.
[...]
62 STATUS "ADwin/Info" (UIDNEXT MESSAGES UNSEEN RECENT)
* STATUS "ADwin/Info" (MESSAGES 1195 RECENT 0 UIDNEXT 1461 UNSEEN 5)
62 OK Status completed.
[...]
74 select "ADwin/Info" (CONDSTORE)
* OK [CLOSED] Previous mailbox closed.
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk Junk $label4
erledigt $Forwarded $MDNSent bearbeitet $label3 erledigt_akk erledigt_bk
erledigt_yr faxausdrucken zu_erledigen_akk $label1 $label2 $label5
zu_erledigen_bk)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk
Junk $label4 erledigt $Forwarded $MDNSent bearbeitet $label3
erledigt_akk erledigt_bk erledigt_yr faxausdrucken zu_erledigen_akk
$label1 $label2 $label5 zu_erledigen_bk \*)] Flags permitted.
* 1195 EXISTS
* 0 RECENT
* OK [UNSEEN 1148] First unseen.
* OK [UIDVALIDITY 1323185171] UIDs valid
* OK [UIDNEXT 1461] Predicted next UID
* OK [HIGHESTMODSEQ 1543] Highest
74 OK [READ-WRITE] Select completed.
75 myrights "ADwin/Info"
* MYRIGHTS "ADwin/Info" lrwstipd
75 OK Myrights completed.
76 UID fetch 1461:* (FLAGS)
* 1195 FETCH (UID 1460 FLAGS (\Seen))
76 OK Fetch completed.
I was wondering if this behaviour is related to the one reported in the
dovecot mailinglist a few days ago (Subject: LIST-STATUS issue).
Regards,
Thomas Herrmann
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#661008: dovecot-imapd: Corrupted index cache file
On Fri, 24 Feb 2012, Thomas Herrmann wrote: On 02/24/2012 12:42 AM, Timo Sirainen wrote: Anyway, I don't have time to try to debug bugs that may have been fixed already. If this keeps happening with v2.0.18 or v2.1.1 I'm happy to help. The problem occurs on a production server running debian, so which packages do you suggest? I am following the dovecot mailing list and read about [1], but it says those packages should not be used for servers that must be stable. On my home network I am running debian testing (contains dovecot 2.0.15) without any problems. Do you know if you fix to cache file handling is already contained in that version? I usually trust debian testing for stable servers, if I only choose a few, widely used and reviewed packages. I could compile from source, but I hardly believe that I end up with something more stable than in [1], especially regarding security updates. The Debian dovecot packages have fallen behind a bit. I'll try to get them updated to 2.0.18 soon. (2.1.1 in experimental maybe.) -- Jaldhar H. Vyas -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#661008: dovecot-imapd: Corrupted index cache file
On 02/24/2012 12:42 AM, Timo Sirainen wrote: > Anyway, I don't have time to try to debug bugs that may have been > fixed already. If this keeps happening with v2.0.18 or v2.1.1 I'm > happy to help. The problem occurs on a production server running debian, so which packages do you suggest? I am following the dovecot mailing list and read about [1], but it says those packages should not be used for servers that must be stable. On my home network I am running debian testing (contains dovecot 2.0.15) without any problems. Do you know if you fix to cache file handling is already contained in that version? I usually trust debian testing for stable servers, if I only choose a few, widely used and reviewed packages. I could compile from source, but I hardly believe that I end up with something more stable than in [1], especially regarding security updates. Regards, Thomas [1] http://wiki2.dovecot.org/PrebuiltBinaries#Automatically_Built_Packages -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#661008: dovecot-imapd: Corrupted index cache file
On 23.2.2012, at 16.36, Thomas Herrmann wrote: > Feb 22 09:19:41 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index > cache file /home/vmail/domain.de/user1/public/.foo/dovecot.index.cache: field > header names corrupted > Feb 23 13:20:27 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index > cache file /home/vmail/domain.de/user2/Maildir/.bar/dovecot.index.cache: > record continues outside its allocated size > Feb 23 13:33:30 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index > cache file /home/vmail/domain.de/user3/public/.baz/dovecot.index.cache: > invalid record size Well, these shouldn't normally happen. I recently did one fix to cache file handling, but I don't know if it's related. > # 1.2.15: /etc/dovecot/dovecot.conf Anyway, I don't have time to try to debug bugs that may have been fixed already. If this keeps happening with v2.0.18 or v2.1.1 I'm happy to help. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#661008: dovecot-imapd: Corrupted index cache file
Package: dovecot-imapd Version: 1:1.2.15-7 Severity: important Every few hours, I see messages like the following in the logs: Feb 22 09:19:41 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index cache file /home/vmail/domain.de/user1/public/.foo/dovecot.index.cache: field header names corrupted Feb 23 13:20:27 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index cache file /home/vmail/domain.de/user2/Maildir/.bar/dovecot.index.cache: record continues outside its allocated size Feb 23 13:33:30 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index cache file /home/vmail/domain.de/user3/public/.baz/dovecot.index.cache: invalid record size (username, domain and path modified for privacy reasons) About 80% of the errors are "invalid record size", but the others occur frequently as well. The server did not crash or anything that could explain the corrupted files. It is a virtual machine running on a SSD. Unmounting the mail store (/home) and checking the filesystem did not yield any errors. Dovecot should repair corrupted cache files automatically, and I remember reading in the mailing lists that errors like the ones I see could be ignored after a power failure, but that each cache file should be listed no more than once. But I see the same file reported several times: Feb 21 09:44:11 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index cache file /home/vmail/domain.de/user1/public/.BCC/dovecot.index.cache: invalid record size Feb 22 10:58:04 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index cache file /home/vmail/domain.de/user1/public/.BCC/dovecot.index.cache: record continues outside its allocated size Feb 22 18:09:11 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index cache file /home/vmail/domain.de/user1/public/.BCC/dovecot.index.cache: record continues outside its allocated size Feb 23 13:33:30 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index cache file /home/vmail/domain.de/user1/public/.BCC/dovecot.index.cache: invalid record size Feb 23 15:18:07 mailsrv2 dovecot: IMAP(us...@domain.de): Corrupted index cache file /home/vmail/domain.de/user1/public/.BCC/dovecot.index.cache: invalid record size I noticed that the problems usually occur in large mail folders (>2 mails), some of them being public folders that are symlinked to a users Maildir. But the problem also occurs with regular mailboxes. At the same time, users sometimes see mail folders with many unread messages, and as soon as they click on the folder (i.e. in Thunderbird) the mail folder is correctly displayed containing no unread messages. It is not a client issue, because I debugged it with wireshark and saw dovecot stating that there were unseen messages. I also set up an imap account on a different machine, and the same erratic behaviour occurred there. So I sometimes do have reproducible situations, which usually resolve themselves when new mail comes into the folder or mails get deleted. Further information: # mount | grep home /dev/mapper/mailstore-mails on /home type xfs (rw,noatime,logbufs=8,logbsize=131072) root@mailsrv2:~# dovecot -n # 1.2.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.4 xfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps managesieve listen(default): *:143 [::]:143 *:144 [::]:144 listen(imap): *:143 [::]:143 *:144 [::]:144 listen(managesieve): *:4190 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(managesieve): login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login login_process_per_connection: no mail_max_userip_connections(default): 100 mail_max_userip_connections(imap): 100 mail_max_userip_connections(managesieve): 10 mail_privileged_group: mail mail_location: maildir:/home/vmail/domain.de/%n/Maildir fsync_disable: yes # note: I just added mbox_write_locks today, but errors ocurred nonetheless mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_plugins(default): acl imap_acl mail_plugins(imap): acl imap_acl mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: public separator: / prefix: ADwin/ location: maildir:/home/vmail/public:INDEX=/home/vmail/domain.de/%n/public list: children lda: postmaster_address: postmas...@domain.de log_path: /home/vmail/dovecot-deliver.log mail_plugins: sieve acl global_script_path: /home/vmail/globalsieverc auth_socket_path: /var/run/dovecot/auth-master fsync_disable: no auth default: passdb: driver: sql
