Can you test if https://github.com/dovecot/core/commit/5068b11e594ad7cc1f7cedf2bd9280520e0e534d.patch <https://github.com/dovecot/core/commit/5068b11e594ad7cc1f7cedf2bd9280520e0e534d.patch> fixes it for you?
> On 10 Oct 2019, at 11.34, MAREN ZUBIZARRETA via dovecot <dovecot@dovecot.org> > wrote: > > Hello: > > I have found the same problem reported above by Patrick Cernko affecting > our system and corrupting our messages. Even worse, Outlook 2016 will no > synchronize and the clients cannot see any message, even if there is only one > corrupted mail per mailbox. > > I cannot figure out a feasible workaround for our system, and I can see > that in new version 2.38 the bug is not fixed. > > Will this issue be treated soon? > > Thanks a lot > > Maren Zubizarreta > > > WARNING: using attachment_dir with plugin zlib can corrupt mails > > Patrick Cernko pcernko at mpi-klsb.mpg.de > <mailto:dovecot%40dovecot.org?Subject=Re:%20Re%3A%20WARNING%3A%20using%20attachment_dir%20with%20plugin%20zlib%20can%20corrupt%20mails&In-Reply-To=%3C1ab8edcc-bef4-3b41-d09a-c523674a5031%40mpi-klsb.mpg.de%3E> > Fri Jul 19 17:52:37 EEST 2019 > Previous message: index worker 2.3.7 undefined symbol errors > <https://dovecot.org/list/dovecot/2019-July/116617.html> > Next message: Address family not supported by protocol > <https://dovecot.org/list/dovecot/2019-July/116571.html> > Messages sorted by: [ date ] > <https://dovecot.org/list/dovecot/2019-July/date.html#116570> [ thread ] > <https://dovecot.org/list/dovecot/2019-July/thread.html#116570> [ subject ] > <https://dovecot.org/list/dovecot/2019-July/subject.html#116570> [ author ] > <https://dovecot.org/list/dovecot/2019-July/author.html#116570> > Hello list, hello Dovecot developers, > > this week, I discovered a serious bug in Dovecot, that lead to several > broken mails on our servers. The bug corrupts the first few characters > of the mail header during saving. On our setup, it was almost always > only the very first line of text, that was corrupted. > > Depending on the IMAP client (they seem to request different header > fields, ... during mail access), the bug causes the imap process to hang > up the TCP connection and log errors like this: > > > imap(USERNAME)<4767><TeQP4ASOTK5/AAAB>: Error: Corrupted record in index > > cache file > > /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache: > > UID 489113: Broken fields in mailbox Trash: > > read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))): > > FETCH BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs > > 122 > > In our case that finally grabbed my attention, the client was the users > iphone that did not display any new messages but his Thunderbird did. > > The bug seems to be triggered by a bad "interaction" of attachment_dir > option and zlib plugin. If you use both, you most likely are affected, > too, except you only use zlib plugin for reading previously compressed > stored mails. That's also the workaround we use now: zlib plugin only > enabled in mail_plugins but no plugin/zlib_save set. > > The bug occurs on very specific mails. Due to privacy reasons I could > not provide sample mails here. Storing such mails seems to trigger the > bug reproducible. > > > I attached a very minimal doveconf -n config, that can be used to > trigger the bug. If one of the developers is interested, I can try to > generate an "anonymized" version of such a specific mail that still > causes the issue. I discovered the bug on our productive systems, > running latest Dovecot 2.2 release, but the latest 2.3 I used during > debugging is affected, too. > > During debugging, I also found one hint, that might help find the bug: > If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and > then disable the zlib plugin in mail_plugins, you can call > > doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress > > on test's mailbox with only that one broken mail. > This will display the beginning of the rfc822 mail text until gzip > terminates with "gzip: stdin: unexpected end of file", approximately > after twice the length of the mail HEADER. This might indicate, that > dovecot stores the uncompressed size of the header in it's data > structures although the mail is stored compressed. > > > I also found a very efficient way to find all affected mails in our setup: > > doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \ > grep -a "^[^ ]+ user=" | \ > grep -avF ' hdr=Return-path: ' | \ > grep -av '.* hdr=[[:print:][:space:]]*$' > (runtime for ~6M mails on our servers was 20-30min) > > This can be even more optimized if you have a powerful storage system > with GNU parallel: > > doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid > > mailbox uid seq flags hdr' all | grep -a '^user=' | grep -avF ' > > hdr=Return-path: ' | grep -av '.* hdr=[[:print:][:space:]]*$' || true" > (runtime for ~6M mails on our servers was ~4min) > > The command will give you a list of mails that possibly are affected, > check the full output of > > doveadm fetch -u USERNAME hdr guid GUID | less > > to verify that the header is really broken. > > On our systems I found 39 mails within ~12M mails. > > I was able to recover these mails "manually" by reconstructing the > Return-Path header line, importing the fixed mails and expunging the > corrupt ones. Before importing, I had to disable zlib_save option obviously. > > Best regards, > -- > Patrick Cernko <pcernko at mpi-klsb.mpg.de > <https://dovecot.org/mailman/listinfo/dovecot>> +49 681 9325 5815 > Joint Administration: Information Services and Technology > Max-Planck-Institute fuer Informatik & Softwaresysteme > -------------- next part -------------- > # 2.3.6.1 (d124cc84b): /etc/dovecot/dovecot.conf > # OS: Linux 4.14.127.1.amd64-smp x86_64 Debian 9.9 > # Hostname: adove.mpi-klsb.mpg.de <http://adove.mpi-klsb.mpg.de/> > listen = * > mail_attachment_dir = /var/vmail/attachments > mail_attachment_fs = posix > mail_gid = nogroup > mail_home = /var/vmail/%u > mail_location = mdbox:~/mdbox > mail_plugins = " zlib" > mail_uid = nobody > passdb { > args = /etc/dovecot/userdb > driver = passwd-file > } > plugin { > zlib_save = gz > } > protocols = imap > userdb { > args = /etc/dovecot/userdb > driver = passwd-file > } > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: smime.p7s > Type: application/pkcs7-signature > Size: 5324 bytes > Desc: S/MIME Cryptographic Signature > URL: > <https://dovecot.org/pipermail/dovecot/attachments/20190719/3da620ad/attachment-0001.p7s > > <https://dovecot.org/pipermail/dovecot/attachments/20190719/3da620ad/attachment-0001.p7s>>
