Re: doveadm sync - I/O has stalled

[Timo Sirainen via dovecot]

We haven't found any specific bugs with lib-ssl-iostream so far, but we did 
find istream-multiplex bug that could cause hangs with doveadm-doveadm 
connections. Could be worth testing if it helps: 
https://github.com/dovecot/core/commit/bbe546bc637a6ac5c9e91fc8abefce62e4950d07

> On 30. Dec 2022, at 14.37, songliny  wrote:
> 
> Hi All,
>   We are using dovecot 2.3.19.1
> We created a account with more than 1000 mail folders in Maildir format to 
> reproduce the issue.
> After weeks of testing, we have found a logic that may cause dsync to 
> encounter the error - no activity for 900 seconds
>   The function, dsync_ibc_stream_input, is the callback function after some 
> data are ready for be read.
> This is part of what it does.
>   o_stream_cork(ibc->output);
> ibc->ibc.io_callback(ibc->ibc.io_context);
> o_stream_uncork(ibc->output);
> Normally, ibc->ibc.io_callback(ibc->ibc.io_context) reads some data and 
> then processes it.
>   But when dsync connects over tcps,
> it uses function implementations in lib-ssl-iostream to send and receive data.
> And this simplified call stack would result in some data are read when 
> calling o_stream_uncork
> 
> o_stream_uncork => o_stream_flush => o_stream_ssl_flush_buffer => 
> openssl_iostream_bio_sync => openssl_iostream_bio_input
> 
> If some data arrive after ibc->ibc.io_callback(ibc->ibc.io_context) and 
> before o_stream_uncork,
> o_stream_uncork would read the data and then return.
> After o_stream_uncork returns, dsync then waits for new data to be read or 
> written.
> But because the data had been read in o_stream_uncork, and there may be no 
> new data to be read,
> dsync may then wait until timeout is met.
>   It may happen, but it is hard to reproduce.
> If you also encounter this situation, you may try to use dsync over tcp 
> connection.
> It does not read data when calling o_stream_uncork.
> As a result, it may not stuck.
>   Back to the error itself,
> Maybe openssl-stream should not read data when doing uncork(flush)?
>   Song-Lin Yang

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [2.3.8] possible replication issue

[Timo Sirainen via dovecot]

I think there's a good chance that upgrading both will fix it. The bug already 
existed in old versions, it just wasn't normally triggered. Since v2.3.8 this 
situation is triggered on one dsync side, so the v2.3.9 fix needs to be on the 
other side.

> On 5. Dec 2019, at 8.34, Piper Andreas via dovecot  
> wrote:
> 
> Hello,
> 
> upgrading to 2.3.9 unfortunately does *not* solve this issue:
> 
> I upgraded one of my replicators from 2.3.7.2 to 2.3.9 and after some seconds 
> replication stopped. The other replicator remained with 2.3.7.2. After 
> downgrading to 2.3.7.2 replication is again working fine.
> 
> I did not try to upgrade both replicators up to now, as this is a live 
> production system. Is there a chance, that upgrading both replicators will 
> solve the problem?
> 
> The machines are running Ubuntu 18.04
> 
> Any help is appreciated.
> 
> Thanks,
> Andreas
> 
> Am 18.10.19 um 13:52 schrieb Carsten Rosenberg via dovecot:
>> Hi,
>> some of our customers have discovered a replication issue after
>> upgraded from 2.3.7.2 to 2.3.8.
>> Running 2.3.8 several replication connections are hanging until defined
>> timeout. So after some seconds there are $replication_max_conns hanging
>> connections.
>> Other replications are running fast and successful.
>> Also running a doveadm sync tcp:... is working fine for all users.
>> I can't see exactly, but I haven't seen mailboxes timeouting again and
>> again. So I would assume it's not related to the mailbox.
>> From the logs:
>> server1:
>> Oct 16 08:29:25 server1 dovecot[5715]:
>> dsync-local(userna...@domain.com): Error:
>> dsync(172.16.0.1): I/O has stalled, no activity for 600 seconds (version
>> not received)
>> Oct 16 08:29:25 server1 dovecot[5715]:
>> dsync-local(userna...@domain.com): Error:
>> Timeout during state=master_recv_handshake
>> server2:
>> Oct 16 08:29:25 server2 dovecot[8113]: doveadm: Error: read(server1)
>> failed: EOF (last sent=handshake, last recv=handshake)
>> There aren't any additional logs regarding the replication.
>> I have tried increasing vsz_limit or reducing replication_max_conns.
>> Nothing changed.
>> --
>> Both customers have 10k+ users. Currently I couldn't reproduce this on
>> smaller test systems.
>> Both installation were downgraded to 2.3.7.2 to fix the issue for now
>> --
>> I've attached a tcpdump showing the client showing the client stops
>> sending any data after the mailbox_guid table headers.
>> Any idea what could be wrong here or the debug this issue?
>> Thanks.
>> Carsten Rosenberg
> 
> 
> -- 
> 
> Dr. Andreas Piper, Hochschulrechenzentrum der Philipps-Univ. Marburg
>  Hans-Meerwein-Straße 6, 35032 Marburg, Germany
> Phone: +49 6421 28-23521  Fax: -26994  E-Mail: pi...@hrz.uni-marburg.de 
> 

Re: Dovecot v2.3.8 released

[Timo Sirainen via dovecot]

On 20 Oct 2019, at 11.37, Tom Sommer via dovecot  wrote:
> 
>>> This is mail_debug from one of the accounts in question:
>>> Oct 18 13:39:37 imap()<7552>: Debug: Mailbox INBOX: 
>>> Mailbox opened because: SELECT
>>> Oct 18 13:39:37 imap()<7552>: Debug: Mailbox INBOX: 
>>> UID 17854: Opened mail because: prefetch
>>> Oct 18 13:39:37 imap()<7552>: Debug: Mailbox INBOX: 
>>> UID 17854: Opened mail because: full mail
>> ..
>>> Oct 18 13:39:48 imap()<7552>: Debug: Mailbox INBOX: 
>>> UID 17947: Opened mail because: full mail
>> Quite a lot of mail downloads for a single session. I wonder if the
>> user really had that many new mails or if they were being redownloaded
>> for some reason?
> 
> They might redownload because of UID FETCH failing?

The client successfully downloaded all but the last mail. So it should be 
redownloading only the latest one, not all of them. (I don't think there are 
any clients stupid enough to redownload everything..)

>>> Oct 18 13:40:56 imap()<7552>: Debug: Mailbox Junk: 
>>> Mailbox opened because: autoexpunge
>>> Oct 18 13:40:56 imap()<7552>: Debug: Mailbox Junk 
>>> E-mail: Mailbox opened because: autoexpunge
>>> Oct 18 13:40:56 imap()<7552>: Info: Connection 
>>> closed: read(size=7902) failed: Connection reset by peer (UID FETCH running 
>>> for 0.542 + waiting input/output for 78.357 secs, 60 B in + 39221480+8192 B 
>>> out, state=wait-output) in=290 out=39401283 deleted=0 expunged=0 trashed=0 
>>> hdr_count=0 hdr_bytes=0 body_count=94 body_bytes=39210315
>> state=wait-output means Dovecot was waiting for client to read the
>> data it is sending. In v2.3.7 there was some changes related to this,
>> but were you previously successfully running v2.3.7? In v2.3.8 I can't
>> really think of such changes.
> 
> Yes, we were successfully running 2.3.7.2 before, the issue started just 
> after the upgrade
> 
> It can't be related to changes in the indexes? Increasing I/O
> 
> There were no input/output errors in the logs prior to 2.3.8

How large are the IO latencies now and before? The IO wait% in e.g. iostat? And 
load average in general?

I can't see any reason for IO to be different in v2.3.8 than v2.3.7. The only 
thing even close to it is the one index file bugfix. I did some further testing 
with it and I can't see it doing any more work now than it used to.

Re: Dovecot v2.3.8 released

[Timo Sirainen via dovecot]

> On 18 Oct 2019, at 13.43, Tom Sommer via dovecot  wrote:
> 
>> I am seeing a lot of errors since the upgrade, on multiple client accounts:
>> Info: Connection closed: read(size=7902) failed: Connection reset by
>> peer (UID FETCH running for 0.242 + waiting input/output for 108.816
>> secs, 60 B in + 24780576+8192 B out, state=wait-output)
>> Using NFS storage (not running with the mail_nfs_index or mail_nfs_storage)
>> Was something changed in terms of IO/Idle timeouts?
> 
> We are also seeing different I/O patterns since the upgrade, more I/O is 
> being used than normal.

What was the previous version you were running?

> This is mail_debug from one of the accounts in question:
> 
> Oct 18 13:39:37 imap()<7552>: Debug: Mailbox INBOX: 
> Mailbox opened because: SELECT
> Oct 18 13:39:37 imap()<7552>: Debug: Mailbox INBOX: UID 
> 17854: Opened mail because: prefetch
> Oct 18 13:39:37 imap()<7552>: Debug: Mailbox INBOX: UID 
> 17854: Opened mail because: full mail
..
> Oct 18 13:39:48 imap()<7552>: Debug: Mailbox INBOX: UID 
> 17947: Opened mail because: full mail

Quite a lot of mail downloads for a single session. I wonder if the user really 
had that many new mails or if they were being redownloaded for some reason?

> Oct 18 13:40:56 imap()<7552>: Debug: Mailbox Junk: 
> Mailbox opened because: autoexpunge
> Oct 18 13:40:56 imap()<7552>: Debug: Mailbox Junk 
> E-mail: Mailbox opened because: autoexpunge
> Oct 18 13:40:56 imap()<7552>: Info: Connection closed: 
> read(size=7902) failed: Connection reset by peer (UID FETCH running for 0.542 
> + waiting input/output for 78.357 secs, 60 B in + 39221480+8192 B out, 
> state=wait-output) in=290 out=39401283 deleted=0 expunged=0 trashed=0 
> hdr_count=0 hdr_bytes=0 body_count=94 body_bytes=39210315

state=wait-output means Dovecot was waiting for client to read the data it is 
sending. In v2.3.7 there was some changes related to this, but were you 
previously successfully running v2.3.7? In v2.3.8 I can't really think of such 
changes.

Re: Coredump v2.3.8 specific msg fetch, corrupted record in index cache, Broken physical size

[Timo Sirainen via dovecot]

On 18 Oct 2019, at 15.36, Erik de Waard via dovecot  wrote:
> 
> Hi, i'm getting a coredump on a specific msg, i've attached the gdb.
> 
> file on disk i noticed W= is missing.
> 1571209735.M744550P1608.rwvirtual65,S=15886:2,S
..
> mail.log
> Oct 18 14:41:39 rwvirtual10 dovecot: imap(john...@company.nl 
> )<15868>: Error: Mailbox 
> INBOX.Debug: UID=1041: 
> read(/data/mail/company.nl/users/johndoe/Maildir/.Debug/cur/1571209735.M744550P1608.rwvirtual65,S=15886:2,S
>  
> )
>  failed: Cached message size smaller than expected (15886 < 16367, 
> box=INBOX.Debug, UID=1041) (read reason=mail stream)
> Oct 18 14:41:39 rwvirtual10 dovecot: imap(john...@company.nl 
> )<15868>: Error: Corrupted 
> record in index cache file 
> /data/indexes/john...@company.nl/.Debug/dovecot.index.cache 
> : UID 1041: Broken 
> physical size in mailbox INBOX.Debug: 
> read(/data/mail/company.nl/users/johndoe/Maildir/.Debug/cur/1571209735.M744550P1608.rwvirtual65,S=15886:2,S
>  
> )
>  failed: Cached message size smaller than expected (15886 < 16367, 
> box=INBOX.Debug, UID=1041)
> Oct 18 14:41:39 rwvirtual10 dovecot: imap(john...@company.nl 
> )<15868>: Panic: file istream.c: 
> line 315 (i_stream_read_memarea): assertion failed: (old_size <= _stream->pos 
> - _stream->skip)

The missing W shouldn't matter, but the S size is wrong. So the error is 
expected, but the panic isn't. I tried to reproduce this, but I couldn't get 
the panic to happen. Do you still have the file? Could you send it to me? You 
can also put it through https://dovecot.org/tools/maildir-obfuscate.pl which 
should remove all sensitive content but hopefully still contain enough to 
reproduce the bug.

Re: BUG: Mailbox renaming algorithm got into a potentially infinite loop, aborting

[Timo Sirainen via dovecot]

On 25 Sep 2019, at 17.03, Alex Ha via dovecot mailto:dovecot@dovecot.org>> wrote:
> 
> Hi all!
> 
> I have two dovecot servers with dsync replication over tcp.
> Replication works fine except for one user.
> 
> # doveadm replicator status
> username  
>priority fast sync full sync success sync failed
> custo...@example.com 
> none 00:00:33  07:03:23  
> 03:22:31 y 
> 
> If i run dsync manually, i get the following error message:
> 
> dsync-local(custo...@example.com ): Debug: brain 
> M: -- Mailbox renamed, restart sync --
> dsync-local(custo...@example.com ): Error: BUG: 
> Mailbox renaming algorithm got into a potentially infinite loop, aborting
> dsync-local(custo...@example.com ): Error: 
> Mailbox INBOX.Foldername sync: mailbox_rename failed: Invalid mailbox name 
> 'Foldername-temp-1': Missing namespace prefix 'INBOX.'
> 
I've never fixed this because I haven't figured out how to reproduce it. If it 
happens with you all the time, could you try:

 - Get a copy of both replica sides, e.g. under /tmp/replica1 and /tmp/replica2
 - Make sure dsync still crashes with them, e.g. doveadm -o 
mail=maildir:/tmp/replica1 sync maildir:/tmp/replica2
 - Delete all mails and dovecot.index* files (but not dovecot.mailbox.log)
 - Make sure dsync still crashes
 - Send me the replicas - they should no longer contain anything sensitive

As for fixing, you could see if deleting dovecot.mailbox.log from both replicas 
happens to fix this.

Re: Using attachment_dir with plugin zlib corrupt mails

[Timo Sirainen via dovecot]

Can you test if 
https://github.com/dovecot/core/commit/5068b11e594ad7cc1f7cedf2bd9280520e0e534d.patch
 

 fixes it for you?

> On 10 Oct 2019, at 11.34, MAREN ZUBIZARRETA via dovecot  
> wrote:
> 
> Hello:
>  
>   I have found the same problem reported above by Patrick Cernko affecting 
> our system and corrupting our messages. Even worse, Outlook 2016 will no 
> synchronize and the clients cannot see any message, even if there is only one 
> corrupted mail per mailbox.
>  
>   I cannot figure out a feasible workaround for our system, and I can see 
> that in new version 2.38 the bug is not fixed.
>  
>  Will this issue be treated soon?
>  
>  Thanks a lot
>  
>  Maren Zubizarreta
>  
> 
> WARNING: using attachment_dir with plugin zlib can corrupt mails
> 
> Patrick Cernko pcernko at mpi-klsb.mpg.de 
> 
> Fri Jul 19 17:52:37 EEST 2019
> Previous message: index worker 2.3.7 undefined symbol errors 
> 
> Next message: Address family not supported by protocol 
> 
> Messages sorted by: [ date ] 
>  [ thread ] 
>  [ subject ] 
>  [ author ] 
> 
> Hello list, hello Dovecot developers,
>  
> this week, I discovered a serious bug in Dovecot, that lead to several 
> broken mails on our servers. The bug corrupts the first few characters 
> of the mail header during saving. On our setup, it was almost always 
> only the very first line of text, that was corrupted.
>  
> Depending on the IMAP client (they seem to request different header 
> fields, ... during mail access), the bug causes the imap process to hang 
> up the TCP connection and log errors like this:
>  
> > imap(USERNAME)<4767>: Error: Corrupted record in index 
> > cache file 
> > /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache:
> >  UID 489113: Broken fields in mailbox Trash: 
> > read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))):
> >  FETCH BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs 
> > 122
>  
> In our case that finally grabbed my attention, the client was the users 
> iphone that did not display any new messages but his Thunderbird did.
>  
> The bug seems to be triggered by a bad "interaction" of attachment_dir 
> option and zlib plugin. If you use both, you most likely are affected, 
> too, except you only use zlib plugin for reading previously compressed 
> stored mails. That's also the workaround we use now: zlib plugin only 
> enabled in mail_plugins but no plugin/zlib_save set.
>  
> The bug occurs on very specific mails. Due to privacy reasons I could 
> not provide sample mails here. Storing such mails seems to trigger the 
> bug reproducible.
>  
>  
> I attached a very minimal doveconf -n config, that can be used to 
> trigger the bug. If one of the developers is interested, I can try to 
> generate an "anonymized" version of such a specific mail that still 
> causes the issue. I discovered the bug on our productive systems, 
> running latest Dovecot 2.2 release, but the latest 2.3 I used during 
> debugging is affected, too.
>  
> During debugging, I also found one hint, that might help find the bug: 
> If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and 
> then disable the zlib plugin in mail_plugins, you can call
>  
> doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress
>  
> on test's mailbox with only that one broken mail.
> This will display the beginning of the rfc822 mail text until gzip 
> terminates with "gzip: stdin: unexpected end of file", approximately 
> after twice the length of the mail HEADER. This might indicate, that 
> dovecot stores the uncompressed size of the header in it's data 
> structures although the mail is stored compressed.
>  
>  
> I also found a very efficient way to find all affected mails in our setup:
>  
> doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \
>grep -a "^[^ ]+ user=" | \
>grep -avF ' hdr=Return-path: ' | \
>grep -av '.* hdr=[[:print:][:space:]]*$'
> (runtime for ~6M mails on our servers was 20-30min)
>  
> This can be even more optimized if you have a powerful storage system 
> with GNU parallel:
> > doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid 
> > mailbox uid seq flags hdr' all | grep -a '^user=' | grep -avF ' 
> > hdr=Return-path: ' | grep -av '.* 

Re: [ext] dovecot 2.3.7.2-1~bionic: Performance issues caused by excessive IO to ~/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.tmp

[Timo Sirainen via dovecot]

On 1 Oct 2019, at 16.45, Ralf Hildebrandt via dovecot  
wrote:
> 
> * Ralf Hildebrandt via dovecot :
> 
>> But why is that? Why would the index file be updated so often?
> 
> BTW: This post is a followup to my "2.3.7 slower than 2.3.6?" post from back 
> in July.

Fixed by 
https://github.com/dovecot/core/commit/5e9e09a041b318025fd52db2df25052b60d0fc98 

 and will be in the soon-to-be-released v2.3.8.

Re: dovecot 2.3.7.2-1~bionic: Performance issues caused by excessive IO to ~/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.tmp

[Timo Sirainen via dovecot]

On 1 Oct 2019, at 16.31, Ralf Hildebrandt via dovecot  
wrote:
> 
> I set up system copying all mails to a backup system.
> 
> This used to work without a hitch - now in the last few days mails
> would pile up in the Postfix Queue, waiting to be delivered using the
> lmtp transport into dovecot.
> 
> So dovecot was being slow, but why? After all, nothing changed.
> 
> After reading some articles on stackoverflow I found a way of finding
> out which file gets the most IO:
> 
> % sysdig -c topfiles_bytes;
> 
> This command quickly pointed to 
> ~/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.tmp
> That file was written excessively. 

Was it one user's dovecot.index.tmp or for a lot of users? This means that 
dovecot.index is being rewritten, which should happen only once in a while, but 
now it sounds like it's happening maybe for every mail delivery. If it's still 
happening, could you send me one folder's dovecot.index and dovecot.index.log 
files? (They don't contain anything sensitive other than maybe message flags.)

> I then put ~/mdbox/mailboxes/INBOX/dbox-Mails/ into tmpfs and alas, the queue 
> would drain quickly.
> 
> But why is that? Why would the index file be updated so often?
> 
> This is dovecot 2.3.7.2-1~bionic

So you had been running this version already for a while, and then it just 
suddenly started getting slow?

I tried to reproduce this with imaptest and Dovecot that is patched to log when 
dovecot.index is being rewritten, but there doesn't seem to be any difference 
with v2.2.36, v2.3.7 or git master.

Re: WARNING: using attachment_dir with plugin zlib can corrupt mails

[Timo Sirainen via dovecot]

On 19 Jul 2019, at 17.52, Patrick Cernko via dovecot  
wrote:
> 
> Hello list, hello Dovecot developers,
> 
> this week, I discovered a serious bug in Dovecot, that lead to several broken 
> mails on our servers. The bug corrupts the first few characters of the mail 
> header during saving. On our setup, it was almost always only the very first 
> line of text, that was corrupted.
> 
> The bug seems to be triggered by a bad "interaction" of attachment_dir option 
> and zlib plugin. If you use both, you most likely are affected, too, except 
> you only use zlib plugin for reading previously compressed stored mails. 
> That's also the workaround we use now: zlib plugin only enabled in 
> mail_plugins but no plugin/zlib_save set.

Actually the mail isn't saved corrupted. The bug is when reading the mail. So 
any existing corrupted mails become fixed after upgrading.

Fix here: 
https://github.com/dovecot/core/commit/5068b11e594ad7cc1f7cedf2bd9280520e0e534d 

Re: WARNING: using attachment_dir with plugin zlib can corrupt mails

[Timo Sirainen via dovecot]

On 19 Jul 2019, at 17.52, Patrick Cernko via dovecot  
wrote:
> 
> Hello list, hello Dovecot developers,
> 
> this week, I discovered a serious bug in Dovecot, that lead to several broken 
> mails on our servers. The bug corrupts the first few characters of the mail 
> header during saving. On our setup, it was almost always only the very first 
> line of text, that was corrupted.
..
> The bug occurs on very specific mails. Due to privacy reasons I could not 
> provide sample mails here. Storing such mails seems to trigger the bug 
> reproducible.
> 
> 
> I attached a very minimal doveconf -n config, that can be used to trigger the 
> bug. If one of the developers is interested, I can try to generate an 
> "anonymized" version of such a specific mail that still causes the issue. I 
> discovered the bug on our productive systems, running latest Dovecot 2.2 
> release, but the latest 2.3 I used during debugging is affected, too.

Getting such a mail that would allow reproducing would be helpful. I can't seem 
to be able to reproduce this with stress testing.

https://dovecot.org/tools/  has a couple of scripts 
that can obfuscate emails in a bit different ways. For example 
https://dovecot.org/tools/maildir-obfuscate.pl 
 might work.

I'm also wondering if Stephan's recent base64 code changes will fix this 
(everything is not merged yet).

Re: Dovecot 2.3.7 - char "-" missing

[Timo Sirainen via dovecot]

On 30 Aug 2019, at 13.44, Domenico Pastore via dovecot  
wrote:
> 
> Hello,
> 
> i have update dovecot from version 2.2.15 to 2.3.7.2.
> I have a problem with mine java software because there is a different 
> response when open connection to doveadm.
> 
> I need open socket to doveadm for get imap quota of a mailbox.
> 
> With version 2.2.15:
> # telnet 192.160.10.4 924
> Trying 192.160.10.4...
> Connected to 192.160.10.4.
> Escape character is '^]'.
> -
> 
> 
> With version 2.3.7.2:
> # telnet 192.160.10.3 924
> Trying 192.160.10.3...
> Connected to 192.160.10.3.
> Escape character is '^]'.
> 
> 
> The difference is "-" character. The version 2.3.7 not respond with "-" 
> character after opening the connection.
> 
> Is it possible to add the character again with a parameter?
> 
> Why did doveadm's answer change?

It got changed as part of some other doveadm protocol changes. The change was 
somewhat accidental though and we didn't notice the difference. Anyway, 
practically this shouldn't have made any difference if the code was implemented 
as was described in https://wiki.dovecot.org/Design/DoveadmProtocol 
 It says that the client needs 
to send VERSION first, and as a reply it receives the "+" or "-" line. So it 
was more of a bug that previous Dovecot versions sent the +/- line too early. I 
added a note about this to the wiki page though.

Re: Feature wishlist: Allow to hide client IP/host in submission service

[Timo Sirainen via dovecot]

On 25 Aug 2019, at 21.51, Sebastian Krause via dovecot  
wrote:
> 
> Hi,
> 
> In many mail setups a required feature (for privacy reasons) is to
> hide the host and IP of clients (in the "Received" header) that use
> the authenticated submission over port 587. In Postfix that's
> possible (https://serverfault.com/q/413533/86332), but not very nice
> to configure especially if you only want want to strip the Received
> header for port 587 submissions, but not on port 25.
> 
> As far as I can see this configuration is not possible at all in the
> Dovecot submission server because the function which adds the
> Received header with the client's IP address
> (smtp_server_transaction_write_trace_record) is always called in
> submission-commands.c.
> 
> It would be very useful if the submission server could anonymize the
> client with a single configuration option, then all the Postfix
> configuration mess (and using SASL) could be skipped by simply using
> the Dovecot submission server instead.

Yeah, it would be useful to hide the client's IP and do it by default. Actually 
I think there shouldn't even be an option to not hide it. Or would it be better 
or worse to just not have the Received header added at all?

Re: Dovecot and hard links?

[Timo Sirainen via dovecot]

On 17 Aug 2019, at 1.57, @lbutlr via dovecot  wrote:
> 
> On 16 Aug 19, at 07:33 , @lbutlr  wrote:
>> I was looking at a mail folder and I noted that a file in the inbox had a 
>> total of 11 hard links to it:
> 
> Ack. I checked the junk folder and there are 379 files in there with 379 
> links!
> 
> Since they were all in jink I just deleted them all, but that cannot possibly 
> be desired behavior.
> 
> What do I check here?

Hard links are created when a mail is copied with the IMAP COPY command. So 
Dovecot just does what the client asks it to do. Maybe you have some 
misbehaving IMAP client?

Re: 2.3.7 + stats

[Timo Sirainen via dovecot]

On 16 Aug 2019, at 14.35, Jean-Daniel via dovecot  wrote:
> 
> Some of the behaviours you observe may be due to the same bug I encountered:
> 
> https://dovecot.org/pipermail/dovecot/2019-July/116475.html
> 
> Especially, regarding the ‘successful' field for auth, which does not exists 
> and is really named ‘success', and which is never set anyway.

As a workaround you could use auth_passdb_request_finished. And since it shows 
differences between "password mismatch" vs "user unknown", that's likely a good 
permanent solution as well.

metric auth_ok {
  event_name = auth_passdb_request_finished
  filter {
result = ok
  }
}
metric auth_user_unknown {
  event_name = auth_passdb_request_finished
  filter {
result = user_unknown
  }
}
metric auth_password_mismatch {
  event_name = auth_passdb_request_finished
  filter {
result = password_mismatch
  }
}

Re: segmentation fault in fs_list_get_path

[Timo Sirainen via dovecot]

On 3 Aug 2019, at 21.22, David M. Johnson via dovecot  
wrote:
> 
> There seems to be a straightforward bug in 
> src/lib-storage/list/mailbox-list-fs.c:79.  set->index_dir is unchecked prior 
> to dereferencing (unlike on line 126 in the same file, where it is properly 
> checked).  This manifested on a FreeBSD server running dovecot 2.3.6 when 
> clients tried to retrieve mail with subscriptions like `~/bar/baz`.  This 
> caused the `imap` child to crash, e.g. (slightly anonymized)

Could you also send your doveconf -n output? Would likely help creating a 
reproducible test.

Re: auth module logging

[Timo Sirainen via dovecot]

On 4 Aug 2019, at 6.23, AP via dovecot  wrote:
> 
> On Sat, Aug 03, 2019 at 11:27:24AM -0600, Michael Slusarz wrote:
>>> Errors hit the logs but I would appreciate seeing successful auths
>>> happen for the additional piece of mind. Cmouse and I couldn't
>>> find a way to do it on irc and it appears that the capability is
>>> missing. Successul /logins/ can be logged but auths, by themselves,
>>> cannot.
>>> 
>>> I would appreciate if the ability was added.
>>> 
>>> Dovecot 2.3.7.1 is in use.
>> 
>> Events (using event exporter) is probably what you want, new in 2.3.7.
>> 
>> https://doc.dovecot.org/admin_manual/list_of_events/
> 
> Hi,
> 
> I've tried using this in various ways but I could never get any real success.
> 
> I came close but the logging was always far too verbose. The info I wanted
> WAS there but so was a ton of other data I didn't want. I'd share the configs
> I tried but they came and went as I was experimenting.
> 
> Would love to know how to configure the events logging such that I only get
> a successful auth line logged as that would, indeed, solve my issue. It's
> quite likely I didn't hit the right config as the docs are somewhat sparse.

There probably isn't yet a name for the event that you want. A kludgy approach 
would be to filter the event based on the source code filename and line number. 
But that likely needs to be modified every time you upgrade Dovecot..

Re: dovecot-uidlist invalid data

[Timo Sirainen via dovecot]

On 4 Aug 2019, at 22.57, Király Balázs via dovecot  wrote:
> 
> Hi!
>  
> I’m struggling with the following error: 
>  
> Aug  4 21:32:00 mx02 dovecot: imap(x...@xxx.tld)<17693>: 
> Error: Mailbox INBOX: Broken file /home/vmail/xxx.tld/xxx/dovecot-uidlist 
> line 6246: Invalid data:
> Aug  4 21:49:22 mx02 dovecot: imap(x...@xxx.tld)<21879>: 
> Error: Mailbox INBOX: Broken file /home/vmail/xxx.tld/xxx/dovecot-uidlist 
> line 6249: Invalid data:
>  
> It seems the first part is not incremented properly and sometimes it has a 
> jump in it, like the line 6246:
>  
> 18810 :1564935891.M816284P8904.mx01.m.ininet.hu,S=12145,W=12409
> 18812 :1564947092.M542714P2651.mx01.m.ininet.hu,S=12275,W=12517

Is there ever anything after the "Invalid data:" text? It seems anyway that 
concurrent reading/writing isn't working as expected in dovecot-uidlist. Most 
likely something to do with NFS.

Can you reproduce this easily by just running "imaptest" with some test account 
(it'll delete mails)? See https://imapwiki.org/ImapTest - it's also available 
in dovecot-imaptest package in repo.dovecot.org.

Re: Emails not visible after renaming folders

[Timo Sirainen via dovecot]

Looks like this happens when you use a combination of FULLDIRNAME and INDEX in 
mail_location. Without one of these, or using DIRNAME instead of FULLDIRNAME it 
works correctly. Tracking internally in DOP-1348.

> On 6 Aug 2019, at 14.22, Aleksandr via dovecot  wrote:
> 
> Hi guys.
> 
> Does anyone have problems with a similar configuration (mdbox)?
> 
> Just tested with latest version (stage servers installation: dovecot 2.3.7), 
> also affected.
> 
> Not critical, but have complaints from users, 1-2 per month.
> 
> 
> 26.06.2019 12:05, Aleksandr пишет:
>> Copying or moving with email client: thunderbird, roundcube (webmail), mutt 
>> or any other email client via imap protocol.
>> 
>> 25.06.2019 22:10, Germán Herrera пишет:
>>> Are you copying/moving the emails with {cp|mv} or with "doveadm 
>>> {copy|move}"?
>>> 
>>> On 2019-06-25 12:00, Aleksandr via dovecot wrote:
 Hello,
 
 I have strange problem with "losing" emails after rename mail
 folder(s) (via imap client: thunderbird, roundcude, etc..)
 
 How to reproduce:
 
 1. Create some folder name, like TEST
 2. Create sub-folder under TEST (like SUBTEST)
 
 Structure:
 
 TEST
   |--SUBTEST
 
 
 # doveadm  mailbox list  -u postmaster@testmailbox
 Spam
 Trash
 Sent
 Drafts
 INBOX
 TEST
 TEST/SUBTEST
 
 3. Move (or copy) mails from INBOX to SUBTEST (all looks fine, and
 mails visible under SUBTEST)
 4. Rename TEST folder to any new name, NEWTEST
 
 Let`s try to view mails in mail client in NEWTEST-SUBTEST, folder have
 no emails :(
 
 
 mailsrv# doveadm -f table mailbox status -u postmaster@testmailbox
 "messages vsize" NEWTEST*
 mailbox  messages vsize
 NEWTEST 00
 NEWTEST/SUBTEST 00
 
 If doveadm force-resync postmaster@testmailbox, mails will be visible in 
 INBOX
 
 mailsrv# doveadm -f table mailbox status -u postmaster@testmailbox
 "messages vsize" INBOX*
 mailbox messages vsize
 INBOX   228
 
 Dovecot installation: CentOS x86_64 Linux 7.5.1804
 
 Storage: HDD Local Partition - XFS filesystem  / multi-dbox (mdbox) as
 mail_storage (this problem is not reproduced with the settings as
 Maildir storage !)
 somthing wrong with mapping indices.
 
 
  [start] 
 
 # dovecot -n
 
 # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf
 # Pigeonhole version 0.4.21 (92477967)
 # OS: Linux 3.10.0-862.2.3.el7.x86_64 x86_64 CentOS Linux release
 7.5.1804 (Core)
 # Hostname: 
 auth_mechanisms = plain login digest-md5 cram-md5
 base_dir = /var/run/dovecot/
 default_client_limit = 2
 default_login_user = dovecot
 default_process_limit = 1
 dict {
   quota = redis:host=127.0.0.1:prefix=user/:timeout_msecs=1000
 }
 disable_plaintext_auth = no
 first_valid_gid = 90
 first_valid_uid = 90
 imapc_features = rfc822.size fetch-headers
 imapc_host = 
 imapc_user = %u
 lda_mailbox_autocreate = yes
 lda_mailbox_autosubscribe = yes
 login_greeting = .
 login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c
 login_trusted_networks = 10.0.1.0/24
 mail_access_groups = mail
 mail_debug = yes
 mail_fsync = never
 mail_gid = 97
 mail_location =
 mdbox:~/mail/mailboxes:FULLDIRNAME=mBoX-MeSsAgEs:INDEX=~/mail/index:CONTROL=~/mail/control:INBOX=~/mail/mailboxes/inbox
 mail_log_prefix = "%{session} %Us(%u): "
 mail_max_lock_timeout = 30 secs
 mail_plugins = quota  zlib
 mail_prefetch_count = 20
 mail_privileged_group = mail
 mail_uid = 97
 managesieve_notify_capability = mailto
 managesieve_sieve_capability = fileinto reject envelope
 encoded-character vacation subaddress comparator-i;ascii-numeric
 relational regex imap4flags copy include variables enotify environment
 mailbox date index ihave duplicate mime foreverypart extracttext
 vacation-seconds editheader
 mbox_lock_timeout = 30 secs
 mbox_very_dirty_syncs = yes
 mbox_write_locks = fcntl
 namespace inbox {
   inbox = yes
   list = yes
   location =
   mailbox Drafts {
 auto = subscribe
 special_use = \Drafts
   }
   mailbox Sent {
 auto = subscribe
 special_use = \Sent
   }
   mailbox Spam {
 auto = subscribe
   }
   mailbox Trash {
 auto = subscribe
 special_use = \Trash
   }
   prefix =
   separator = /
   type = private
 }
 passdb {
   args = /etc/dovecot/dovecot-ldap.conf
   driver = ldap
 }
 plugin {
   cgroup_basedir = /usr/sys/cgroup
   hostingAccount = default
   quota = dict:User quota::proxy::quota
   quota_grace = 0%%
   quota_over_flag_value = TRUE
   quota_over_script = account-quota 

Re: Autoexpunge not working for Junk?

[Timo Sirainen via dovecot]

On 14 Aug 2019, at 22.35, Amir Caspi via dovecot  wrote:
> 
> On Aug 14, 2019, at 1:26 PM, Timo Sirainen via dovecot  <mailto:dovecot@dovecot.org>> wrote:
>> 
>> It probably has something to do with using mbox format. Are the IMAP UIDs 
>> changing unexpectedly? Errors/warnings logged related to it? Unfortunately 
>> it's a rather troublesome mailbox format. There are likely some bugs in 
>> Dovecot mbox code, but it's difficult and time consuming to try to reproduce 
>> any of the bugs so I've mostly given up trying.
> 
> I'm not getting any errors or warnings as far as I can tell, and I don't 
> think the UIDs are changing unexpectedly -- messages are not getting 
> re-downloaded randomly.  That is, everything SEEMS to be working fine, as far 
> as I can tell.
> 
> So many people still use mbox that I hope we can fix this issue.
> 
> I'm happy to help test or provide further debug output... this problem is 
> certainly reproducible here, and it seems like lbutlr has a similar problem, 
> so hopefully we can address at least this one...
> 
> (I'm also happy to give you the Junk mailbox and index files... there's 
> nothing sensitive in my spam!)

It's not very helpful to look at the indexes after the problem already 
happened. But if you can find a reliably reproducible way to make this happen 
starting from an empty mailbox, I could look into it further. Ideally it would 
be a standalone script that reproduces the problem every time. Possibly 
something like:

 * Deliver mails with procmail
 * Read the mails with doveadm fetch
 * Maybe expunge the mails with doveadm expunge
 * Keep checking the uid and date.saved with doveadm fetch to see if they 
unexpectedly change at some point

Re: Auth driver

[Timo Sirainen via dovecot]

On 9 Aug 2019, at 15.08, Riccardo Paolo Bestetti via dovecot 
 wrote:
> 
> 
> Could you point me to any documentation or examples? While I can find many 
> plugins in the repo and around the Internet, I could find none which add 
> authdb drivers.

https://dovecot.org/patches/2.2/passdb-openam.c 
 although it's not using 
autotools to do it in a nice way. For the autotools stuff you can use some 
other plugins as example. Pigeonhole for example, although it's much more 
complicated than most. https://dovecot.org/patches/2.2/mail-filter.tar.gz 
 probably has also (I 
didn't look into it).

Re: Should dovecot not be using different logging facility and severity levels?

[Timo Sirainen via dovecot]

On 9 Aug 2019, at 17.39, Marc Roos via dovecot  wrote:
> 
> Should dovecot not be using different severity levels like auth.warn? On 
> my system everything goes to loglevel info:

My thinking has been:

 * Panic: There's a bug that needs fixing
 * Fatal: Somewhat stronger error
 * Error: Something's broken or misconfigured - admin should fix something
 * Warning: Something seems to be at least temporarily broken, like maybe some 
limit was reached because the system was overloaded. Admin may need to do 
something or possibly just wait. Either way, these should be looked into.
 * Info: Events that admin doesn't necessarily need to look at, except while 
debugging or for gathering stats or something
 * Debug: Only when really debugging

> lev_info:Aug  9 16:18:24 mail03 dovecot: imap-login: Aborted login (auth 
> failed, 1 attempts in 2 secs): user=, method=PLAIN, rip=x.x.x.x, 
> lip=x.x.x.x, TLS, session=
> lev_info:Aug  9 16:18:29 mail03 dovecot: auth-worker(28656): 
> pam(krinfo,188.206.104.240,): unknown user

These are regular events that happen all the time due to brute force attacks 
and such. I don't know why you'd want to see them as warnings?

Re: Autoexpunge not working for Junk?

[Timo Sirainen via dovecot]

On 13 Aug 2019, at 5.57, Amir Caspi via dovecot  wrote:
> 
> On Aug 12, 2019, at 8:54 PM, Thomas Zajic via dovecot  
> wrote:
>> 
>> * Amir Caspi via dovecot, 12.08.19 22:01
>> 
>>> [~]# doveadm mailbox status -u cepheid firstsaved Junk
>>> Junk firstsaved=1563154976
>>> 
>>> I can't tell how that timestamp corresponds to a human-readable date, 
>>> however.
>> 
>> [zlatko@disclosure:~]$ date -d @1563154976
>> Mon Jul 15 03:42:56 CEST 2019
> 
> So this is the same timestamp as date.saved on message 1... as it should be, 
> I guess.  Except that, as I showed, the timestamps are definitely messed up 
> somehow.  The timestamps in my MUA (whether webmail or local mail app) show 
> just fine... so something seems to be corrupted with the timestamps in the 
> dovecot index file, I think.  But the weird thing is that this is affecting 
> all users, not just me.

It probably has something to do with using mbox format. Are the IMAP UIDs 
changing unexpectedly? Errors/warnings logged related to it? Unfortunately it's 
a rather troublesome mailbox format. There are likely some bugs in Dovecot mbox 
code, but it's difficult and time consuming to try to reproduce any of the bugs 
so I've mostly given up trying.

Re: [BUG?] Double quota calulation when special folder is present

[Timo Sirainen via dovecot]

> On 6 Aug 2019, at 21.08, Mark Moseley via dovecot  wrote:
> 
>> 
>> I've bisected this down to this commit: 
>> 
>> git diff 
>> 7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa
>> 
>> Prior to this commit, anything updating the quota would do the right thing 
>> for any .INBOX. folders (i.e. not double count the contents of 
>> "INBOX" against the quota). After this commit, anything updating quota (new 
>> mail, quota recalc, etc) does the double counting of INBOX.
> 
> Thank you for the bisect! We'll look into this.
> 
> Hi. I was curious if there were any fixes for this? We're still affected by 
> this (and I imagine others are too but don't realize it). Thanks!

Looks like this happens only with Maildir++ quota. As a workaround you could 
switch to dict-file or "count" quota. Anyway added to internal tracking as 
DOP-1336.

Re: Dovecot Director upgrade from 2.2 to 2.3

[Timo Sirainen via dovecot]

On 31 Jul 2019, at 15.41, Alessio Cecchi via dovecot  
wrote:
> 
> Hi Timo,
> 
> here you can see two images with the load average and CPU usage with Dovecot 
> 2.2 (Centos 6) and 2.3 (Centos 7) on the same hardware and same configuration:
> 
> https://imgur.com/a/1hsItlc 
> Load average increment is relevant but CPU usage is similar.
> 

Load average changes can be rather difficult to debug if it's not because of 
any CPU usages or disk IO, and in your graphs those hadn't really changed. One 
more possibility could be to look at context switches, but do you have those 
from before the upgrade? (sar database?) Also since you changed CentOS 6 -> 7 
it could be possible that it has nothing to do with Dovecot at all.

Re: IMAP frontend authenticating proxy with GSSAPI/Kerberos SSO

[Timo Sirainen via dovecot]

On 1 Aug 2019, at 12.26, Gert van Dijk via dovecot  wrote:
> 
> passdb {
>  args = proxy=y host=127.0.0.1 port=1143 pass=#hidden_use-P_to_show#
..
> auth: Info: static(username,1.2.3.4,<9WOjSwWP8toKAAYE>): No password
> returned (and no nopassword)

I think this is why it's not using the passdb at all. Try adding 
password=something to the args.

Re: doveadm: Error: open(/proc/self/io) failed

[Timo Sirainen via dovecot]

On 31 Jul 2019, at 20.45, A. Schulze via dovecot  wrote:
> 
> 
> 
> Am 31.07.19 um 08:27 schrieb Sami Ketola via dovecot:
>> service lmtp {
>> user = vmail
>> }
>> 
>> please remove user = vmail from here or change it to root.
>> 
>> for security reasons lmtp service must be started as root since version 
>> 2.2.36. lmtp will drop root privileges after initialisation but it needs to 
>> open /self/proc/io as root before that.
> 
> Hello Sami,
> 
> I don't read "root is required for lmtp" in 
> https://wiki.dovecot.org/LMTP#Security neither does 
> https://dovecot.org/doc/NEWS-2.2 say so.
> Could you proof that statement somehow?


Alternative is:

service lmtp {
  user = vmail
  drop_priv_before_exec = yes
}

I'm not sure if you run into other problems with that.

Re: Autoexpunge not working for Junk?

[Timo Sirainen via dovecot]

On 25 Jul 2019, at 7.18, Amir Caspi via dovecot  wrote:
> 
> Hi all,
> 
>   I set up dovecot a couple of months ago and am having trouble getting 
> autoexpunge=30d to work on my Trash and Junk mailboxes.  Not sure why not 
> because I'm not getting error messages in my log.
>   Running "doveadm search -u  mailbox Junk savedbefore 30d" shows 
> me many messages (I've got messages back to mid-May, and a couple of other 
> users have them back to early April, although if this setting were working, 
> there should be nothing earlier than June 24).  Running a manual doveadm 
> expunge works fine... it's just autoexpunge that seems to not be running at 
> all.

Autoexpunging tries to be efficient, so it looks only at the first email's 
saved-timestamp. It's also cached in dovecot.list.index. So you should check:

1. What's the first mail's saved-timestamp?
doveadm fetch -u user date.saved mailbox Junk 1

2. That timestamp should also be the same in dovecot.list.index:
doveadm mailbox status -u user firstsaved Junk

Re: corrupt mdbox index / zero mails showing in imap

[Timo Sirainen via dovecot]

On 27 Jul 2019, at 14.13, Timo Sirainen  wrote:
> 
> On 25 Jul 2019, at 20.55, Mike via dovecot  wrote:
>> 
>> Hi,
>> 
>> 
>> I have recently migrated (under emergency conditions) a dovecot imap/pop
>> based server to a new instance. The mailboxes used mdbox format and due
>> to various screwups I had corrupt indexes. I thought I'd cleaned this up
>> but then I found that this new instance hadn't been set up correctly for
>> nfs. Long story short, I still get users with new cases of corrupt
>> indexes. The symptom is imap either showing NO mail in their inbox, or,
>> not showing any recently delivered mail in the box, until I rm -f
>> dovecot.map.index / doveadm force-resync -u user.
>> 
>> It would be a huge help if there could be some method to detect if this
>> is the case for any given user and to proactively do the force-resync
>> process for them instead of waiting for that support call (or service
>> cancellation...). I have looked around and have not found any tool
>> capable of 'linting' an mdbox format inbox, and it seems like something
>> like this should have been or would be an extremely useful debugging
>> tool both during development as well as to troubleshoot stuff in the
>> field. I would love to know if anyone either has such a tool, or any
>> general suggestion how I could go about finding these cases and
>> addressing them. I believe I have the nfs issue resolved and will not be
>> creating new cases, so I just want to fix the ~3000 boxes I have now and
>> move forward.
> 
> 
> I think you could do something with using "doveadm dump" command. I'm not 
> sure in what ways your mdboxes are corrupted, so there might be an easier way 
> to do it, but for a generic checker I think this would work:
> 
> * doveadm dump dovecot.map.index and remember all the "uid" numbers
> * doveadm dump dovecot.index for each folder and remember all the "map_uid" 
> numbers.
> * See if any map_uid is missing in dovecot.map.index's uid numbers. yes -> 
> force-resync
> 
> You can also use doveadm dump for the storage/m.* files to see what they 
> contain, but this likely won't be useful for this case.

Or actually reading further: It looks like all your indexes are gone, even the 
folders' dovecot.index* files? Wouldn't a simple solution then be just to check 
if "doveadm mailbox status" if the user looks empty (or mostly empty) you'd run 
the force-resync? Or another thought: If INBOX's dovecot.index was completely 
lost and was recreated, you can see the index's creation timestamp in the 
"indexid" field in "doveadm dump dovecot.index" - if it's new enough do the 
force-resync.

BTW. If all the indexes are gone, force-resync adds all storage/m.* mails back 
to their original folder. So if user had moved mails to another folder they 
come back to e.g. INBOX. It also loses message flags, and brings back mails 
that were already expunged but not yet doveadm purged.

Re: corrupt mdbox index / zero mails showing in imap

[Timo Sirainen via dovecot]

On 25 Jul 2019, at 20.55, Mike via dovecot  wrote:
> 
> Hi,
> 
> 
> I have recently migrated (under emergency conditions) a dovecot imap/pop
> based server to a new instance. The mailboxes used mdbox format and due
> to various screwups I had corrupt indexes. I thought I'd cleaned this up
> but then I found that this new instance hadn't been set up correctly for
> nfs. Long story short, I still get users with new cases of corrupt
> indexes. The symptom is imap either showing NO mail in their inbox, or,
> not showing any recently delivered mail in the box, until I rm -f
> dovecot.map.index / doveadm force-resync -u user.
> 
> It would be a huge help if there could be some method to detect if this
> is the case for any given user and to proactively do the force-resync
> process for them instead of waiting for that support call (or service
> cancellation...). I have looked around and have not found any tool
> capable of 'linting' an mdbox format inbox, and it seems like something
> like this should have been or would be an extremely useful debugging
> tool both during development as well as to troubleshoot stuff in the
> field. I would love to know if anyone either has such a tool, or any
> general suggestion how I could go about finding these cases and
> addressing them. I believe I have the nfs issue resolved and will not be
> creating new cases, so I just want to fix the ~3000 boxes I have now and
> move forward.


I think you could do something with using "doveadm dump" command. I'm not sure 
in what ways your mdboxes are corrupted, so there might be an easier way to do 
it, but for a generic checker I think this would work:

 * doveadm dump dovecot.map.index and remember all the "uid" numbers
 * doveadm dump dovecot.index for each folder and remember all the "map_uid" 
numbers.
 * See if any map_uid is missing in dovecot.map.index's uid numbers. yes -> 
force-resync

You can also use doveadm dump for the storage/m.* files to see what they 
contain, but this likely won't be useful for this case.

[Dovecot-news] Dovecot v2.3.7.1 & Pigeonhole v0.5.7.1 released

[Timo Sirainen via Dovecot-news]

https://dovecot.org/releases/2.3/dovecot-2.3.7.1.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.7.1.tar.gz.sig

https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.7.1.tar.gz
https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.7.1.tar.gz.sig

Binary packages in https://repo.dovecot.org/

These releases fix the reported regressions in v2.3.7 & v0.5.7.

Dovecot core:
- Fix TCP_NODELAY errors being logged on non-Linux OSes
- lmtp proxy: Fix assert-crash when client uses BODY=8BITMIME
- Remove wrongly added checks in namespace prefix checking

Pigeonhole:
- dsync: Sieve script syncing failed if mailbox attributes weren't
  enabled.

___
Dovecot-news mailing list
Dovecot-news@dovecot.org
https://dovecot.org/mailman/listinfo/dovecot-news

Dovecot v2.3.7.1 & Pigeonhole v0.5.7.1 released

[Timo Sirainen via dovecot]

https://dovecot.org/releases/2.3/dovecot-2.3.7.1.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.7.1.tar.gz.sig

https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.7.1.tar.gz
https://pigeonhole.dovecot.org/releases/2.3/dovecot-2.3-pigeonhole-0.5.7.1.tar.gz.sig

Binary packages in https://repo.dovecot.org/

These releases fix the reported regressions in v2.3.7 & v0.5.7.

Dovecot core:
- Fix TCP_NODELAY errors being logged on non-Linux OSes
- lmtp proxy: Fix assert-crash when client uses BODY=8BITMIME
- Remove wrongly added checks in namespace prefix checking

Pigeonhole:
- dsync: Sieve script syncing failed if mailbox attributes weren't
  enabled.

Re: Dovecot Director upgrade from 2.2 to 2.3

[Timo Sirainen via dovecot]

On 22 Jul 2019, at 17.45, Alessio Cecchi  wrote:
> 
> one server of the ring is now running Dovecot 2.3.7 and works fine with the 
> others Dovecot 2.2 since 3 days.
> I notice only that the load avarage of this CentOS 7 server is higher 
> compared with CentOS 6 and Dovecot 2.2, but I don't know if is related to the 
> new operating system or Dovecot (hardware is the same).
> 
How much higher? Can you check the individual dovecot processes' CPU usage? I 
guess mainly director, imap-login and pop3-login. The director code should be 
pretty much the same though.

The SSL code in login processes changed in v2.3, so I wonder if the new code 
has some performance issues.

Re: Character not allowed in mailbox name

[Timo Sirainen via dovecot]

On 22 Jul 2019, at 10.34, Lothar Schilling via dovecot  
wrote:
> 
> Hi Arnold,
> 
> thanks for your assistance. I solved the issue right now by changing
> 
> prefix = Shared/%%u/
> to
> prefix = Shared/%%n/
> 
> This omits the @mydomain.de part which holds the dot causing the
> trouble. Why this became a problem due to the update I wouldn't know.

Accidental bug. Fixed by 
https://github.com/dovecot/core/commit/62f3b738efd4a6444a4bde4a80bea208c5b39ccd 

> 
> Regards
> 
> Lothar
> 
> Am 22.07.2019 um 10:49 schrieb Arnold Opio Oree:
>> Hi Lothar,
>> 
>> I haven't yet had the opportunity to go deep into Dovecot engineering /
>> development, (so please correct me if necessary anyone).
>> 
>> I have experienced this exception where attempting to copy mailbox data
>> within a client from IMAP account to IMAP account, but I have not had
>> it cause a mailbox to become inaccessible.
>> 
>> The reason was because Maildir++ format uses the "." separator to
>> delineate child directories, and therefore the character is illegal in
>> mailbox names. The issue was addressed by either changing the
>> filesystem layout to LAYOUT=fs or by changing the mailbox format
>> altogether to e.g. sdbox.
>> 
>>>  location = maildir:%%h:INDEX=%h/shared/%%u
>> I can see from your configuration that the LAYOUT is not set as fs, so
>> I would assume is still using Maildir++.
>> 
>> Perhaps this setting was lost during the upgrade?
>> 
>> Regards,
>> 
>> Arnold Opio Oree
>> Chief Executive Officer
>> Parallax Digital Technologies
>> 
>> arnoldo...@parallaxdt.com
>> 
>> http://www.parallaxdt.com
>> 
>> tel : +44 (0) 333 577 8587
>> fax : +44 (0) 20 8711 2477
>> 
>> Parallax Digital Technologies is a trading name of Parallax Global
>> Limited. U.K. Co. No. 08836288
>> 
>> The contents of this e-mail are confidential. If you are not the
>> intended recipient you are to delete this e-mail immediately, disregard
>> its contents and disclose them to no other persons.
>> 
>> -Original Message-
>> From: Lothar Schilling via dovecot 
>> Reply-To: Lothar Schilling 
>> To: dovecot@dovecot.org
>> Subject: Character not allowed in mailbox name
>> Date: Mon, 22 Jul 2019 08:33:45 +0200
>> 
>> Hi everybody,
>> 
>> after an update this morning to dovecot 2.3.7 we cannot connect to our
>> shared mailboxes anymore. The error message as issued in Thunderbird:
>> 
>> "Character not allowed in mailbox name '.'"
>> 
>> I didn't change anything about the configuration. Output of dovecot -n:
>> 
>> [...]
>> namespace {
>>  list = children
>>  location = maildir:%%h:INDEX=%h/shared/%%u
>>  prefix = Shared/%%u/
>>  separator = /
>>  subscriptions = yes
>>  type = shared
>> }
>> [...]
>> 
>> Any help would be appreciated, thank you very much.
>> 
>> Lothar Schilling
>> 
>> 
>> 
>> 
>> 
> 

Re: index worker 2.3.7 undefined symbol errors

[Timo Sirainen via dovecot]

On 19 Jul 2019, at 13.20, Dirk Koopman via dovecot  wrote:
> But I am left with this:
> 
> Jul 19 14:09:52 localhost dovecot: indexer-worker: Error: User  
> lookup failed: Couldn't load required plugin 
> /usr/lib/dovecot/modules/lib90_sieve_plugin.so: dlopen() faile: 
> /usr/lib/dovecot/modules/lib90_sieve_plugin.so: undefined symbol: 
> mail_deliver_ctx_get_log_var_expand_table

Oh.. it's logged by indexer-worker.

> mail_plugins = mail_log notify replication fts fts_lucene sieve

You can't load sieve globally. It needs to be inside protocol lmtp {}

Re: Dovecot 2.3.6 on Solaris10: build issues, segfaults

[Timo Sirainen via dovecot]

Ah, okay, I see. submission_max_mail_size should be defined as uoff_t instead 
of size_t in struct submission_settings and struct submission_settings.

> On 20 Jul 2019, at 1.47, Joseph Tam via dovecot  wrote:
> 
> 
> Looking further into this segfault at
> 
>   settings-parser.c:setting_copy():1519
>   *dest_size = *src_size;
> 
> *src_size points to type size_t (typedef unsigned long), a 4-byte
> aligned value consistent with a 32-bit build.  This is mismatched with
> declared type
> 
>   (gdb) whatis src_size
>   type = const uoff_t *
>   (gdb) whatis uoff_t
>   type = unsigned long long
>   (gdb) p sizeof(uoff_t)
>   $1 = 8
> 
> resulting in the segfault when *src_size is dereferened.  The implied
> condition of this code segment is typeof(uoff_t)==typeof(size_t) which
> is clearly not the case.
> 
> I'm not sure how/if uoff_t is defined, but configure reports
> 
>   checking for uoff_t... no
>   checking type of off_t... long long
> 
> The latter is weird, because if I compile and run using the same compiler 
> flags
> 
>   #include 
>   int main(void) { printf("%d %d\n",sizeof(long long),sizeof(off_t)); }
> 
> the output is "8 4".
> 
> Joseph Tam 

Re: Dovecot with MySQL over SSL.

[Timo Sirainen via dovecot]

On 20 Jul 2019, at 23.02, Reio Remma via dovecot  wrote:
> 
> On 20.07.2019 22:37, Aki Tuomi via dovecot wrote:
>> 
>>> On 20/07/2019 21:07 Reio Remma via dovecot  
>>>  wrote:
>>> 
>>> 
>>> On 20.07.2019 18:03, Aki Tuomi via dovecot wrote: 
 
> On 20/07/2019 13:12 Reio Remma via dovecot < dovecot@dovecot.org 
> > wrote:
> 
> 
> On 19.07.2019 0:24, Reio Remma via dovecot wrote:
>> I'm attempting to get Dovecot working with MySQL user database on
>> another machine. I can connect to the MySQL (5.7.26) instance with SSL
>> enabled:
>> mysql -h db.mrst.ee --ssl-ca=/etc/dovecot/ca.pem
>> --ssl-cert=/etc/dovecot/client-cert.pem
>> --ssl-key=/etc/dovecot/client-key.pem --ssl-cipher=DHE-RSA-AES256-SHA
>> -u vmail -p
>> However if I use the same values in dovecot-sql.conf.ext, I get the
>> following error:
>> Jul 19 00:20:18 turin dovecot: auth-worker(82996): Error:
>> mysql(db.mrst.ee): Connect failed to database (vmail): SSL connection
>> error: protocol version mismatch - waiting for 1 seconds before retry
>> Jul 19 00:20:19 turin dovecot: auth-worker(82996): Error:
>> mysql(db.mrst.ee): Connect failed to database (vmail): Connections
>> using insecure transport are prohibited while
>> --require_secure_transport=ON. - waiting for 5 seconds before retry
>> Database connection string:
>> connect = host=db.mrst.ee dbname=vmail user=vmail password=stuff \
>> ssl_ca=/etc/dovecot/ca.pem \
>> ssl_cert=/etc/dovecot/client-cert.pem \
>> ssl_key=/etc/dovecot/client-key.pem \
>> ssl_cipher=DHE-RSA-AES256-SHA
> Update: I got it to connect successfully now after downgrading the MySQL
> server tls-version from TLSv1.1 to TLSv1.
> 
> Is there a reason why Dovecot MySQL doesn't support TLSv1.1?
> 
> Thanks!
> Reio
 
 Dovecot mysql uses libmysqlclient. We do not enforce any particular tls 
 protocol version. If it requires you to downgrade I suggest you review 
 your client my.cnf for any restrictions.
 ---
 Aki Tuomi
>>> 
>>> Thanks Aki! I'm looking at it now and despite identical MySQL 5.7.26 
>>> versions on both systems, it seems Dovecot is using libmysqlclient 5.6.37. 
>>> 
>>> Dovecot seems to be using the older libmysqlclient.so.18.1.0 (5.6.37) from 
>>> mysql-community-libs-compat 5.7.26 instead of the newer 
>>> libmysqlclient.so.20.3.13 (5.7.26) from mysql-community-libs 5.7.26. 
>>> 
>>> If I try to remove the libs-compat, yum also insists on removing 
>>> dovecot-mysql, so it depends on the older libmysqlclient and ignores the 
>>> newer one. 
>>> 
>>> I don't suspect I can do anything on my end to force the Dovecot CentOS 
>>> package to use the non-compat libmysqlclient? 
>>> 
>>> Thanks, 
>>> Reio
>> 
>> What repo are you using?
>> ---
>> Aki Tuomi
> 
> Installed Packages
> dovecot-mysql.x86_64  
>   2:2.3.7-8   
>   
> @dovecot-2.3-latest
> mysql-community-libs.x86_64   
>   5.7.26-1.el7
>   
> @mysql57-community
> 
> Both are from official repos.

dovecot-mysql package is built against the mariadb library that comes with 
CentOS 7. If you want it to work against other libmysqlclient versions you'd 
need to compile it yourself: 
https://repo.dovecot.org/ce-2.3.7/centos/7/SRPMS/2.3.7-8_ce/ 

Re: index worker 2.3.7 undefined symbol errors (more info)

[Timo Sirainen via dovecot]

On 21 Jul 2019, at 23.14, Dirk Koopman via dovecot  wrote:
> 
> Some supplemental information:
> 
> This is happening on every email delivered into Dovecot via LMTP. The curious 
> things are that the message is a) successfully delivered and b) sieved into 
> the correct directory. 
> 
> Another observation is that:
> 
> mail_deliver_ctx_get_log_var_expand_table
> 
> is defined globally in core/src/lib-lda/mail-deliver.c (and used there) but 
> the ONLY external call in the entire dovecot tree is from 
> pigeonhole/src/plugins/lda-sieve/lda-sieve-log.c. 
> 
> I am not using lda but it seems to be part of core. So, as I am only using 
> lmtp, why is pigeonhole using lda-sieve at all? 

It's part of lib-lda, which is also used by lmtp.

> Can I work around the error message by some config magic (as I did by calling 
> the correct plugin for imap_sieve) or is this an actual bug? Could this be 
> fixed simply by including mail-deliver.h in lda-sieve-log.c? 

I think you're not linking lmtp binary correctly somehow. That symbol should be 
part of it:

% nm /usr/libexec/dovecot/lmtp | grep mail_deliver_ctx_get_log_var_expand_table
00061960 T mail_deliver_ctx_get_log_var_expand_table

Re: Dovecot Director upgrade from 2.2 to 2.3

[Timo Sirainen via dovecot]

On 18 Jul 2019, at 11.44, Alessio Cecchi via dovecot  
wrote:
> 
> Hi,
> 
> I have a setup with 3 Dovecot Director v2.2.36 and 
> director_consistent_hashing = yes ;-)
> 
> Now I would like to upgrade to 2.3.7, first only Director and after also 
> Backend.
> 
> Can works fine a ring of director with mixed 2.2 and 2.3 version?
> 
> Mi idea is to setup a new Director server with 2.3, stop one server with 2.2 
> and insert the new 2.3 in the current ring to check if works fine.
> 
> If all works fine I will replace all Director 2.2 with 2.3 version.
> 

There's no known reason why it wouldn't work. But be prepared in case there is 
an unknown reason.

Re: auth: Warning: Event 0x1280fe20 leaked

[Timo Sirainen via dovecot]

On 17 Jul 2019, at 22.13, Jos Chrispijn via dovecot  wrote:
> 
> On 16-7-19 9:30, John Doe Vecot via dovecot wrote:
>> Lately I get some errors in my logfile, saying:
>> 
>> dovecot[72337]: auth: Warning: Event 0x1280fe20 leaked (parent=0x0): 
>> auth-client-connection.c:338: 1 Time(s)
> Same overhere; can someone pls explain?
> 

What's your doveconf -n? I suppose this is coming either when 
stopping/reloading, or maybe auth process stops when it's been idling for 1 
minute. Do you use Postfix/Exim (or something else) that authenticates via 
Dovecot?

Re: Dovecot v2.3.7 - TLS/SSL issue

[Timo Sirainen via dovecot]

On 17 Jul 2019, at 13.59, Christos Chatzaras via dovecot  
wrote:
> 
> I use a helpdesk system that connects to dovecot using POP3 with SSL enabled 
> to fetch the e-mails.
> 
> After I upgrade to v2.3.7 the helpdesk randomly (some times it works, some 
> times not) doesn't fetch the e-mails. If I configure the e-mail accounts with 
> SSL/TLS disabled then it works.
> 
> Any idea about this?

What was the previous version you were using? Can you send log files about the 
time when it was broken? (You can send them to me privately.)

Re: [ext] 2.3.7 slower than 2.3.6?

[Timo Sirainen via dovecot]

On 16 Jul 2019, at 11.15, Ralf Hildebrandt via dovecot  
wrote:
> 
> * Timo Sirainen via dovecot :
> 
>>> And alas, I had a look at the monitoring and found that disk IO has
>>> increase A LOT after the update: 
>>> https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
>>> 
>>> I zoomed in and found that the sudden increace in IO coincides with
>>> the update to 2.3.7 (14:35): 
>>> https://www.arschkrebs.de/images/dovecot-io-2.png 
>>> <https://www.arschkrebs.de/images/dovecot-io-2.png>
>> 
> 
>> Do the different disks have different kinds of data? 
> 
> All of the data is an /dev/sda1

What filesystem is this?

I did a bunch of testing, and after initially thinking I saw an increase it was 
just due to bad testing because the new TCP_NODELAY changes allowed imaptest to 
do more work. So I can't see that there is any disk IO difference.

There are a few intentional changes to lib-index and also mdbox code. One of 
those is that dovecot.index.cache files are being recreated more often now. In 
some cases they were wasting a lot of disk space because expunged mails hadn't 
been removed from them. I guess it's a possibility that all the disk IO was 
because in your system Dovecot started recreating all those files at the same 
time. Maybe you could try upgrading again during slower hours and see if the 
disk IO normalizes after a while?

Re: Unexpected result from LIST EXTENDED command

[Timo Sirainen via dovecot]

On 16 Jul 2019, at 11.41, Emil Kalchev  wrote:
> 
> There is no error in the server logs. I checked those particular folders on 
> the server and they don’t seems to have anything special about them, like 
> permission or etc.
>  
> Yes, calling STATUS on those particular folders returns the status. The 
> folders can be opened and they have emails in them so nothing special about 
> those folders.

https://github.com/dovecot/core/blob/master/src/imap/cmd-list.c#L195 
 seems to 
have a bug. If LIST is requesting SUBSCRIBED results, and it finds there is a 
folder that is not subscribed but has a child that is subscribed, then the 
parent isn't requested for STATUS. That matches:

S: * LIST (\HasChildren \UnMarked) "." INBOX.spam.&-BD0EOQQ9BDkEPQ-
S: * LIST (\Subscribed \HasChildren \UnMarked) "." 
INBOX.spam.&-BD0EOQQ9BDkEPQ-.jhfhg

But in your LIST output INBOX.Archive didn't have any children, so I'm not sure 
if that's the same issue or not.

Re: Unexpected result from LIST EXTENDED command

[Timo Sirainen via dovecot]

On 16 Jul 2019, at 9.51, Emil Kalchev via dovecot  wrote:
> 
> I am executing this command below to dovecot-2.3.5-6.cp1178.x86_64 server
>  
> Notice that some status responses are missing (For folders INBOX.Archive, 
> INBOX.spam.&-BD0EOQQ9BDkEPQ-). I wonder If this is a bug or working as 
> expected
>  
> In rfc5819 there is this:
>  
> If the server runs into unexpected problems while trying to look up
> the STATUS information, it MAY drop the corresponding STATUS reply.
> In such a situation, the LIST command would still return a tagged OK
> reply.
>  
> May be that is the reason for this response? Is it possible to find more 
> details in server logs why STATUS is missing?

Do you see any errors logged? Does it work if you ask with STATUS command 
directly those folders? What's your doveconf -n?

Re: Dovecot release v2.3.7

[Timo Sirainen via dovecot]

On 13 Jul 2019, at 14.44, Tom Sommer via dovecot  wrote:
> 
> LMTP is broken on director:
> 
> Jul 13 13:42:41 lmtp(34824): Panic: file smtp-params.c: line 685 
> (smtp_params_mail_add_body_to_event): assertion failed: ((caps & 
> SMTP_CAPABILITY_8BITMIME) != 0)

Thanks, fixed: 
https://github.com/dovecot/core/commit/c4de81077c11d09eddf6a5c93676ee82350343a6 

Re: Dovecot release v2.3.7

[Timo Sirainen via dovecot]

On 13 Jul 2019, at 18.39, Michael Grimm via dovecot  wrote:
> 
> Now replication is working from my point of view, besides occational error 
> messages like:
> 
> | imap-login: Error: file_ostream.net_set_tcp_nodelay(, TRUE) failed: 
> Connection reset by peer

Here's the final fix for it:

https://github.com/dovecot/core/commit/25028730cd1b76e373ff989625132d526eea2504 

Re: [ext] 2.3.7 slower than 2.3.6?

[Timo Sirainen via dovecot]

On 15 Jul 2019, at 23.17, Ralf Hildebrandt via dovecot  
wrote:
> 
> * Ralf Hildebrandt via dovecot :
>> We're using dovecot (via LMTP) as a backup for all incoming mail.
>> 
>> I upgraded from 2.3.6 to 2.3.7 on the 12th:
>> 2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 
>> 2:2.3.7-8~bionic
>> 
>> and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup
>> IMAP box is suddenly taking quite some time to get delivered and is piling up
>> in the queue.
> 
> And alas, I had a look at the monitoring and found that disk IO has
> increase A LOT after the update: 
> https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
> 
> I zoomed in and found that the sudden increace in IO coincides with
> the update to 2.3.7 (14:35): 
> https://www.arschkrebs.de/images/dovecot-io-2.png 
> 

Do the different disks have different kinds of data? Like you seem to be using 
external attachments and fts? Also your doveconf -n doesn't show what fts 
backend you're using. Or is it just half-configured and not really used?

Re: Error since Dovecot v2.3.7

[Timo Sirainen via dovecot]

On 15 Jul 2019, at 10.58, Paul Hecker via dovecot  wrote:
> 
> Hi,
> 
> since upgrading to Dovecot 2.3.7 I get the following new errors:

What was the old version? Did you upgrade your kernel as well?

> 2019-07-15 09:10:52 mail dovecot:  
> imap(p...@iwascoding.com)<32484>: Error: file_lock_free(): 
> Unexpectedly failed to retry locking 
> /var/spool/mail/iwascoding/paul/mdbox/mailboxes/INBOX/dbox-Mails/dovecot-vsize.lock:
>  
> fcntl(/var/spool/mail/iwascoding/paul/mdbox/mailboxes/INBOX/dbox-Mails/dovecot-vsize.lock,
>  write-lock, F_SETLK) locking failed: No such file or directory

What filesystem are you using for mdboxes?

The lock fd is already open. Locking is not supposed to fail this way. Also I 
can't think of any recent changes that could be related to this. v2.2.36 
already had the same locking code.

Re: Purpose of stats-writer and why doveadm try to open it to dump stats ?

[Timo Sirainen via dovecot]

On 14 Jul 2019, at 10.10, Jean-Daniel via dovecot  wrote:
> 
> Hello,
> 
> I want to monitor dovecot stats, and so I have an exporter process that run 
> with limited rights. 
> The monitoring user has only access to /var/run/dovecot/stats-reader and it 
> works fine.
> Doveadm stats dump returns the list of all stats as expected.
> 
> But each time I run doveadm stats dump, it logs the following error:
> 
> Error: net_connect_unix(/var/run/dovecot/stats-writer) failed: Permission 
> denied
> 
> So what is the purpose of the stats-writer socket, and why doveadm try to 
> open it to simply dump stats ? 
> Is it really something it needs and I should update my user permissions or is 
> it a doveadm bug ?

All Dovecot processes nowadays connect to the stats-writer process early on 
before they drop privileges, unless it's explicitly disabled in the code. In 
doveadm case I suppose most commands would want to connect to stats-writer, but 
we could have a per-command flag to specify that the command doesn't want 
stats. I'll add this to our internal Jira.

Re: 2.3.7/FreeBSD: Seeing lots of stats-writer errors.

[Timo Sirainen via dovecot]

On 12 Jul 2019, at 22.10, Larry Rosenman via dovecot  
wrote:
> 
> A user reported this to me, and I'm seeing it too:
> Hi all :)
> 
> After mail/dovecot upgrade to v2.3.7 it seems something went wrong with 
> stat-writer...
> 
> IMAP and POP3 logs:
> 
> Jul 12 19:45:34 mail dovecot: imap-login: Error: 
> file_ostream.net_set_tcp_nodelay((conn:unix:/var/run/dovecot/stats-writer), 
> TRUE) failed: Invalid argument

See the other thread: 
https://dovecot.org/pipermail/dovecot/2019-July/116479.html 

Re: Pigeonhole release v0.5.7

[Timo Sirainen via dovecot]

On 12 Jul 2019, at 21.09, Reio Remma via dovecot  wrote:
> 
>> - dsync: dsync-replication does not synchronize Sieve scripts.
> 
> Sieve replication still doesn't work for me. dsync now replicated sieve and 
> sieve/tmp directories, but neither actual sieve files nor @.dovecot.sieve 
> link.

What if you change the Sieve script? It probably doesn't immediately replicates 
old scripts.

Re: Dovecot release v2.3.7

[Timo Sirainen via dovecot]

On 12 Jul 2019, at 21.05, Michael Grimm via dovecot  wrote:
> 
> Aki Tuomi via dovecot  wrote:
> 
>> We are pleased to release Dovecot release v2.3.7.
> 
> My upgrade from 2.3.6 to 2.3.7 broke replication (FreeBSD 12.0-STABLE 
> r349799):
> 
> | dovecot[76032]: master: Dovecot v2.3.7 (494d20bdc) starting up for imap, 
> lmtp, sieve
> | dovecot[76035]: replicator: Error: 
> file_ostream.net_set_tcp_nodelay((conn:unix:/var/run/dovecot/stats-writer), 
> TRUE) failed: Invalid argument
> | dovecot[76035]: replicator: Error: 
> file_ostream.net_set_tcp_nodelay((conn:unix:auth-userdb), TRUE) failed: 
> Invalid argument
> | dovecot[76035]: auth: Error: 
> file_ostream.net_set_tcp_nodelay((conn:unix:/var/run/dovecot/stats-writer), 
> TRUE) failed: Invalid argument

I don't think these cause any actual breakage? Of course, there are tons of 
errors logged..

This likely fixes it anyway:

diff --git a/src/lib/ostream-file.c b/src/lib/ostream-file.c
index e7e6f62d1..766841f2f 100644
--- a/src/lib/ostream-file.c
+++ b/src/lib/ostream-file.c
@@ -334,7 +334,7 @@ static void o_stream_tcp_flush_via_nodelay(struct 
file_ostream *fstream)
 {
if (net_set_tcp_nodelay(fstream->fd, TRUE) < 0) {
if (errno != ENOTSUP && errno != ENOTSOCK &&
-   errno != ENOPROTOOPT) {
+   errno != ENOPROTOOPT && errno != EINVAL) {
i_error("file_ostream.net_set_tcp_nodelay(%s, TRUE) 
failed: %m",
o_stream_get_name(>ostream.ostream));
}

Re: Frequent Out of Memory for service(config)

[Timo Sirainen via dovecot]

On 13 May 2019, at 22.56, Root Kev via dovecot  wrote:
> 
> May 13 11:35:43 config: Fatal: master: service(config): child 26900 returned 
> error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need 
> to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)


If you set ssl_dh setting this most likely gets fixed.

Re: Error: o_stream_send_istream and Disconnected in APPEND

[Timo Sirainen via dovecot]

On 11 Jul 2019, at 10.13, Alessio Cecchi via dovecot  
wrote:
> 
> Hi,
> 
> I'm running some Dovecot servers configured with LVS + Director + Backend + 
> NFS and version 2.2.36.3 (a7d78f5a2).
> 
> In the last days I see an increased number of these error:
> 
> Error: 
> o_stream_send_istream(/nfs/mail/company.com/info/Maildir/.Sent/tmp/1562771349.M255624P9151.pop01)
>  failed: Broken pipe
> 
> always with action "Disconnected in APPEND", when users try to upload a 
> message in Sent or Drafts.
> 

I think it simply means that Dovecot sees that the client disconnected while it 
was APPENDing the mail. Although I don't know why they would suddenly start 
now. And I especially don't understand why the error is "Broken pipe". Dovecot 
uses it internally when it closes input streams, so it's possibly that, but why 
would isn't that happening elsewhere then.. Did you upgrade your kernel 
recently? I guess it's also possible that there is some bug in Dovecot, but I 
don't remember any changes related to this for a long time.

I guess it could actually be writing as well, because "Broken pipe" is set also 
for closed output streams, so maybe some failed NFS write could cause it 
(although it really should have logged a different error in that case, so if 
that was the reason this is a bug).

Dovecot v2.3.x would log a different error depending on if the problem was 
reading or writing, which would make this clearer.

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Timo Sirainen via dovecot]

On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot  
wrote:
> 
> Dovecot Team,
> 
> I'd like to report a number of bugs, that are to my view all critical.

It would help to get your doveconf -n, example command lines causing the 
problems and the error messages it outputs or what the wrong behavior looks 
like in filesystem. It's now rather difficult to guess what exactly you tried 
and what happened.

Also what kind of output does readpst make? I'm not sure why you're using 
DIRNAMEs here.

> doveadm-sync -1/general
> 
> 1) If DIRNAMEs are not different between command line and mail_location 
> doveadm sync will fail, saying that the source and destination directories 
> are the same

This sounds very strange. I'm not sure what exactly you did, and I couldn't 
reproduce with a small test.

> 2) The -n / -N flags do not work, and a sync will fail strangely if location 
> is specified in the namespace definition

Again, sounds strange.

> 3) Adds mbox to path name under mailbox directory (where syncing from an mbox 
> source)

Probably with different parameters you could avoid it.

> 4) Not having the mailboxes at source named the same as those at destination 
> causes errors and partial sync 
> 
> 5) Not having the target mailboxes formatted to receive the sync 
> (//DIRNAME/) will cause sync errors.

I don't understand these. Target mailboxes are supposed to be empty initially, 
and after the initial sync they should be in the expected format. Why would 
they be different?

> doveadm-sync
> 
> 1) With large synchronizations UIDs are corrupted where multiple syncs are 
> executed and the program can no longer synchronize

What exactly is the error message?

> dovecot
> 
> 1) Panics and fails to expand ~ to user home: observed cases are where 
> multiple namespaces are being used

Panic message and more details would also help.

> With regards to the last error that I requested help on i.e. \Noselect. This 
> has been resolved more-or-less by the workarounds that I have implemented for 
> the bugs reported above.
> 
> I have seen a number of threads whilst researching the \Noselect issue where 
> people have been very confused. My finding was that \Noselect is a function 
> of the IMAP specification server-side implementation RFC3501 ( 
> https://tools.ietf.org/html/rfc3501#section-6.3.
>  
> 6).
>  And for me the server was returning directories with \Noselect because the 
> mailboxes were malformed on account of dovadm-sync errors. In order to fix 
> this I formed a bash command to transverse the mailbox hierarchy and create 
> the missing folders critical to the sdbox format, namely DIRNAME.

Nowadays we have also an option to disable creation of \Noselect folders, 
because they confuse people. Using mail_location = ...:NO-NOSELECT - It won't 
affect existing folders immediately though.

Re: Dovecot 2.3.6 on Solaris10: build issues, segfaults

[Timo Sirainen via dovecot]

On 9 Jul 2019, at 3.02, Joseph Tam via dovecot  wrote:
> 
> Issue 3) dovecot/doveconf segfaults on startup
> 
>   It crashes here while processing dovecot.conf, as does "doveconf"
> 
>   (settings-parser.c:1519 in setting_copy())
>   *dest_size = *src_size;

This is correct code.

>   It appears *src_size is not an 8-byte address aligned (0x5597c).
>   It inherits this value from the calling routine as the sum of
>   "set" (8-byte aligned) + "def->offset"=20 => misaligned address.
> 
>   (settings-parser.c:1597 in settings_dup_full())
>   src = CONST_PTR_OFFSET(set, def->offset);
> 
>   (gdb) p set
>   $2 = (const void *) 0x55968
>   (gdb) p *def
>   $3 = {type = SET_SIZE, key = 0x2d548 
> "submission_max_mail_size", offset = 20, list_info = 0x0}

This is unexpected. But I don't see how it's a Dovecot bug. It seems as if your 
compiler doesn't do padding correctly and then crashes because it didn't do it 
correctly. I guess you're compiling this as 32bit? Is size_t 32bit or 64bit?

Can you try with the below small test program if it prints the same 20?

#include 
#include 
#include 

#define in_port_t unsigned short

struct submission_settings {
bool verbose_proctitle;
const char *rawlog_dir;

const char *hostname;

const char *login_greeting;
const char *login_trusted_networks;

/* submission: */
size_t submission_max_mail_size;
unsigned int submission_max_recipients;
const char *submission_client_workarounds;
const char *submission_logout_format;

/* submission backend: */
const char *submission_backend_capabilities;

/* submission relay: */
const char *submission_relay_host;
in_port_t submission_relay_port;
bool submission_relay_trusted;

const char *submission_relay_user;
const char *submission_relay_master_user;
const char *submission_relay_password;

const char *submission_relay_ssl;
bool submission_relay_ssl_verify;

const char *submission_relay_rawlog_dir;
unsigned int submission_relay_max_idle_time;

unsigned int submission_relay_connect_timeout;
unsigned int submission_relay_command_timeout;

/* imap urlauth: */
const char *imap_urlauth_host;
in_port_t imap_urlauth_port;

int parsed_workarounds;
};

int main(void)
{
struct submission_settings set;

printf("offset = %ld\n", offsetof(struct submission_settings, 
submission_max_mail_size));
printf("size = %ld\n", sizeof(set.submission_max_mail_size));
return 0;
}

Re: Dovecot LMTP mixing up users on multi-recipient mail

[Timo Sirainen via dovecot]

On 27 Jun 2019, at 14.21, Bernhard Schmidt via dovecot  
wrote:
> 
> Hi,
> 
> I've upgraded a mailstore from Debian Jessie (aka oldstable) with
> Dovecot 2.2.13 to Debian Buster (next stable) with Dovecot 2.3.4.1
> today. It worked pretty well, except that we're seeing error messages
> very similar to this old thread
> 
> https://dovecot.org/pipermail/dovecot/2015-July/101396.html
> 
> It appears to be happening when a mail with multiple recipients on this
> message store is getting delivered through lmtp.
> 
> Jun 27 11:47:36 lxmhs74 dovecot: 
> lmtp(user1)<47683>: Error: 
> open(/var/cache/dovecot/index/n/user2n/.INBOX/dovecot.index.cache) failed: 
> Permission denied (euid=3814520() egid=12(man) missing +x perm: 
> /var/cache/dovecot/index/n/user2, dir owned by 3391995:12 mode=0700)
> 
> user1 uid is 3814520, user2n uid is 3391995. Dovecot appears to be trying
> to deliver the message to user1 while using the index directory of user2n.

When delivering multiple mails with LMTP it first writes the mail to the first 
recipient. It then leaves this mail open and uses it to copy the mail to the 
next recipient. This allows the possibility of e.g. using hard links if the 
filesystem permissions are the same with both recipients, although that won't 
happen in your case. Anyway, apparently this copying attempts to update the 
first recipient's dovecot.index.cache for some reason. I'm not sure why exactly 
this is different in v2.2 and v2.3.

I wasn't able to reproduce this easily though, except with some special plugin 
it happened. This change helped with it:

diff --git a/src/lmtp/lmtp-local.c b/src/lmtp/lmtp-local.c
index e43f156d3..93848ef27 100644
--- a/src/lmtp/lmtp-local.c
+++ b/src/lmtp/lmtp-local.c
@@ -669,6 +669,9 @@ lmtp_local_deliver_to_rcpts(struct lmtp_local *local,
   will be unreferenced later on */
local->rcpt_user = NULL;
src_mail = local->first_saved_mail;
+   struct mail_private *pmail =
+   (struct mail_private *)src_mail;
+   pmail->v.set_uid_cache_updates(src_mail, TRUE);
first_uid = geteuid();
i_assert(first_uid != 0);
}

Re: FTS delays

[Timo Sirainen via dovecot]

It's because you're misunderstanding how the lookup() function works. It gets 
ALL the search parameters, including the "mailbox inbox". This is intentional, 
and not a bug. Two reasons being:

1) The FTS plugin in theory could support indexing/searching any kinds of 
searches, not just regular word searches. So I didn't want to limit it 
unnecessarily.

2) Especially with "mailbox inbox" this is important when searching from 
virtual mailboxes. If you configure "All mails in all folders" virtual mailbox, 
you can do a search in there that restricts which physical mailboxes are 
matched. In this case the FTS backend can optimize this lookup so it can filter 
only the physical mailboxes that have matches, leaving the others out. And it 
can do this in a single query if all the mailboxes are in the same FTS index.

So again: Your lookup() function needs to be changed to only use those search 
args that it really wants to search, and ignore the others. Use 
solr_add_definite_query_args() as the template.

Also I see now the reason for the timeout problem. It's because you're not 
setting search_arg->match_always=TRUE. These need to be set for the search args 
that you're actually using to generate the Xapian query. If it's not set, then 
Dovecot core doesn't think that the arg was part of the FTS search and it 
processes it itself. Meaning that it opens all the emails and does the search 
the slow way, practically making the FTS lookup ignored.

> On 21 Apr 2019, at 19.50, Joan Moreau  wrote:
> 
> No, the parsing is made by dovecot core, that is nothing the backend can do 
> about it. The backend shall *never*  reveive this. (would it be buggy or no)
> 
> 
> 
> PLease, have a look deeper
> 
> And the loop is a very big problem as it times out all the time (and once 
> again, this is not in any of the backend  functions)
> 
>  
> 
> 
> On 2019-04-21 10:42, Timo Sirainen via dovecot wrote:
> 
>> Inbox appears in the list of arguments, because fts_backend_xapian_lookup() 
>> is parsing the search args wrong. Not sure about the other issue.
>> 
>>> On 21 Apr 2019, at 19.31, Joan Moreau >> <mailto:j...@grosjo.net>> wrote:
>>> 
>>> For this first point, the problem is that dovecot core sends TWICE the 
>>> request and "Inbox" appears in the list of arguments ! (inbox shall serve 
>>> to select teh right mailbox, never sent to the backend)
>>> 
>>> And even if this would be solved, the dovecot core loops *after* the 
>>> backend hs returneds the results
>>> 
>>> 
>>> 
>>> # doveadm search -u j...@grosjo.net <mailto:j...@grosjo.net> mailbox inbox 
>>> text milan
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Get last UID of 
>>> INBOX = 315526
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Get last UID of 
>>> INBOX = 315526
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query: FLAG=AND
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query(1): add 
>>> term(wilcard) : inbox
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query(2): add 
>>> term(wilcard) : milan
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Testing if wildcard
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query: set GLOBAL 
>>> (no specified header)
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query : ( 
>>> bcc:inbox OR body:inbox OR cc:inbox OR from:inbox OR message-id:inbox OR 
>>> subject:inbox OR to:inbox ) AND ( bcc:milan OR body:milan OR cc:milan OR 
>>> from:milan OR message-id:milan OR subject:milan OR to:milan )
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query: 2 results 
>>> in 1 ms // THIS IS WHEN BACKEND HAS FOUND RESULTS AND STOPPED
>>> d82b4b0f550d3859364495331209 847
>>> d82b4b0f550d3859364495331209 1569
>>> d82b4b0f550d3859364495331209 2260
>>> d82b4b0f550d3859364495331209 2575
>>> d82b4b0f550d3859364495331209 2811
>>> d82b4b0f550d3859364495331209 2885
>>> d82b4b0f550d3859364495331209 3038
>>> d82b4b0f550d3859364495331209 3121 -> LOOPING FOREVER
>>> 
>>> 
>>> 
>>>  
>>> 
>>> 
>>> On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:
>>> 
>>> On 3 Apr 2019, at 20.30, Joan Moreau via dovecot >> <mailto:dovecot@dovecot.org>> wrote:
>>> doveadm search -u j...@grosjo.net <mailto:j...@grosjo.net> mailbox inbox 
>>> text milan
>>> output
>>> 
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query : ( 
>>> bcc:inbox OR body:inbox OR cc:inbox OR from:inbox OR message-id:inbox OR 
>>> subject:inbox OR to:inbox OR uid:inbox ) AND ( bcc:milan OR body:milan OR 
>>> cc:milan OR from:milan OR message-id:milan OR subject:milan OR to:milan OR 
>>> uid:milan )
>>> 
>>> 1 - The query is wrong
>>> 
>>> That's because fts_backend_xapian_lookup() isn't anywhere close to being 
>>> correct. Try to copy the logic based on solr_add_definite_query_args().
>>> 
>>> 

Re: FTS delays

[Timo Sirainen via dovecot]

Inbox appears in the list of arguments, because fts_backend_xapian_lookup() is 
parsing the search args wrong. Not sure about the other issue.

> On 21 Apr 2019, at 19.31, Joan Moreau  wrote:
> 
> For this first point, the problem is that dovecot core sends TWICE the 
> request and "Inbox" appears in the list of arguments ! (inbox shall serve to 
> select teh right mailbox, never sent to the backend)
> 
> And even if this would be solved, the dovecot core loops *after* the backend 
> hs returneds the results
> 
> 
> 
> # doveadm search -u j...@grosjo.net mailbox inbox text milan
> doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
> doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
> doveadm(j...@grosjo.net): Info: Query: FLAG=AND
> doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
> doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
> doveadm(j...@grosjo.net): Info: Testing if wildcard
> doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
> doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
> OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox ) AND ( 
> bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
> subject:milan OR to:milan )
> doveadm(j...@grosjo.net): Info: Query: 2 results in 1 ms // THIS IS WHEN 
> BACKEND HAS FOUND RESULTS AND STOPPED
> d82b4b0f550d3859364495331209 847
> d82b4b0f550d3859364495331209 1569
> d82b4b0f550d3859364495331209 2260
> d82b4b0f550d3859364495331209 2575
> d82b4b0f550d3859364495331209 2811
> d82b4b0f550d3859364495331209 2885
> d82b4b0f550d3859364495331209 3038
> d82b4b0f550d3859364495331209 3121 -> LOOPING FOREVER
> 
> 
> 
>  
> 
> 
> On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:
> 
>> On 3 Apr 2019, at 20.30, Joan Moreau via dovecot > <mailto:dovecot@dovecot.org>> wrote:
>>> 
>>> doveadm search -u j...@grosjo.net <mailto:j...@grosjo.net> mailbox inbox 
>>> text milan
>>> output
>>> 
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query : ( 
>>> bcc:inbox OR body:inbox OR cc:inbox OR from:inbox OR message-id:inbox OR 
>>> subject:inbox OR to:inbox OR uid:inbox ) AND ( bcc:milan OR body:milan OR 
>>> cc:milan OR from:milan OR message-id:milan OR subject:milan OR to:milan OR 
>>> uid:milan )
>>> 
>>> 1 - The query is wrong
>> 
>> That's because fts_backend_xapian_lookup() isn't anywhere close to being 
>> correct. Try to copy the logic based on solr_add_definite_query_args().
>> 
>> 

Re: FTS delays

[Timo Sirainen via dovecot]

On 3 Apr 2019, at 20.30, Joan Moreau via dovecot  wrote:
> doveadm search -u j...@grosjo.net mailbox inbox text milan
> output
> 
> doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
> OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) 
> AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan 
> OR subject:milan OR to:milan OR uid:milan )
> 
> 1 - The query is wrong

That's because fts_backend_xapian_lookup() isn't anywhere close to being 
correct. Try to copy the logic based on solr_add_definite_query_args().

Re: sieve scripts not synching for 2.3.5.1 pre-built

[Timo Sirainen via dovecot]

On 2 Apr 2019, at 22.37, Timo Sirainen via dovecot  wrote:
> 
> On 2 Apr 2019, at 17.03, Jan-Pieter Cornet via dovecot  <mailto:dovecot@dovecot.org>> wrote:
>> 
>> Hi,
>> 
>> We're synching mailboxes, changing format from maildir to mdbox, using 
>> doveadm backup/doveadm sync.
>> 
>> When still running 2.2.36, 'doveadm backup' also synched the sieve scripts, 
>> without issues.
>> 
>> After the upgrade to 2.3.5.1, the sieve sync stopped working. We're using 
>> the pre-built 2.3 packages from 
>> https://repo.dovecot.org/ce-2.3-latest/debian/stretch 
>> <https://repo.dovecot.org/ce-2.3-latest/debian/stretch>
> 
> Looks like this is trivial to reproduce. It used to work still in v2.3.1, but 
> then something broke it. Tracking internally in DOP-1062.

Reverting 
https://github.com/dovecot/pigeonhole/commit/479c5e57046dec76078597df844daccbfc0eb75f
 
<https://github.com/dovecot/pigeonhole/commit/479c5e57046dec76078597df844daccbfc0eb75f>
 fixes this.

Re: sieve scripts not synching for 2.3.5.1 pre-built

[Timo Sirainen via dovecot]

On 2 Apr 2019, at 17.03, Jan-Pieter Cornet via dovecot  
wrote:
> 
> Hi,
> 
> We're synching mailboxes, changing format from maildir to mdbox, using 
> doveadm backup/doveadm sync.
> 
> When still running 2.2.36, 'doveadm backup' also synched the sieve scripts, 
> without issues.
> 
> After the upgrade to 2.3.5.1, the sieve sync stopped working. We're using the 
> pre-built 2.3 packages from 
> https://repo.dovecot.org/ce-2.3-latest/debian/stretch 
> 

Looks like this is trivial to reproduce. It used to work still in v2.3.1, but 
then something broke it. Tracking internally in DOP-1062.

Re: Trying to track down source of duplicate messages

[Timo Sirainen via dovecot]

On 1 Apr 2019, at 19.40, Alex via dovecot  wrote:
> 
> Hi,
> 
> I haven't received any responses to my duplicate messages problem. It
> occurred to me that I posted my full dovecot config instead of just
> the changes we've made locally. I thought it might help to follow up
> with just the specific config to make it easier to identify a
> potential problem.

How are you delivering the mails? With dovecot-lda or something else? Do you 
see any errors/warnings in your MTA log? Similar problems at least can happen 
if the delivery takes a very long time and MTA times out and retries the 
delivery later on, but the original delivery actually succeeded eventually. You 
might see these differences in Received headers.

Re: FTS delays

[Timo Sirainen via dovecot]

On 2 Apr 2019, at 6.38, Joan Moreau via dovecot  wrote:
> 
> Further on this topic:
> 
> 
> 
> When choosing any headers in the search box, dovecot core calls the plugin 
> TWICE (and returns the results quickly, but not immediatly after getting the 
> IDs from the plugins)
> 
> When choosing the BODY search, dovecot core calls the plugin ONCE (and never 
> returns) (whereas the plugins returns properly the IDs)
> 

If we simplify this, do you mean this calls it once and is fast:

doveadm search -u user@domain mailbox inbox body helloworld

But this calls twice and is slow:

doveadm search -u user@domain mailbox inbox text helloworld

And what about searching e.g. subject? :

doveadm search -u user@domain mailbox inbox subject helloworld

And does the slowness depend on whether there were any matches or not?

> This is based on GIT version. (previous versions were working properly)

Previous versions were fast? Do you mean v2.3.5?

Re: Panic: file mail-transaction-log-file.c: line 105 (mail_transaction_log_file_free): assertion failed: (!file->locked)

[Timo Sirainen via dovecot]

On 28 Mar 2019, at 10.15, Arkadiusz Miśkiewicz  wrote:
> 
>  error = 0x55e3e2b40ac0 "Fixed index file
> /var/mail/piast_efaktury/dovecot.index: log_file_seq 13 -> 15",
>  nodiskspace = true,

This was one of the things I was first wondering, but I'm not sure why it's not 
logging an error. Anyway, you're using filesystem quota? And this index is 
large enough that trying to rewrite it brings the user over quota?

Re: v2.2.27 Panic: file rfc822-parser.h: line 23 (rfc822_parser_deinit): assertion failed: (ctx->data <= ctx->end)

[Timo Sirainen via dovecot]

On 27 Mar 2019, at 1.25, Jason Lewis via dovecot  wrote:
> 
> Hi Aki,
> 
> debian jessie backports has been moved to archive.debian.org and
> initially I was unable to install dovecot-dbg because of that. But I've
> managed to resolve that issue now.
> 
> This was the command I ran:
> doveadm -D -f flow fetch imap.envelope mailbox crm-spam.2008.g
> 
> Backtrace follows.

I've a feeling Debian's security fix backports didn't work properly:

> #5  0x7f3a7c34a97d in rfc822_parser_deinit (ctx=0x7ffdc7615e38,
> ctx=0x7ffdc7615e38) at rfc822-parser.h:23

rfc822_parser_deinit() wasn't added until v2.2.31. I think it was added as part 
of a security fix.

>data=data@entry=0x5563c13f3910 "To: bluef...@dickson.st,
> ja...@dickson.st, lewisja...@dickson.st, 05 Jul 2008 16:39:47 -0500
> PDT6Q--q=dns; c=nofws;d sender)
> smtp.mail=matt_coo...@postnewsweektech.com; domainkeys=pass (test mode)
> hea"..., size=size@entry=64,

I tried fetching a mail with these contents in v2.2.27, v2.2.33 and master. 
They all worked fine.

Re: Panic: file mail-transaction-log-file.c: line 105 (mail_transaction_log_file_free): assertion failed: (!file->locked)

[Timo Sirainen via dovecot]

On 27 Mar 2019, at 14.58, Timo Sirainen via dovecot  wrote:
> 
>> dovecot isn't able to auto fix the indexes and manual deletion is
>> required in all such cases
> 
> So if it keeps repeating, it's very strange. Could you send me such broken 
> dovecot.index and dovecot.index.log files (without dovecot.index.cache)? They 
> shouldn't contain anything sensitive (only message flags).

Tested with the index files you sent. It gets fixed automatically in my tests.

The backtrace shows that after fsck it fails to write the fixed index to the 
disk, because mail_index_write() fails for some reason. Except there's no error 
logged about it, which is rather weird. Do you still have the lmtp core? Could 
you do:

fr 9
p *log.index

Re: Panic: file mail-transaction-log-file.c: line 105 (mail_transaction_log_file_free): assertion failed: (!file->locked)

[Timo Sirainen via dovecot]

On 27 Mar 2019, at 12.42, Arkadiusz Miśkiewicz via dovecot 
 wrote:
> 
> 
> Hello.
> 
> I have one account with heavy traffic (big mails) and quite often
> indexes get corrupted.
> 
> This is dovecot 2.3.5 on local fs (xfs), Linux kernel 4.19.20, glibc 2.28.
> 
> When corruption happens lmtp and pop3 segfault on accessing it like:
> 
>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(24428): Connect from local 
>>  
>>  
>> [0/0]
>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(piast_efaktury): pid=<24428> 
>> session=, Error: Index 
>> /var/mail/piast_efaktury/dovecot.index: Lost log for seq=13 offset=25648: 
>> Missing middle file seq=13 (between 13..4294967295, we have seqs 14,15): 
>> .log.2 contains file_seq=14 (initial_mapped=0, reason=Index mapped)
>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(piast_efaktury): pid=<24428> 
>> session=, Warning: fscking index file 
>> /var/mail/piast_efaktury/dovecot.index
>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(piast_efaktury): pid=<24428> 
>> session=, Error: Fixed index file 
>> /var/mail/piast_efaktury/dovecot.index: log_file_seq 13 -> 15

dovecot.index says that it was generated against dovecot.index.log sequence 13, 
but the .log file already has sequence 15. I could maybe believe such a bug 
with long-running IMAP connections, but this is LMTP. And it's supposed to be 
fixing the problem here..

>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(piast_efaktury): pid=<24428> 
>> session=, Panic: file mail-transaction-log-file.c: 
>> line 105 (mail_transaction_log_file_free): assertion failed: (!file->locked)

Even though it crashes here, it's already supposed to have fixed the problem.

> dovecot isn't able to auto fix the indexes and manual deletion is
> required in all such cases

So if it keeps repeating, it's very strange. Could you send me such broken 
dovecot.index and dovecot.index.log files (without dovecot.index.cache)? They 
shouldn't contain anything sensitive (only message flags).

Also what's your doveconf -n?

Re: imap ---- LIST "" * The returned mailbox does not display quotes

[Timo Sirainen via dovecot]

On 21 Mar 2019, at 6.22, lty via dovecot  wrote:
> 
> dovecot  version
> 
> v2.2.29
> 
> v2.2.36
> 
> v2.3.5
> 
> LIST "" * The returned mailbox does not display quotes
> 
> v2.1.17
> LIST "" * The returned mailbox shows quotation marks
> 
> Why is the quotation mark removed in the new version?
> 

Because they were unnecessary. The code was changed to use more generic IMAP 
token writer, which adds quotes only as necessary.

> Is there any configuration option in the new version to add quotes?
> 

No. Does it break some client?

Re: flags not synced correctly with dovecot sync (dsync)

[Timo Sirainen via dovecot]

On 13 Mar 2019, at 22.43, Dan Christensen via dovecot  
wrote:
> 
> On Mar 12, 2019, Dan Christensen via dovecot  wrote:
> 
>> In another thread, Timo wrote:
>> 
>> On Mar 12, 2019, Timo Sirainen via dovecot  wrote:
>> 
>>> That bug is fixed with attached patch.
>> 
>> I'll report back once I've tested it.
> 
> I applied 2656.patch to version 2.3.5 as found at
> 
>  https://repo.dovecot.org/ce-2.3-latest/ubuntu/bionic/pool/main/2.3.5-1_ce/
> 
> and rebuilt the ubuntu package.  I installed the resulting package on
> all three machines.  And still I can reproduce the bug involving unread
> messages getting marked as read.

Looks like you're also using Maildir, which has another bug of keywords not 
being copied correctly.

Re: question about %u and %h used in mail_location

[Timo Sirainen via dovecot]

On 12 Mar 2019, at 12.04, Joe Wong via dovecot  wrote:
> 
> Hello,
> 
>  I have defined the following: 
> 
> mail_location = maildir:~:INBOX=~/Maildir:LAYOUT=fs:INDEX=%u
> 
> %u is retrieve via database in that my username contain ":", in which it 
> create some confusion to dovecot:
> 
> doveadm index -u user1:site@domain iNBOX
> doveadm(user1:site@domain): Error: remote(192.168.10.22:24245 
> ): Namespace '': Unknown setting: site@domain
> 
> I cannot change the value in DB so is there a workaround to this problem?

Convert the username to not have ':' when it comes out of auth. Could be 
possible with SQL userdb. With others .. I'm not sure, you might have to use 
Lua to convert it.

Re: Delayed flags changes over IDLE

[Timo Sirainen via dovecot]

On 12 Mar 2019, at 10.21, Kostya Vasilyev via dovecot  
wrote:
> 
> It makes no difference if the IDLE connection does SELECT or SELECT 
> (CONDSTORE) prior to going IDLE.
> 
> But then as far as I know (?) - in Dovecot, once any connection uses 
> CONDSTORE ever, even once, Dovecot creates data structures to track MODSEQ 
> values, and those data structures are forever.

So are you saying that you can reproduce if you do for a completely new user:

doveadm exec imap -u testuser1
a select inbox
b idle

And then run:
echo foo | doveadm save -u testuser1
doveadm flags add -u testuser1 '\Seen' mailbox inbox 1

And the EXISTS shows up immediately after saving, but the flag change won't 
show up? It works fine with me.

Do you see any errors in "doveadm log errors"? Can you reproduce this if you 
try with some other mailbox format than mbox?

Re: dovecot-keywords are not preserved any more when moving mails between folders

[Timo Sirainen via dovecot]

On 12 Mar 2019, at 17.55, Dan Christensen via dovecot  
wrote:
> 
> On Mar 12, 2019, Aki Tuomi via dovecot  wrote:
> 
>> On 12.3.2019 13.46, Piper Andreas via dovecot wrote:
>> 
>>> after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
>>> which in my case are set by thunderbird, are not preserved any more when
>>> moving a mail between folders.
>> 
>> We are aware of this bug, and it's being tracked as DOP-842.
> 
> Could this bug also be causing flags to be lost when using dsync
> (as I described in some messages to this list Feb 16 to 23)?
> 
> It seems like it might be a different bug, since in my experience
> the flags are sometimes synced and then removed later.

That bug is fixed with attached patch.



2656.patch
Description: Binary data

Re: “doveadm mailbox” command fails with UTF-8 mailboxes

[Timo Sirainen via dovecot]

On 12 Mar 2019, at 21.20, Felipe Gasper via dovecot  wrote:
> 
> Hello,
> 
>   I’ve got a strange misconfiguration where the following command:
> 
> doveadm -f pager mailbox status -u spamutf8 'messages vsize guid' INBOX 
> 'INBOX.*'
> 
> … fails with error code 68, saying that it can’t find one of the mailboxes. 
> (It lists the user’s other mailboxes.) The name of the mailbox in question is 
> saved to disk in UTF-8 rather than mUTF-7, but strace shows that doveadm is 
> stat()ing the mUTF-7 path; the failure of that stat() is, assumedly, what 
> causes doveadm to report the error status.
> 
>   I’ve tried to paw through the source code to see what might be causing 
> this but haven’t made much headway. Can someone here point out where the 
> misconfiguration might be that is causing doveadm to stat() the mUTF-7 path 
> rather than UTF-8? Or perhaps offer any tips as to how I might diagnose 
> what’s going on? What causes doveadm to stat() one path or the other?

What's your doveconf -n? Using UTF-8 on filesystem requires using "UTF-8" 
option in mail_location. Do you have it set? 
https://wiki2.dovecot.org/MailLocation

Re: Regression ACL & namespace prefix

[Timo Sirainen via dovecot]

On 18 Sep 2018, at 17.10, Michal Hlavinka  wrote:
> 
> Seems that for Global ACL directory, namespace prefix is not part of the 
> path, when looking for acl file.

Is there a reason you're using ACL directory instead of ACL file? I've rather 
been thinking about removing code for ACL directories entirely at some point.

Re: Delayed flags changes over IDLE

[Timo Sirainen via dovecot]

On 10 Mar 2019, at 10.14, Kostya Vasilyev via dovecot  
wrote:
> 
> My mail is stored under ~/mail/.imap (not sure what this format is called), I 
> mean not "single file mbox".
> 
> I have not changed any IDLE related config settings:
> 
> doveconf  | grep -i idle
> default_idle_kill = 1 mins
> director_ping_idle_timeout = 30 secs
> imap_idle_notify_interval = 2 mins
> imapc_max_idle_time = 29 mins
> mailbox_idle_check_interval = 30 secs
> 
> What can I do to make Dovecot notify IDLE clients about flags changes - more 
> quickly? Preferably near-instant?

It should simply just work, assuming there aren't any weird inotify limits, but 
you should get errors logged about reaching those. You could see if it makes 
any difference to set mailbox_idle_check_interval=1s

Re: imap-hibernate not working

[Timo Sirainen via dovecot]

On 8 Mar 2019, at 20.44, Marcelo Coelho via dovecot  wrote:
> 
> Hi,
> 
> I follow different setup instructions and I can't make imap-hibernate work. 
> I've tried vmail and dovecot as users, tried to set mode to 0666, without 
> success. I'm using FreeBSD 11.2.
> 
> Is imap-hibernate compatible with FreeBSD 11.2?
> 
> 
> 
> My operational system:
> 
> # uname -v
> FreeBSD 11.2-RELEASE-p9 #0: Tue Feb  5 15:30:36 UTC 2019 
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC 
> 
> Here are my logs:
> 
> Mar  8 15:30:57 servername dovecot: imap(u...@domain.com:52.125.128.90): 
> Error: kevent(-1) for notify remove failed: Bad file descriptor
> Mar  8 15:30:57 servername dovecot: imap(u...@domain.com:52.125.128.90): 
> Error: close(-1) for notify remove failed: Bad file descriptor
> Mar  8 15:30:57 servername dovecot: imap-hibernate: Error: Failed to parse 
> client input: Invalid peer_dev_minor value: 18446744073709486335
> Mar  8 15:30:57 servername dovecot: imap(u...@domain.com:52.125.128.90): 
> Error: /opt/dovecot/2.3.5/var/run/dovecot/imap-hibernate returned failure: 
> Failed to parse client input: Invalid peer_dev_minor value: 
> 18446744073709486335

Looks bad. I suppose it's broken with FreeBSD.

Re: submission-login: Fatal: master: service(submission-login):

[Timo Sirainen via dovecot]

On 11 Mar 2019, at 13.53, Marcelo Coelho via dovecot  
wrote:
> 
> Hi everyone!
> 
> I’m using dovecot 2.3.5. submission-login is crashing many times in a day:
> 
> Here is a sample error message:
> 
> dovecot: submission-login: Fatal: master: service(submission-login): child 
> 34247 killed with signal 11 (core not dumped - 
> https://dovecot.org/bugreport.html#coredumps - set service submission-login { 
> drop_priv_before_exec=yes })
> 
> After I added drop_priv_before_exec, I got these error messages:

If you're using Linux, you could alternatively: sysctl -w fs.suid_dumpable=2

> submission-login: Error: master-service: cannot get filters: 
> net_connect_unix(/var/run/dovecot/config) failed: Permission denied
> dovecot: master: Error: service(submission-login): command startup failed, 
> throttling for 2 secs


This could be avoided with adding to dovecot.conf:

service config {
  unix_listener config {
mode = 0666
  }
}

Re: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)

[Timo Sirainen via dovecot]

Hi,

The rescan() function is a bit badly designed. Currently what you could do what 
fts-lucene does and:
 - Get list of UIDs for all mails in each folder
 - If Xapian has UID that doesn't exist -> delete it from Xapian
 - If UID is missing from Xapian -> expunge the rest of the UIDs in that 
folder, so the next indexing will cause them to be indexed

The expunging of rest of the mails is rather ugly, yes.. A better API would be 
if backend simply had a way to iterate all mails in the index, preferrably 
sorted by folder. Then a more generic code could go through them and expunge 
the necessary mails and index the missing mails. Although not all FTS backends 
support indexing in the middle. Anyway, we don't really have time to implement 
this new API soon.

I'm not sure if this is a big problem though. I don't think most people running 
FTS have ever run rescan.

> On 8 Feb 2019, at 9.54, Joan Moreau via dovecot  wrote:
> 
> 
> 
>  
> Hi,
> 
> THis is a core problem in Dovecot in my understanding.
> 
> In my opinion, the rescan in dovecot should send to the FTS plugin the list 
> of "supposedly" indexed emails (UID), and the plugin shall purge the 
> redundant UID (i..e UID present in the index but not in the list sent by 
> dovecot) and send back the list of UID not in its indexes to dovecot, so 
> Dovect can send one by one the missing emails
> 
> 
> 
> WHat do you think ?
> 
> 
> 
>  Original Message 
> 
> Subject:  [grosjo/fts-xapian] `doveadm fts rescan` removes all indices 
> (#15)
> Date: 2019-02-08 08:28
> From: Leonard Lausen 
> To:   grosjo/fts-xapian 
> Cc:   Subscribed 
> Reply-To: grosjo/fts-xapian 
> 
> 
> 
> doveadm fts rescan -A deletes all indices, ie. all folders and files in the 
> xapian-indexes are deleted. However, according to man doveadm fts, the rescan 
> command should only
> 
>> Scan what mails exist in the full text search index and compare those to what
>> actually exist in mailboxes. This removes mails from the index that have 
>> already
>> been expunged and makes sure that the next doveadm index will index all the
>> missing mails (if any).
>> 
> Deleting all indices does not seem to be the intended action, especially as 
> constructing the index anew may take very long on large mailboxes.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub 
> , or mute the thread 
> .
> 
> 
> 

Re: system wakeup caused by write operations to /var/lib/dovecot/instances

[Timo Sirainen via dovecot]

On 2 Feb 2019, at 6.44, Tijl  wrote:
> How can dovecot be run without writing to /var/lib/dovecot/instances 
> everyday? Is there a configuration setting for this?

You'd need to patch src/master/main.c instance_update_now() to remove:

to_instance = timeout_add((3600 * 12 + i_rand_limit(60 * 30)) * 1000,
  instance_update_now, list);

I'm not quite sure why I wrote such code to update it continuously.