Re: FTS delays

2019-04-21 Thread Timo Sirainen via dovecot
It's because you're misunderstanding how the lookup() function works. It gets 
ALL the search parameters, including the "mailbox inbox". This is intentional, 
and not a bug. Two reasons being:

1) The FTS plugin in theory could support indexing/searching any kinds of 
searches, not just regular word searches. So I didn't want to limit it 
unnecessarily.

2) Especially with "mailbox inbox" this is important when searching from 
virtual mailboxes. If you configure "All mails in all folders" virtual mailbox, 
you can do a search in there that restricts which physical mailboxes are 
matched. In this case the FTS backend can optimize this lookup so it can filter 
only the physical mailboxes that have matches, leaving the others out. And it 
can do this in a single query if all the mailboxes are in the same FTS index.

So again: Your lookup() function needs to be changed to only use those search 
args that it really wants to search, and ignore the others. Use 
solr_add_definite_query_args() as the template.

Also I see now the reason for the timeout problem. It's because you're not 
setting search_arg->match_always=TRUE. These need to be set for the search args 
that you're actually using to generate the Xapian query. If it's not set, then 
Dovecot core doesn't think that the arg was part of the FTS search and it 
processes it itself. Meaning that it opens all the emails and does the search 
the slow way, practically making the FTS lookup ignored.

> On 21 Apr 2019, at 19.50, Joan Moreau  wrote:
> 
> No, the parsing is made by dovecot core, that is nothing the backend can do 
> about it. The backend shall *never*  reveive this. (would it be buggy or no)
> 
> 
> 
> PLease, have a look deeper
> 
> And the loop is a very big problem as it times out all the time (and once 
> again, this is not in any of the backend  functions)
> 
>  
> 
> 
> On 2019-04-21 10:42, Timo Sirainen via dovecot wrote:
> 
>> Inbox appears in the list of arguments, because fts_backend_xapian_lookup() 
>> is parsing the search args wrong. Not sure about the other issue.
>> 
>>> On 21 Apr 2019, at 19.31, Joan Moreau >> <mailto:j...@grosjo.net>> wrote:
>>> 
>>> For this first point, the problem is that dovecot core sends TWICE the 
>>> request and "Inbox" appears in the list of arguments ! (inbox shall serve 
>>> to select teh right mailbox, never sent to the backend)
>>> 
>>> And even if this would be solved, the dovecot core loops *after* the 
>>> backend hs returneds the results
>>> 
>>> 
>>> 
>>> # doveadm search -u j...@grosjo.net <mailto:j...@grosjo.net> mailbox inbox 
>>> text milan
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Get last UID of 
>>> INBOX = 315526
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Get last UID of 
>>> INBOX = 315526
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query: FLAG=AND
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query(1): add 
>>> term(wilcard) : inbox
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query(2): add 
>>> term(wilcard) : milan
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Testing if wildcard
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query: set GLOBAL 
>>> (no specified header)
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query : ( 
>>> bcc:inbox OR body:inbox OR cc:inbox OR from:inbox OR message-id:inbox OR 
>>> subject:inbox OR to:inbox ) AND ( bcc:milan OR body:milan OR cc:milan OR 
>>> from:milan OR message-id:milan OR subject:milan OR to:milan )
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query: 2 results 
>>> in 1 ms // THIS IS WHEN BACKEND HAS FOUND RESULTS AND STOPPED
>>> d82b4b0f550d3859364495331209 847
>>> d82b4b0f550d3859364495331209 1569
>>> d82b4b0f550d3859364495331209 2260
>>> d82b4b0f550d3859364495331209 2575
>>> d82b4b0f550d3859364495331209 2811
>>> d82b4b0f550d3859364495331209 2885
>>> d82b4b0f550d3859364495331209 3038
>>> d82b4b0f550d3859364495331209 3121 -> LOOPING FOREVER
>>> 
>>> 
>>> 
>>>  
>>> 
>>> 
>>> On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:
>>> 
>>> On 3 Apr 2019, at 20.30, Joan Moreau via dovecot >> <mailto:dovecot@dovecot.org>> wrote:
>>> doveadm search -u j...@grosjo.net <mailto:j...@grosjo.net> mailbox inbox 
>>> text milan
>>> output
>>> 
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query : ( 
>>> bcc:inbox OR body:inbox OR cc:inbox OR from:inbox OR message-id:inbox OR 
>>> subject:inbox OR to:inbox OR uid:inbox ) AND ( bcc:milan OR body:milan OR 
>>> cc:milan OR from:milan OR message-id:milan OR subject:milan OR to:milan OR 
>>> uid:milan )
>>> 
>>> 1 - The query is wrong
>>> 
>>> That's because fts_backend_xapian_lookup() isn't anywhere close to being 
>>> correct. Try to copy the logic based on solr_add_definite_query_args().
>>> 
>>> 



Re: FTS delays

2019-04-21 Thread Timo Sirainen via dovecot
Inbox appears in the list of arguments, because fts_backend_xapian_lookup() is 
parsing the search args wrong. Not sure about the other issue.

> On 21 Apr 2019, at 19.31, Joan Moreau  wrote:
> 
> For this first point, the problem is that dovecot core sends TWICE the 
> request and "Inbox" appears in the list of arguments ! (inbox shall serve to 
> select teh right mailbox, never sent to the backend)
> 
> And even if this would be solved, the dovecot core loops *after* the backend 
> hs returneds the results
> 
> 
> 
> # doveadm search -u j...@grosjo.net mailbox inbox text milan
> doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
> doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
> doveadm(j...@grosjo.net): Info: Query: FLAG=AND
> doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
> doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
> doveadm(j...@grosjo.net): Info: Testing if wildcard
> doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
> doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
> OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox ) AND ( 
> bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
> subject:milan OR to:milan )
> doveadm(j...@grosjo.net): Info: Query: 2 results in 1 ms // THIS IS WHEN 
> BACKEND HAS FOUND RESULTS AND STOPPED
> d82b4b0f550d3859364495331209 847
> d82b4b0f550d3859364495331209 1569
> d82b4b0f550d3859364495331209 2260
> d82b4b0f550d3859364495331209 2575
> d82b4b0f550d3859364495331209 2811
> d82b4b0f550d3859364495331209 2885
> d82b4b0f550d3859364495331209 3038
> d82b4b0f550d3859364495331209 3121 -> LOOPING FOREVER
> 
> 
> 
>  
> 
> 
> On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:
> 
>> On 3 Apr 2019, at 20.30, Joan Moreau via dovecot > <mailto:dovecot@dovecot.org>> wrote:
>>> 
>>> doveadm search -u j...@grosjo.net <mailto:j...@grosjo.net> mailbox inbox 
>>> text milan
>>> output
>>> 
>>> doveadm(j...@grosjo.net <mailto:j...@grosjo.net>): Info: Query : ( 
>>> bcc:inbox OR body:inbox OR cc:inbox OR from:inbox OR message-id:inbox OR 
>>> subject:inbox OR to:inbox OR uid:inbox ) AND ( bcc:milan OR body:milan OR 
>>> cc:milan OR from:milan OR message-id:milan OR subject:milan OR to:milan OR 
>>> uid:milan )
>>> 
>>> 1 - The query is wrong
>> 
>> That's because fts_backend_xapian_lookup() isn't anywhere close to being 
>> correct. Try to copy the logic based on solr_add_definite_query_args().
>> 
>> 



Re: FTS delays

2019-04-21 Thread Timo Sirainen via dovecot
On 3 Apr 2019, at 20.30, Joan Moreau via dovecot  wrote:
> doveadm search -u j...@grosjo.net mailbox inbox text milan
> output
> 
> doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
> OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) 
> AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan 
> OR subject:milan OR to:milan OR uid:milan )
> 
> 1 - The query is wrong

That's because fts_backend_xapian_lookup() isn't anywhere close to being 
correct. Try to copy the logic based on solr_add_definite_query_args().



Re: sieve scripts not synching for 2.3.5.1 pre-built

2019-04-02 Thread Timo Sirainen via dovecot
On 2 Apr 2019, at 22.37, Timo Sirainen via dovecot  wrote:
> 
> On 2 Apr 2019, at 17.03, Jan-Pieter Cornet via dovecot  <mailto:dovecot@dovecot.org>> wrote:
>> 
>> Hi,
>> 
>> We're synching mailboxes, changing format from maildir to mdbox, using 
>> doveadm backup/doveadm sync.
>> 
>> When still running 2.2.36, 'doveadm backup' also synched the sieve scripts, 
>> without issues.
>> 
>> After the upgrade to 2.3.5.1, the sieve sync stopped working. We're using 
>> the pre-built 2.3 packages from 
>> https://repo.dovecot.org/ce-2.3-latest/debian/stretch 
>> <https://repo.dovecot.org/ce-2.3-latest/debian/stretch>
> 
> Looks like this is trivial to reproduce. It used to work still in v2.3.1, but 
> then something broke it. Tracking internally in DOP-1062.

Reverting 
https://github.com/dovecot/pigeonhole/commit/479c5e57046dec76078597df844daccbfc0eb75f
 
<https://github.com/dovecot/pigeonhole/commit/479c5e57046dec76078597df844daccbfc0eb75f>
 fixes this.



Re: sieve scripts not synching for 2.3.5.1 pre-built

2019-04-02 Thread Timo Sirainen via dovecot
On 2 Apr 2019, at 17.03, Jan-Pieter Cornet via dovecot  
wrote:
> 
> Hi,
> 
> We're synching mailboxes, changing format from maildir to mdbox, using 
> doveadm backup/doveadm sync.
> 
> When still running 2.2.36, 'doveadm backup' also synched the sieve scripts, 
> without issues.
> 
> After the upgrade to 2.3.5.1, the sieve sync stopped working. We're using the 
> pre-built 2.3 packages from 
> https://repo.dovecot.org/ce-2.3-latest/debian/stretch 
> 

Looks like this is trivial to reproduce. It used to work still in v2.3.1, but 
then something broke it. Tracking internally in DOP-1062.



Re: Trying to track down source of duplicate messages

2019-04-02 Thread Timo Sirainen via dovecot
On 1 Apr 2019, at 19.40, Alex via dovecot  wrote:
> 
> Hi,
> 
> I haven't received any responses to my duplicate messages problem. It
> occurred to me that I posted my full dovecot config instead of just
> the changes we've made locally. I thought it might help to follow up
> with just the specific config to make it easier to identify a
> potential problem.

How are you delivering the mails? With dovecot-lda or something else? Do you 
see any errors/warnings in your MTA log? Similar problems at least can happen 
if the delivery takes a very long time and MTA times out and retries the 
delivery later on, but the original delivery actually succeeded eventually. You 
might see these differences in Received headers.



Re: FTS delays

2019-04-02 Thread Timo Sirainen via dovecot
On 2 Apr 2019, at 6.38, Joan Moreau via dovecot  wrote:
> 
> Further on this topic:
> 
> 
> 
> When choosing any headers in the search box, dovecot core calls the plugin 
> TWICE (and returns the results quickly, but not immediatly after getting the 
> IDs from the plugins)
> 
> When choosing the BODY search, dovecot core calls the plugin ONCE (and never 
> returns) (whereas the plugins returns properly the IDs)
> 

If we simplify this, do you mean this calls it once and is fast:

doveadm search -u user@domain mailbox inbox body helloworld

But this calls twice and is slow:

doveadm search -u user@domain mailbox inbox text helloworld

And what about searching e.g. subject? :

doveadm search -u user@domain mailbox inbox subject helloworld

And does the slowness depend on whether there were any matches or not?

> This is based on GIT version. (previous versions were working properly)

Previous versions were fast? Do you mean v2.3.5?



Re: Panic: file mail-transaction-log-file.c: line 105 (mail_transaction_log_file_free): assertion failed: (!file->locked)

2019-03-28 Thread Timo Sirainen via dovecot
On 28 Mar 2019, at 10.15, Arkadiusz Miśkiewicz  wrote:
> 
>  error = 0x55e3e2b40ac0 "Fixed index file
> /var/mail/piast_efaktury/dovecot.index: log_file_seq 13 -> 15",
>  nodiskspace = true,

This was one of the things I was first wondering, but I'm not sure why it's not 
logging an error. Anyway, you're using filesystem quota? And this index is 
large enough that trying to rewrite it brings the user over quota?



Re: v2.2.27 Panic: file rfc822-parser.h: line 23 (rfc822_parser_deinit): assertion failed: (ctx->data <= ctx->end)

2019-03-28 Thread Timo Sirainen via dovecot
On 27 Mar 2019, at 1.25, Jason Lewis via dovecot  wrote:
> 
> Hi Aki,
> 
> debian jessie backports has been moved to archive.debian.org and
> initially I was unable to install dovecot-dbg because of that. But I've
> managed to resolve that issue now.
> 
> This was the command I ran:
> doveadm -D -f flow fetch imap.envelope mailbox crm-spam.2008.g
> 
> Backtrace follows.

I've a feeling Debian's security fix backports didn't work properly:

> #5  0x7f3a7c34a97d in rfc822_parser_deinit (ctx=0x7ffdc7615e38,
> ctx=0x7ffdc7615e38) at rfc822-parser.h:23

rfc822_parser_deinit() wasn't added until v2.2.31. I think it was added as part 
of a security fix.

>data=data@entry=0x5563c13f3910 "To: bluef...@dickson.st,
> ja...@dickson.st, lewisja...@dickson.st, 05 Jul 2008 16:39:47 -0500
> PDT6Q--q=dns; c=nofws;d sender)
> smtp.mail=matt_coo...@postnewsweektech.com; domainkeys=pass (test mode)
> hea"..., size=size@entry=64,

I tried fetching a mail with these contents in v2.2.27, v2.2.33 and master. 
They all worked fine.



Re: Panic: file mail-transaction-log-file.c: line 105 (mail_transaction_log_file_free): assertion failed: (!file->locked)

2019-03-28 Thread Timo Sirainen via dovecot
On 27 Mar 2019, at 14.58, Timo Sirainen via dovecot  wrote:
> 
>> dovecot isn't able to auto fix the indexes and manual deletion is
>> required in all such cases
> 
> So if it keeps repeating, it's very strange. Could you send me such broken 
> dovecot.index and dovecot.index.log files (without dovecot.index.cache)? They 
> shouldn't contain anything sensitive (only message flags).

Tested with the index files you sent. It gets fixed automatically in my tests.

The backtrace shows that after fsck it fails to write the fixed index to the 
disk, because mail_index_write() fails for some reason. Except there's no error 
logged about it, which is rather weird. Do you still have the lmtp core? Could 
you do:

fr 9
p *log.index





Re: Panic: file mail-transaction-log-file.c: line 105 (mail_transaction_log_file_free): assertion failed: (!file->locked)

2019-03-28 Thread Timo Sirainen via dovecot
On 27 Mar 2019, at 12.42, Arkadiusz Miśkiewicz via dovecot 
 wrote:
> 
> 
> Hello.
> 
> I have one account with heavy traffic (big mails) and quite often
> indexes get corrupted.
> 
> This is dovecot 2.3.5 on local fs (xfs), Linux kernel 4.19.20, glibc 2.28.
> 
> When corruption happens lmtp and pop3 segfault on accessing it like:
> 
>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(24428): Connect from local 
>>  
>>  
>> [0/0]
>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(piast_efaktury): pid=<24428> 
>> session=, Error: Index 
>> /var/mail/piast_efaktury/dovecot.index: Lost log for seq=13 offset=25648: 
>> Missing middle file seq=13 (between 13..4294967295, we have seqs 14,15): 
>> .log.2 contains file_seq=14 (initial_mapped=0, reason=Index mapped)
>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(piast_efaktury): pid=<24428> 
>> session=, Warning: fscking index file 
>> /var/mail/piast_efaktury/dovecot.index
>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(piast_efaktury): pid=<24428> 
>> session=, Error: Fixed index file 
>> /var/mail/piast_efaktury/dovecot.index: log_file_seq 13 -> 15

dovecot.index says that it was generated against dovecot.index.log sequence 13, 
but the .log file already has sequence 15. I could maybe believe such a bug 
with long-running IMAP connections, but this is LMTP. And it's supposed to be 
fixing the problem here..

>> Mar 27 11:13:50 mbox dovecot[22370]: lmtp(piast_efaktury): pid=<24428> 
>> session=, Panic: file mail-transaction-log-file.c: 
>> line 105 (mail_transaction_log_file_free): assertion failed: (!file->locked)

Even though it crashes here, it's already supposed to have fixed the problem.

> dovecot isn't able to auto fix the indexes and manual deletion is
> required in all such cases

So if it keeps repeating, it's very strange. Could you send me such broken 
dovecot.index and dovecot.index.log files (without dovecot.index.cache)? They 
shouldn't contain anything sensitive (only message flags).

Also what's your doveconf -n?



Re: imap ---- LIST "" * The returned mailbox does not display quotes

2019-03-21 Thread Timo Sirainen via dovecot
On 21 Mar 2019, at 6.22, lty via dovecot  wrote:
> 
> dovecot  version
> 
> v2.2.29
> 
> v2.2.36
> 
> v2.3.5
> 
> LIST "" * The returned mailbox does not display quotes
> 
> v2.1.17
> LIST "" * The returned mailbox shows quotation marks
> 
> Why is the quotation mark removed in the new version?
> 

Because they were unnecessary. The code was changed to use more generic IMAP 
token writer, which adds quotes only as necessary.

> Is there any configuration option in the new version to add quotes?
> 

No. Does it break some client?



Re: flags not synced correctly with dovecot sync (dsync)

2019-03-14 Thread Timo Sirainen via dovecot
On 13 Mar 2019, at 22.43, Dan Christensen via dovecot  
wrote:
> 
> On Mar 12, 2019, Dan Christensen via dovecot  wrote:
> 
>> In another thread, Timo wrote:
>> 
>> On Mar 12, 2019, Timo Sirainen via dovecot  wrote:
>> 
>>> That bug is fixed with attached patch.
>> 
>> I'll report back once I've tested it.
> 
> I applied 2656.patch to version 2.3.5 as found at
> 
>  https://repo.dovecot.org/ce-2.3-latest/ubuntu/bionic/pool/main/2.3.5-1_ce/
> 
> and rebuilt the ubuntu package.  I installed the resulting package on
> all three machines.  And still I can reproduce the bug involving unread
> messages getting marked as read.

Looks like you're also using Maildir, which has another bug of keywords not 
being copied correctly.



Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Timo Sirainen via dovecot
On 13 Mar 2019, at 1.07, Helmut K. C. Tessarek  wrote:
> 
> On 2019-03-12 17:23, Timo Sirainen via dovecot wrote
>> https://wiki2.dovecot.org/MailLocation
> 
> Sorry, this might be off-topic, but while reading up on the link you sent,
> I've noticed the following sentence:
> 
> Use only absolute paths. Even if relative paths would appear to work, they
> might just as well break some day.
> 
> Yet, all examples in the documentation use ~ which is a relative path.

Well, I suppose it depends on definitions.. But I'm not calling ~/ relative 
paths, because it expands to an absolute path. The problem is using paths like 
"mail/" where it depends on the current chdir.



Re: question about %u and %h used in mail_location

2019-03-12 Thread Timo Sirainen via dovecot
On 12 Mar 2019, at 12.04, Joe Wong via dovecot  wrote:
> 
> Hello,
> 
>  I have defined the following: 
> 
> mail_location = maildir:~:INBOX=~/Maildir:LAYOUT=fs:INDEX=%u
> 
> %u is retrieve via database in that my username contain ":", in which it 
> create some confusion to dovecot:
> 
> doveadm index -u user1:site@domain iNBOX
> doveadm(user1:site@domain): Error: remote(192.168.10.22:24245 
> ): Namespace '': Unknown setting: site@domain
> 
> I cannot change the value in DB so is there a workaround to this problem?

Convert the username to not have ':' when it comes out of auth. Could be 
possible with SQL userdb. With others .. I'm not sure, you might have to use 
Lua to convert it.



Re: Delayed flags changes over IDLE

2019-03-12 Thread Timo Sirainen via dovecot
On 12 Mar 2019, at 10.21, Kostya Vasilyev via dovecot  
wrote:
> 
> It makes no difference if the IDLE connection does SELECT or SELECT 
> (CONDSTORE) prior to going IDLE.
> 
> But then as far as I know (?) - in Dovecot, once any connection uses 
> CONDSTORE ever, even once, Dovecot creates data structures to track MODSEQ 
> values, and those data structures are forever.

So are you saying that you can reproduce if you do for a completely new user:

doveadm exec imap -u testuser1
a select inbox
b idle

And then run:
echo foo | doveadm save -u testuser1
doveadm flags add -u testuser1 '\Seen' mailbox inbox 1

And the EXISTS shows up immediately after saving, but the flag change won't 
show up? It works fine with me.

Do you see any errors in "doveadm log errors"? Can you reproduce this if you 
try with some other mailbox format than mbox?



Re: dovecot-keywords are not preserved any more when moving mails between folders

2019-03-12 Thread Timo Sirainen via dovecot
On 12 Mar 2019, at 17.55, Dan Christensen via dovecot  
wrote:
> 
> On Mar 12, 2019, Aki Tuomi via dovecot  wrote:
> 
>> On 12.3.2019 13.46, Piper Andreas via dovecot wrote:
>> 
>>> after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
>>> which in my case are set by thunderbird, are not preserved any more when
>>> moving a mail between folders.
>> 
>> We are aware of this bug, and it's being tracked as DOP-842.
> 
> Could this bug also be causing flags to be lost when using dsync
> (as I described in some messages to this list Feb 16 to 23)?
> 
> It seems like it might be a different bug, since in my experience
> the flags are sometimes synced and then removed later.

That bug is fixed with attached patch.



2656.patch
Description: Binary data




Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Timo Sirainen via dovecot
On 12 Mar 2019, at 21.20, Felipe Gasper via dovecot  wrote:
> 
> Hello,
> 
>   I’ve got a strange misconfiguration where the following command:
> 
> doveadm -f pager mailbox status -u spamutf8 'messages vsize guid' INBOX 
> 'INBOX.*'
> 
> … fails with error code 68, saying that it can’t find one of the mailboxes. 
> (It lists the user’s other mailboxes.) The name of the mailbox in question is 
> saved to disk in UTF-8 rather than mUTF-7, but strace shows that doveadm is 
> stat()ing the mUTF-7 path; the failure of that stat() is, assumedly, what 
> causes doveadm to report the error status.
> 
>   I’ve tried to paw through the source code to see what might be causing 
> this but haven’t made much headway. Can someone here point out where the 
> misconfiguration might be that is causing doveadm to stat() the mUTF-7 path 
> rather than UTF-8? Or perhaps offer any tips as to how I might diagnose 
> what’s going on? What causes doveadm to stat() one path or the other?

What's your doveconf -n? Using UTF-8 on filesystem requires using "UTF-8" 
option in mail_location. Do you have it set? 
https://wiki2.dovecot.org/MailLocation



Re: Regression ACL & namespace prefix

2019-03-12 Thread Timo Sirainen via dovecot
On 18 Sep 2018, at 17.10, Michal Hlavinka  wrote:
> 
> Seems that for Global ACL directory, namespace prefix is not part of the 
> path, when looking for acl file.

Is there a reason you're using ACL directory instead of ACL file? I've rather 
been thinking about removing code for ACL directories entirely at some point.



Re: Delayed flags changes over IDLE

2019-03-11 Thread Timo Sirainen via dovecot
On 10 Mar 2019, at 10.14, Kostya Vasilyev via dovecot  
wrote:
> 
> My mail is stored under ~/mail/.imap (not sure what this format is called), I 
> mean not "single file mbox".
> 
> I have not changed any IDLE related config settings:
> 
> doveconf  | grep -i idle
> default_idle_kill = 1 mins
> director_ping_idle_timeout = 30 secs
> imap_idle_notify_interval = 2 mins
> imapc_max_idle_time = 29 mins
> mailbox_idle_check_interval = 30 secs
> 
> What can I do to make Dovecot notify IDLE clients about flags changes - more 
> quickly? Preferably near-instant?

It should simply just work, assuming there aren't any weird inotify limits, but 
you should get errors logged about reaching those. You could see if it makes 
any difference to set mailbox_idle_check_interval=1s



Re: imap-hibernate not working

2019-03-11 Thread Timo Sirainen via dovecot
On 8 Mar 2019, at 20.44, Marcelo Coelho via dovecot  wrote:
> 
> Hi,
> 
> I follow different setup instructions and I can't make imap-hibernate work. 
> I've tried vmail and dovecot as users, tried to set mode to 0666, without 
> success. I'm using FreeBSD 11.2.
> 
> Is imap-hibernate compatible with FreeBSD 11.2?
> 
> 
> 
> My operational system:
> 
> # uname -v
> FreeBSD 11.2-RELEASE-p9 #0: Tue Feb  5 15:30:36 UTC 2019 
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC 
> 
> Here are my logs:
> 
> Mar  8 15:30:57 servername dovecot: imap(u...@domain.com:52.125.128.90): 
> Error: kevent(-1) for notify remove failed: Bad file descriptor
> Mar  8 15:30:57 servername dovecot: imap(u...@domain.com:52.125.128.90): 
> Error: close(-1) for notify remove failed: Bad file descriptor
> Mar  8 15:30:57 servername dovecot: imap-hibernate: Error: Failed to parse 
> client input: Invalid peer_dev_minor value: 18446744073709486335
> Mar  8 15:30:57 servername dovecot: imap(u...@domain.com:52.125.128.90): 
> Error: /opt/dovecot/2.3.5/var/run/dovecot/imap-hibernate returned failure: 
> Failed to parse client input: Invalid peer_dev_minor value: 
> 18446744073709486335

Looks bad. I suppose it's broken with FreeBSD.



Re: submission-login: Fatal: master: service(submission-login):

2019-03-11 Thread Timo Sirainen via dovecot
On 11 Mar 2019, at 13.53, Marcelo Coelho via dovecot  
wrote:
> 
> Hi everyone!
> 
> I’m using dovecot 2.3.5. submission-login is crashing many times in a day:
> 
> Here is a sample error message:
> 
> dovecot: submission-login: Fatal: master: service(submission-login): child 
> 34247 killed with signal 11 (core not dumped - 
> https://dovecot.org/bugreport.html#coredumps - set service submission-login { 
> drop_priv_before_exec=yes })
> 
> After I added drop_priv_before_exec, I got these error messages:

If you're using Linux, you could alternatively: sysctl -w fs.suid_dumpable=2

> submission-login: Error: master-service: cannot get filters: 
> net_connect_unix(/var/run/dovecot/config) failed: Permission denied
> dovecot: master: Error: service(submission-login): command startup failed, 
> throttling for 2 secs


This could be avoided with adding to dovecot.conf:

service config {
  unix_listener config {
mode = 0666
  }
}



Re: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)

2019-02-14 Thread Timo Sirainen via dovecot
Hi,

The rescan() function is a bit badly designed. Currently what you could do what 
fts-lucene does and:
 - Get list of UIDs for all mails in each folder
 - If Xapian has UID that doesn't exist -> delete it from Xapian
 - If UID is missing from Xapian -> expunge the rest of the UIDs in that 
folder, so the next indexing will cause them to be indexed

The expunging of rest of the mails is rather ugly, yes.. A better API would be 
if backend simply had a way to iterate all mails in the index, preferrably 
sorted by folder. Then a more generic code could go through them and expunge 
the necessary mails and index the missing mails. Although not all FTS backends 
support indexing in the middle. Anyway, we don't really have time to implement 
this new API soon.

I'm not sure if this is a big problem though. I don't think most people running 
FTS have ever run rescan.

> On 8 Feb 2019, at 9.54, Joan Moreau via dovecot  wrote:
> 
> 
> 
>  
> Hi,
> 
> THis is a core problem in Dovecot in my understanding.
> 
> In my opinion, the rescan in dovecot should send to the FTS plugin the list 
> of "supposedly" indexed emails (UID), and the plugin shall purge the 
> redundant UID (i..e UID present in the index but not in the list sent by 
> dovecot) and send back the list of UID not in its indexes to dovecot, so 
> Dovect can send one by one the missing emails
> 
> 
> 
> WHat do you think ?
> 
> 
> 
>  Original Message 
> 
> Subject:  [grosjo/fts-xapian] `doveadm fts rescan` removes all indices 
> (#15)
> Date: 2019-02-08 08:28
> From: Leonard Lausen 
> To:   grosjo/fts-xapian 
> Cc:   Subscribed 
> Reply-To: grosjo/fts-xapian 
> 
> 
> 
> doveadm fts rescan -A deletes all indices, ie. all folders and files in the 
> xapian-indexes are deleted. However, according to man doveadm fts, the rescan 
> command should only
> 
>> Scan what mails exist in the full text search index and compare those to what
>> actually exist in mailboxes. This removes mails from the index that have 
>> already
>> been expunged and makes sure that the next doveadm index will index all the
>> missing mails (if any).
>> 
> Deleting all indices does not seem to be the intended action, especially as 
> constructing the index anew may take very long on large mailboxes.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub 
> , or mute the thread 
> .
> 
> 
> 



Re: system wakeup caused by write operations to /var/lib/dovecot/instances

2019-02-07 Thread Timo Sirainen via dovecot
On 2 Feb 2019, at 6.44, Tijl  wrote:
> How can dovecot be run without writing to /var/lib/dovecot/instances 
> everyday? Is there a configuration setting for this?

You'd need to patch src/master/main.c instance_update_now() to remove:

to_instance = timeout_add((3600 * 12 + i_rand_limit(60 * 30)) * 1000,
  instance_update_now, list);

I'm not quite sure why I wrote such code to update it continuously.



Re: Dovecot v2.2.36.1 released

2019-02-05 Thread Timo Sirainen
On 5 Feb 2019, at 7.48, Gerald Galster  wrote:
> 
> Hello Aki,
> 
>> https://dovecot.org/releases/2.2/dovecot-2.2.36.1.tar.gz
>> https://dovecot.org/releases/2.2/dovecot-2.2.36.1.tar.gz.sig
>> 
>>- pop3_no_flag_updates=no: Don't expunge RETRed messages without QUIT
> 
> is this in any way related to the problem that has first been reported in 
> march last year:
> 
> "Duplicate mails on pop3 expunge with dsync replication on 2.2.35 (2.2.33.2 
> works)"

Unlikely.



Re: dovecot/config processes one more time - which are safe to kill?

2019-01-13 Thread Timo Sirainen
On 13 Dec 2018, at 11.18, Arkadiusz Miśkiewicz  wrote:
> 
> 
> Hello.
> 
> The problem with dovecot/config processes never ending and spawning new
> one on each reload
> (https://www.dovecot.org/list/dovecot/2016-November/106058.html) is
> becoming a problem here:
> 
> # ps aux|grep dovecot/config|wc -l
> 206

I think you also have 206 other Dovecot processes that are keeping the config 
process open? Maybe 206 imap-login processes or something? Anyway I'd expect 
that this would happen only if some other process is keeping a UNIX socket 
connection open to the config process. Unless of course there's some bug that 
just isn't shutting them down even though they don't have any connections.. But 
at least I couldn't easily reproduce that.

I suppose there isn't much of a reason for existing processes to keep the 
config socket open after reload, so a patch like below would likely help. 
Although it probably should be delayed so that existing imap/pop3-login 
connections doing STARTTLS wouldn't fail if that causes a new config lookup.

diff --git a/src/lib-master/master-service.c b/src/lib-master/master-service.c
index 3de11fa1b..41005cb5e 100644
--- a/src/lib-master/master-service.c
+++ b/src/lib-master/master-service.c
@@ -815,6 +815,7 @@ void master_service_stop_new_connections(struct 
master_service *service)
}
if (service->login != NULL)
master_login_stop(service->login);
+   master_service_close_config_fd(service);
 }

 bool master_service_is_killed(struct master_service *service)

> That's a lot of wasted memory - dovecot/config processes ate over 30GB
> of ram on 64GB box.

Are you saying your config processes take 149 MB each? That doesn't sound 
right, unless you have a huge number of SSL certificates?

Re: Indexing paralelism

2019-01-13 Thread Timo Sirainen
On 13 Jan 2019, at 19.23, Joan Moreau via dovecot  wrote:
> 
> Hi
> 
> Observing the processes of FTS, I observe the following:
> 
> 
> 
> 1 - For one user, indexer-wroker does not start several threads for each 
> request. On teh contrary, it waits for the first request to finish before 
> starting the second. How to make sure all requests (or a limited number of 
> it, for instance linked to the CPU number on the machine) are started asap ?

I guess two answers:

1) Dovecot doesn't use threads anywhere. It uses only processes. So a single 
indexer-worker couldn't start multiple parallel threads.

2) It's intentional that there aren't more than one indexer-worker per user. 
This is especially because fts-lucene would break if there were more. Also 
generally in larger installations it's better if all the workers weren't stuck 
processing a single user, blocking other users' indexing.

> 2 - If a IMAP query is received, the dovecot checks the last UID from FTS and 
> launch a request of indexing to finish the index *before* running the search 
> query . THis creates timeouts (and can take a while if many request are 
> pending - see point 1) How to prevent that (i.e. the serach request is 
> launched (read only) no matter what ? THe completeion of missing UIDs is 
> launched in a separate thread ?

This would be violating IMAP protocol if it didn't include latest mails in the 
SEARCH response.. Generally I haven't noticed this being a big practical 
problem. The initial indexing can be done before FTS searches are enabled for 
the user, and afterwards the indexing shouldn't have especially long queues. 
Note that when doing indexing due to a SEARCH, that indexing's priority is 
higher than the indexing triggered by new mail deliveries. So unless all the 
indexer-workers are busy indexing mails from large folders without any indexes, 
this shouldn't be a huge problem normally.



Re: Solr -> Xapian ?

2019-01-13 Thread Timo Sirainen
On 13 Jan 2019, at 10.45, Joan Moreau via dovecot  wrote:
> 
> Now, I can see in the logs that several times, the dovecot calls the 
> fts_backend_xapian_update_set_mailbox with box == NULL. WHy so ?
> 
fts-api.h says:

/* Switch to updating the specified mailbox. box may also be set to NULL to
   make sure the previous mailbox won't tried to be accessed anymore. */
void fts_backend_update_set_mailbox(struct fts_backend_update_context *ctx,
struct mailbox *box);

So it's just telling you that you can close/free any stuff related to that 
mailbox.
>> additionally, my logic is that the backend stores one databalse per mailox 
>> in /xapian-indexes (in the "root" dir of the user), the name od the database 
>> is the GUID of the mailbox
>> 
>> For INBOX, that works perfectly, and database is properly createdm and 
>> backed starts indexing all emails
>> 
>> For other folder, somehow, the process can not access that (root) folder.
>> 
>> Am I missing something ?
>> 

This is a bit ambiguous, because some people mean mailbox=folder and others 
mean mailbox=user account, and GUID can also be the internal Dovecot folder 
GUID, or a GUID of the user.

I'd recommend using a single database per user anyway.



Re: Solr -> Xapian ?

2019-01-12 Thread Timo Sirainen
On 11 Jan 2019, at 21.23, Joan Moreau via dovecot  wrote:
> 
> The below patch resolves the compilation error
> 
> $ diff -p compat.h compat.h.joan 
> *** compat.h 2019-01-11 20:21:00.726625427 +0100
> --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100
> *** struct iovec;
> *** 202,207 
> --- 202,211 
> ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len);
> #endif
> 
> + #ifdef __cplusplus
> + extern "C" {
> + #endif
> 

You should put this extern "C" into the C++ file you're creating. See for 
example how fts-lucene/lucene-wrapper.cc does this.

> 1 - WHat does represent "subargs" in mail_search_args

It's set only for SEARCH_OR and SEARCH_SUB. So for example:

SEARCH TEXT foo TEXT bar TEXT baz

results in:

type=SEARCH_SUB
value.subargs = (
  { type=SEARCH, value.str="foo" },
  { type=SEARCH, value.str="bar" },
  { type=SEARCH, value.str="baz" },
)

Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other 
combination of OR/ANDs.
 
> 2 - for rescan : who is responsible for passing again the new email ? Is
> the Dovecot core sending again all the emails to index ? or the fts
> shall somehow access the mailbox and read all emails ? Wouldn't just be
> saying "delete all index and get_last_uid is now 0" the easy way ? or
> the fts must process all emails (and block the current thread as a
> mailbx maybe quite large)

The next indexing run is responsible for it. If you return get_last_uid=0, then 
indexer starts feeding you all mails. So fts backend doesn't have to know about 
it.

> 3 - for get_last_uid : this uncertainity is very unclear. "If there is a
> gap, then indexer first indexes all the missing" -> this mean at a
> certain point, indexer maybe rebuilding a previous email, so *last* uid
> is something different than max. And how indexer does know whther there
> is a gap wihtout callong the fts backend (whch it does not as there are
> no function for that) ?

I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 
have been indexed by the FTS backend. It's possible that at this point there 
are already mails with UIDs 101..200 in the folder. So when UID=201 is 
delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so 
far, and starts feeding it UIDs 101..201 in that order.

You can implement get_last_uid() simply by keeping track of it in 
dovecot.index* files, similar to how Lucene and Solr already do it with 
fts_index_get_header() / fts_index_set_header(). They also have a fallback that 
if the index doesn't have the last_uid value, they do a slower search from the 
Lucene/Solr index to find the last UID.



Re: Solr -> Xapian ?

2019-01-07 Thread Timo Sirainen
On 7 Jan 2019, at 16.05, Joan Moreau via dovecot  wrote:
> 
> Hi
> 
> ANyone to answer specifically ?
> 
> Q1 : get_last_uid -> Is this the last UID indexed (which may be not the 
> greatest value), or the gratest value (which may not be the latest) (the code 
> of existing plugins is unclear about this, Solr looks for the greatest for 
> insance)

All the mails are always supposed to be indexed from the beginning to the last 
indexed mail. If there's a gap, indexer first indexes all the missing mails. So 
the latest UID is supposed to be the greatest UID. (Supporting out-of-order 
indexing would be rather difficult to keep track of.)

> Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? 
> What is the link with "build_more" ?

The idea is that it calls something like:

 - build_key(type=hdr, hdr_name=From)
 - build_more("t...@iki.fi")
 - build_key(type=hdr, hdr_name=Subject)
 - build_more("Re: Solr -> Xapian ?")
 - build_key(type=body_part)
 - build_more("message body piece")
 - build_more("message body piece2")
 ...

> Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least 
> among "cc, to, from, subject, body") is not appearing in the 'struct' data. 
> WHere to find it ?

lookup() gets struct mail_search_arg *args, which contains the entire IMAP 
SEARCH query. This could be used for more or less complex query builders.

In case of a single header search, you should have args->args->hdr_field_name 
contain the header name and args->args->value.str contain the content you're 
searching for.

> Q4 : Refresh : this is very unclear. How come there would not be the "latest" 
> view on index. What is the real meaning of this function ?

In case of Xapian it might not matter if it automatically refreshes its indexes 
between each query. But with some other indexes this could happen:

 - IMAP session is opened
 - IMAP SEARCH is run, which opens and searches the index
 - a new mail is delivered to the mailbox and indexed
 - IMAP SEARCH is run. Without refresh() it doesn't see the newly indexed mail 
and doesn't include it in the search results.

> Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ?

It's run when "doveadm fts rescan" is run manually. Usually that's only run 
manually to fix up some brokenness. So it's intended to verify that the current 
mailbox contents match the FTS indexes:
 - If there are any mails in FTS index that no longer exist in the actual 
mailbox, delete those mails from FTS
 - If FTS is missing any mails in the middle of the mailbox, make sure that the 
next mailbox indexing will index those missing mails. I think currently this 
basically means reindexing all the mails since the first missing mail, even the 
mails that are already in the index.

fts-lucene implements this, but other FTS backends are lazy and simply rebuild 
all mails. Actually fts-solr is bad because it doesn't even delete the extra 
mails.

> Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?
>> and finally , for fts_backend__lookup_multi, why is that backend 
>> dependent ?

This function is called only when searching in virtual folders. So for example 
the virtual "All mails" folder, which would contain all mails in all folders. 
In that case the boxes[] would contain a list of user's all folders, except 
Trash and Spam. If lookup_multi() isn't implemented (left to NULL), the search 
is run separately via lookup() for each folder. With lookup_multi() there can 
be just one lookup, and the backend can filter only the wanted folders and 
return them directly. So it's an optimization for FTS indexes that support 
user-global searches rather than only per-folder searches.

>> static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend, 
>> struct mailbox *const boxes[], struct mail_search_arg *args, enum 
>> fts_lookup_flags flags, struct fts_multi_result *result)
>> {
>> struct xapian_fts_backend_update_context *ctx =
>> (struct xapian_fts_backend_update_context *)_ctx;
>> 
>> int i=0;
>> 
>> while(boxes[i]!=NULL)
>> {
>> if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0)
>>  return -1;
>> i++;
>> }
>> return 0;
>> }

See fts_backend_lookup_multi() - if you leave lookup_multi=NULL it basically 
does this.

>> For "rescan " and "optimize", wouldn't it be the dovecot core who indicate 
>> which are to be dismissed (expunged), or re-ask for indexing a particular 
>> (or all) uid ? WHy would the backend be aware of the transactions on the 
>> mailbox ???

rescan() is about fixing up a more or less broken index, or simply to verify 
that it's all ok. So core doesn't know what messages exist in the FTS index and 
can't request specific reindexing or expunging. I guess an alternative API 
could have been to have functions that iterate through all mails in the index, 
and use that to implement rescan in core. Now thinking about it, that sounds 
like a simpler and better way.


Re: gcc -> clang

2019-01-07 Thread Timo Sirainen
The unit test problems aren't clang issues, they're OSX issues. So:

test-net.c:79: Assert failed: strcmp(net_ip2addr(), "::5") == 0
test-net.c:83: Assert failed: strcmp(net_ip2addr(), "::5") == 0

This is because OSX writes the address out as "::0.0.0.5" instead of "::5". I 
don't remember if the core code should be fixed or if it's just the unit test 
that needs fixing.

test-lib(26058,0x7fff95cf7380) malloc: *** 
mach_vm_map(size=9223372036854775808) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
fatal_mempool_alloconly .. : ok
test-lib(26058,0x7fff95cf7380) malloc: *** 
mach_vm_map(size=9223372036854775808) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
fatal_mempool_allocfree .. : ok

I'm pretty sure these are intentional. The unit tests are succeeding, and the 
unit tests are there to test failing memory allocations. OSX just wants to also 
write some extra errors to stderr.

> On 3 Jan 2019, at 13.38, Rupert Gallagher via dovecot  
> wrote:
> 
> The compiler returns many warnings, and the test returns two IPv6-related 
> errors. I am attaching both logs as reference. 
> 
>> ‐‐‐ Original Message ‐‐‐
>> On Thursday, January 3, 2019 9:53 AM, Aki Tuomi  
>> wrote:
>> 
>>> We compile all core code with both gcc and clang. What sort of interesting 
>>> things did you find?
>>> 
>>> Aki
 On 03 January 2019 at 11:50 Rupert Gallagher via dovecot < 
 dovecot@dovecot.org > wrote:
 
 
 Please, use clang instead of gcc. Code quality can only profit from it. I 
 just compiled 2.3.4 and compiler stderr is full of interesting problems.
>>> 
>>> --- 
>>> Aki Tuomi
> 



Re: Bug in IDLE implementation for virtual mailbox

2018-12-17 Thread Timo Sirainen
On 17 Dec 2018, at 10.44, Pali Rohár  wrote:
> 
> On Sunday 16 December 2018 21:55:23 Timo Sirainen wrote:
>> On 16 Dec 2018, at 21.26, Pali Rohár  wrote:
>>> 
>>> Hello!
>>> 
>>> I found bug in Dovecot's IDLE implementation when virtual mailbox is in
>>> use. IDLE does not notify about new emails when email appears in newly
>>> created mailbox and IDLE was issued in virtual folder which matches "*"
>>> wildcard and that mailbox was created after opening virtual mailbox.
>> 
>> It actually has nothing to do with IDLE specifically. It's just that the 
>> virtual storage code doesn't try to look for new folders after the virtual 
>> mailbox is opened.
>> 
>>> To get notifications, it is needed to re-open that "All" mailbox again.
>> 
>> Right. I don't think this is going to be fixed anytime soon. Quite a lot of 
>> effort and it can be worked around.
> 
> How to workaround it? Imap clients uses either IDLE or STATUS or LIST
> (with STATUS) commands for checking if there is a new messages.
> 
> But none of these commands reports existence of new message until that
> virtual folder is re-opened.

I mean, workaround is for the user to just reopen the folder. I think it's not 
very common for this situation to happen and cause problems?



Re: Bug in IDLE implementation for virtual mailbox

2018-12-16 Thread Timo Sirainen
On 16 Dec 2018, at 21.26, Pali Rohár  wrote:
> 
> Hello!
> 
> I found bug in Dovecot's IDLE implementation when virtual mailbox is in
> use. IDLE does not notify about new emails when email appears in newly
> created mailbox and IDLE was issued in virtual folder which matches "*"
> wildcard and that mailbox was created after opening virtual mailbox.

It actually has nothing to do with IDLE specifically. It's just that the 
virtual storage code doesn't try to look for new folders after the virtual 
mailbox is opened.

> To get notifications, it is needed to re-open that "All" mailbox again.

Right. I don't think this is going to be fixed anytime soon. Quite a lot of 
effort and it can be worked around.



Re: Panic…

2018-12-12 Thread Timo Sirainen
On 13 Dec 2018, at 7.31, SH Development  wrote:
> 
> I have started getting these in my log.  What does this mean and what do I 
> need to do?
> 
>  Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): 
> assertion failed: (offset < 0x4000)

Your dovecot.index.cache file has grown too huge. The only solution now is to 
delete it, and perhaps try to shrink the number of mails in the folder as well. 
The downside to deleting cache is that it may temporarily slow down performance 
for accessing the folder, depending on the IMAP client.



Re: Indexer worker small bug

2018-12-09 Thread Timo Sirainen
On 10 Dec 2018, at 7.50, André Rodier  wrote:
> 
> On 2018-12-09 23:13, Timo Sirainen wrote:
>> On 9 Dec 2018, at 16.44, André Rodier via dovecot  
>> wrote:
>>> Hello,
>>> I think I submitted this before, but I am not sure this has been addressed
>>> I am using AppArmor with Dovecot, without any issue.
>>> However, I think there is a bug in the indexer working, from what I can 
>>> see, a missing trailing slash. See:
>>> 
>>> Dec 09 14:35:53 portal2 kernel: audit: type=1400 
>>> audit(1544366153.379:3035): apparmor="DENIED" operation="file_mmap" 
>>> info="Failed name lookup - disconnected path" error=-13 
>>> profile="/usr/lib/dovecot/indexer-worker" name="var/cache/nscd/hosts" 
>>> pid=10540 comm="indexer-worker" requested_mask="r" denied_mask="r" 
>>> fsuid=1001 ouid=0
>>> 
>>> The indexer worker is trying to open the file "var/cache/nscd/hosts" 
>>> instead of "/var/cache/nscd/hosts", which of course fails.
>>> Can someone double check the code of the indexer worker, or this has been 
>>> fixed?
>> Dovecot is definitely not trying to open that file itself. It has to
>> be libc or some other library. I also can't think of anything special
>> in indexer-worker compared to other Dovecot binaries that could cause
>> this. What's your doveconf -n?
> 
> You are probably right, I will continue to investigate on my side.
> My configuration is attached.

Does it make a difference if you use "127.0.0.1" instead of "localhost" in 
fts_solr setting?



Re: Indexer worker small bug

2018-12-09 Thread Timo Sirainen
On 9 Dec 2018, at 16.44, André Rodier via dovecot  wrote:
> 
> Hello,
> 
> I think I submitted this before, but I am not sure this has been addressed
> 
> I am using AppArmor with Dovecot, without any issue.
> 
> However, I think there is a bug in the indexer working, from what I can see, 
> a missing trailing slash. See:
> 
> 
> Dec 09 14:35:53 portal2 kernel: audit: type=1400 audit(1544366153.379:3035): 
> apparmor="DENIED" operation="file_mmap" info="Failed name lookup - 
> disconnected path" error=-13 profile="/usr/lib/dovecot/indexer-worker" 
> name="var/cache/nscd/hosts" pid=10540 comm="indexer-worker" 
> requested_mask="r" denied_mask="r" fsuid=1001 ouid=0
> 
> 
> The indexer worker is trying to open the file "var/cache/nscd/hosts" instead 
> of "/var/cache/nscd/hosts", which of course fails.
> 
> Can someone double check the code of the indexer worker, or this has been 
> fixed?

Dovecot is definitely not trying to open that file itself. It has to be libc or 
some other library. I also can't think of anything special in indexer-worker 
compared to other Dovecot binaries that could cause this. What's your doveconf 
-n?



Re: Dovecot 2.3.4 crash

2018-12-03 Thread Timo Sirainen
On 2 Dec 2018, at 22.22, Guillaume via dovecot  wrote:
> 
> I also have this kind of segfault since the update :
> 
> Dec  2 21:12:11 xxx dovecot: auth-worker: Error: *** Error in 
> `dovecot/auth': double free or corruption (fasttop): 0x55573bb99f70

Is this easy to reproduce? Can you try with valgrind? It will slow down the 
logins a bit though.

service auth-worker {
  executable = /usr/bin/valgrind -q --vgdb=no /usr/lib/dovecot/auth -w
}

At least one good thing about this crash is that it seems to happen only at 
deinit, so it's not impacting users.



Re: How to unsubscribe

2018-12-01 Thread Timo Sirainen
On 1 Dec 2018, at 12.37, Keith Edmunds  wrote:
> 
>> I am very sorry that I had to enable the delivery for people who had
>> intentionally disabled delivery, but I didn't want to spend days
>> figuring out the best possible way to fix the situation and in the end
>> it's not a horrible bother to disable yourself again
> 
> 
> I have disabled mail delivery in Mailman.
> 
> I'm still receiving the list mails.

You and at least another person I just looked at was subscribed via a different 
email address. You actually had 2 addresses from this domain. I disabled the 
other one now as well.



Re: Dovecot 2.3.4 crash

2018-11-29 Thread Timo Sirainen
On 29 Nov 2018, at 15.46, azu...@pobox.sk wrote:
> 
> Hi,
> 
> is this a know problem? Newest Dovecot 2.3.4 package from repo.dovecot.org , 
> Debian Stretch (fully upgraded).
> 
> 
> 
> Nov 29 14:25:11 server00 dovecot: lmtp(16854): Panic: file ostream-dot.c: 
> line 208 (o_stream_dot_sendv): assertion failed: ((size_t)ret == sent + added)

Is this proxying LMTP traffic?

> dovecot/lmtp [local DATA](_start+0x2a) [0x565287c310ca]

Based on this I was thinking it wouldn't be. But this apparently happens only 
with sending SMTP/LMTP traffic with SSL. So do you think it's SSL-related with 
you also? https://www.dovecot.org/pipermail/dovecot/2018-November/113532.html 




Re: [2.3.4] Segmentation faults

2018-11-28 Thread Timo Sirainen
See https://dovecot.org/bugreport.html#coredumps

Without a backtrace it's not really possible to figure out where it's crashing.

> On 28 Nov 2018, at 13.20, Joan Moreau  wrote:
> 
> Where to get that ?
> 
>  
> On 2018-11-27 08:50, Aki Tuomi wrote:
> 
>> It's still missing core dump (or bt full from it)
>> 
>> Aki
>> 
>> On 27.11.2018 8.39, Joan Moreau wrote:
>>> Thank you Aki
>>> 
>>> here the requested data (below)
>>> 
>>> Please not as well that we have numerous subfolders (>50) and pretty big 
>>> mailbox sizes (>20G)
>>> 
>>> Bug appears mostly in auth process and index-worker
>>> 
>>> 
>>> 
>>> dovecot -n :
>>> 
>>> # 2.4.devel (de42b54aa): /etc/dovecot/dovecot.conf
>>> # Pigeonhole version 0.6.devel (65909cfa)
>>> # OS: Linux 4.19.4-arch1-1-ARCH x86_64 ext4
>>> # Hostname: gjserver
>>> base_dir = /run/dovecot
>>> default_login_user = dovecot
>>> default_vsz_limit = 16 G
>>> disable_plaintext_auth = no
>>> listen = *
>>> log_path = /var/log/mail/dovecot.log
>>> mail_gid = mail
>>> mail_location = mdbox:/data/mail/%d/%n:ALT=/data/mail/archives/%d/%n
>>> mail_plugins = fts fts_squat
>>> mail_uid = mailusers
>>> managesieve_notify_capability = mailto
>>> managesieve_sieve_capability = fileinto reject envelope encoded-character 
>>> vacation subaddress comparator-i;ascii-numeric relational regex imap4flags 
>>> copy include variables body enotify environment mailbox date index ihave 
>>> duplicate mime foreverypart extracttext
>>> mdbox_rotate_size = 24 M
>>> 
>>> (...)
>>> 
>>> passdb {
>>> args = /etc/dovecot/dovecot-sql.conf
>>> driver = sql
>>> }
>>> (the rest default values)
>>> 
>>>  
>>> 
>>> 
>>> On 2018-11-25 08:08, Aki Tuomi wrote:
>>> 
>>>  
>>> On 25 November 2018 at 06:29 Joan Moreau < j...@grosjo.net 
>>> > wrote:
>>>  
>>>  
>>> Hi
>>>  
>>> THis is the lines I have in my dmesg (see below)
>>>  
>>> In dovecot log , I see:
>>>  
>>> Nov 25 04:26:47 auth-worker: Error: double free or corruption (fasttop)
>>>  
>>> What do to about it ?
>>>  
>>> Using lastest 2.3.4 version
>>>  
>>> Thank you
>>>  
>>> 
>>>  
>>> [132932.169265] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00
>>> 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 aa 36 f5 7a 7f 00
>>> 00 <40> 24 3d f5 7a 7f 00 00 21 80 00 00 00 00 00 00 00 94 16 d6 ee 55
>>> [134031.969596] auth[27031]: segfault at 55e509612c30 ip
>>> 55e509612c30 sp 7ffeb96dee48 error 15
>>> [134031.969603] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00
>>> 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 3a d3 e4 ef 7f 00
>>> 00 <40> b4 d9 e4 ef 7f 00 00 21 80 00 00 00 00 00 00 00 04 5f 09 e5 55
>>> [134081.497871] doveadm[28930]: segfault at 7ffe4a16efc8 ip
>>> 7f393841013e sp 7ffe4a16efc0 error 6 in
>>> libdovecot.so.0.0.0[7f3938363000+e2000]
>>> [134081.497876] Code: 7d be e9 68 ff ff ff e8 10 4c f5 ff 41 57 41 89 cf
>>> 41 56 49 89 f6 41 55 41 89 fd 31 ff 41 54 55 44 89 c5 53 89 d3 48 83 ec
>>> 58 <4c> 89 4c 24 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 48 31 c0 e8
>>> [134084.145731] doveadm[29186]: segfault at 7fff1cfdbff8 ip
>>> 7f4376e32ffb sp 7fff1cfdc000 error 6 in
>>> libdovecot.so.0.0.0[7f4376d86000+e2000]
>>> [134084.145735] Code: ff 66 0f 1f 44 00 00 e9 9d dc f6 ff 0f 1f 00 48 83
>>> ec 08 48 83 3d 14 16 0b 00 00 0f 85 d2 76 f6 ff 31 f6 48 8d 3d 05 16 0b
>>> 00  00 54 f5 ff 85 c0 0f 88 e4 76 f6 ff 48 83 c4 08 e9 69 dc f6 ff
>>> [135453.211242] indexer-worker[2539]: segfault at 7ffec3ba4ff8 ip
>>> 7ffec43fdcff sp 7ffec3ba5000 error 6
>>> [135453.211245] Code: 95 4c 89 f7 48 89 75 d0 e8 5e fc ff ff 48 8b 75 d0
>>> e9 56 ff ff ff 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 49 89
>>> f4 <53> 48 83 ec 08 48 85 ff 0f 84 b3 00 00 00 48 89 fb 4c 8d 35 69 c3
>>> [135453.730250] indexer-worker[9236]: segfault at 7fffed921ff8 ip
>>> 7fb7c9c4f5b1 sp 7fffed922000 error 6 in
>>> libdovecot.so.0.0.0[7fb7c9ba2000+e2000]
>>> [135453.730256] Code: 2e 0f 1f 84 00 00 00 00 00 41 57 4d 89 cf 41 56 41
>>> 89 fe 41 55 49 89 f5 41 54 41 89 d4 55 89 cd 53 48 83 ec 08 4c 8b 4c 24
>>> 40  6a fb ff ff 85 c0 7e 4f 48 8b 05 7f f9 0a 00 be 38 00 00 00 48
>>> [135796.171575] auth[11121]: segfault at 555f8645cc30 ip
>>> 555f8645cc30 sp 7ffcbb510868 error 15
>>> [135796.171586] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00
>>> 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 ba ed 38 b7 7f 00
>>> 00 <40> 34 f4 38 b7 7f 00 00 21 80 00 00 00 00 00 00 00 a4 43 86 5f 55
>>> [136710.562003] auth[17828]: segfault at 563443604c30 ip
>>> 563443604c30 sp 7ffc1aa8b498 error 15
>>> [136710.562013] Code: 00 00 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00
>>> 00 21 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 20 fa 48 da d5 7f 00
>>> 00 <40> 74 4f da d5 7f 00 00 21 80 00 00 00 00 00 00 00 24 5e 43 34 56
>>> [138331.686718] auth[31046]: segfault at 55b27bc63c30 ip
>>> 55b27bc63c30 sp 7ffd5d5b9298 error 15
>>> [138331.686721] Code: 00 00 00 00 

Re: Event 0x2b1a5f270bd0 leaked (parent=(nil)): auth-client-connection.c:338

2018-11-26 Thread Timo Sirainen
On 26 Nov 2018, at 13.16, Mart Pirita  wrote:
> 
> Hi,
> 
> Auth process is not constantly being shutdown and/or restarted and Dovecot is 
> used for SMTP authentication (Postfix).
> 
> Checked few servers logs, they are running v2.3.3, for example latest
> (some day none, some day a lot, some day few) logs:
> 
> Nov 25 18:48:11 server1 dovecot: auth: Warning: Event 0x2b79250f15f0
> leaked (parent=(nil)): auth-client-connection.c:338

I suppose these are happening because of:

> dovecot: auth: Warning: auth client 0 disconnected with 1 pending
> requests: EOF: 12 Time(s)

Which probably happens when Postfix disconnects from Dovecot before the 
authentication has finished.

I can reproduce these if I set up PAM authentication and then do:

doveadm auth test testuser wrongpass


Repeat the above a few times. Each time logs:

Nov 26 13:36:13.588354 auth: Warning: auth client 0 disconnected with 1 pending 
requests: EOF

Then stop Dovecot (or auth process at least):

Nov 26 13:36:23.403778 auth: Warning: Event 0x561565277db0 leaked 
(parent=(nil)): auth-client-connection.c:338



Re: Event 0x2b1a5f270bd0 leaked (parent=(nil)): auth-client-connection.c:338

2018-11-26 Thread Timo Sirainen
On 3 Nov 2018, at 17.41, Mart Pirita  wrote:
> 
> Hi,
> 
> 
> But this harmless is spamming logs, so how to disable it:
> 
> grep auth-client-connection.c:338 maillog | wc -l
>1259

Actually this specific event leak isn't a known issue. I don't really 
understand how it could happen. These event leaks are supposed to be checked 
only at process deinit. Is the auth process constantly being shutdown and 
restarted? What's your doveconf -n? Are you using Dovecot for SMTP 
authentication or some other external auth?


> Aki Tuomi wrote:
>>> On 03 November 2018 at 12:12 Mart Pirita < sysad...@e-positive.ee 
>>>  
>>> >> wrote:
>>> 
>>> 
>>> Hi,
>>> 
>>> 
>>> Noticed with latest v2.3.3 some new warning in logs, for example:
>>> 
>>> dovecot: auth: Warning: Event 0x80a6fc0 leaked (parent=(nil)):
>>> auth-client-connection.c:338: 1 Time(s)
>>> dovecot: auth: Warning: Event 0x80aa1c8 leaked (parent=(nil)):
>>> auth-client-connection.c:338: 1 Time(s)
>>> dovecot: auth: Warning: Event 0x80aa718 leaked (parent=(nil)):
>>> auth-client-connection.c:338: 1 Time(s)
>>> dovecot: auth: Warning: Event 0x80adac0 leaked (parent=(nil)):
>>> auth-client-connection.c:338: 1 Time(s)
>>> dovecot: auth: Warning: Event 0x80b6c38 leaked (parent=(nil)):
>>> auth-client-connection.c:338: 1 Time(s)
>>> dovecot: auth: Warning: Event 0x80c0e00 leaked (parent=(nil)):
>>> auth-client-connection.c:338: 1 Time(s)
>>> dovecot: auth: Warning: auth client 0 disconnected with 1 pending
>>> requests: EOF: 12 Time(s)
>>> 
>>> 
>>> What are they?
>>> 
>>> 
>>> -- 
>>> Mart
>> 
>> Hi! It's harmless event leak. This is a known issue to us.
>> ---
>> Aki Tuomi
> 
> 
> -- 
> Mart



Re: v2.3.4 released

2018-11-24 Thread Timo Sirainen
On 24 Nov 2018, at 8.33, Odhiambo Washington  wrote:
> 
> 
> I installed 2.3.4 and just used it with the config files for 2.3.3 without 
> changing anything in the configuration.
> I then realized that the LDA was throwing errors.
> 
> 2018-11-24 00:02:51 1gQIaw-000AZS-Bc 
>  >: dovecot_virtual_delivery transport 
> output: lda(john@our.domain.name )Error: 
> net_connect_unix(/var/run/dovecot//stats-writer) failed: Permission denied
> 
> I checked on the presence of the sockets in /var/run/dovecot:
> 
> srw---   1 root  wheel0 Nov 24 09:07 stats-reader
> srw-rw   1 root  dovecot  0 Nov 24 09:07 stats-writer

What user/group does dovecot_virtual_delivery run as? Change the stats-writer 
socket's owner to be that user. For example:

service stats {
  unix_listener stats-writer {
  user = vmail
  }
}

Or alternatively change dovecot_virtual_delivery to use dovecot group.

> I have tried to find any mention of stats-{writer|reader} in the example 
> configs shipped with 2.3.4 and found nothing. I have backed-off 2.3.4 for now 
> till I can figure out how to assign proper permissions to these sockets - or 
> just to figure out why by default, permission is being denied.

Looks like this is happening now because in earlier versions the dovecot-lda 
process wasn't sending any statistics.



Re: Problems after upgrading to 2.3.2.1

2018-11-23 Thread Timo Sirainen
On 22 Nov 2018, at 22.28, Joel Dahl  wrote:
> 
> Hi,
> 
> I've just upgraded dovecot from 2.2.35 to 2.3.2.1 on my mail server
> and noticed something odd. My mail clients (mutt and Apple mail) both stopped
> seeing new mail in my mailboxes.
> 
> For example, ~/Maildir/mailinglist1/new has 5 new mails. In mutt, vieweing the
> list of mailboxes mutt reports 0 new mail for "mailinglist1". Opening up the
> actual mailbox in mutt reveals the 5 new mails. Going back to the list of
> mailboxes, mutt now correctly reports 5 unread mails. Waiting a few hours, and
> about 10 new mails have arrived, but mutt still says 5 unread.
> 
> This is the same for every mailbox. As a test I deleted the index files for a
> specific mailbox and restarted dovecot. But it didn't help. I use procmail to
> deliver my mail into the Maildir folders, if that matters.

See if mailbox_list_index=no helps? Or switch to delivering via dovecot-lda 
(just tell procmail to deliver using it).



[Dovecot-news] v2.3.4 released

2018-11-23 Thread Timo Sirainen
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig
Binary packages in https://repo.dovecot.org/

 * The default postmaster_address is now "postmaster@". If username contains the @domain part, that's
   used. If not, then the server's hostname is used.
 * "doveadm stats dump" now returns two decimals for the "avg" field.

 + Added push notification driver that uses a Lua script
 + Added new SQL, DNS and connection events.
   See https://wiki2.dovecot.org/Events
 + Added "doveadm mailbox cache purge" command.
 + Added events API support for Lua scripts
 + doveadm force-resync -f parameter performs "index fsck" while opening
   the index. This may be useful to fix some types of broken index files.
   This may become the default behavior in a later version.
 - director: Kicking a user crashes if login process is very slow
 - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages
   unless QUIT is sent.
 - auth: Fix crypt() segfault with glibc-2.28+
 - imap: Running UID FILTER script with errors assert-crashes
 - dsync, pop3-migration: POP3 UIDLs weren't added to
   dovecot.index.cache while mails were saved.
 - dict clients may have been using 100% CPU while waiting for dict
   server to finish commands.
 - doveadm user: Fixed user listing via HTTP API
 - All levels of Cassandra log messages were logged as Dovecot errors.
 - http/smtp client may have crashed after SSL handshake
 - Lua auth converted strings that looked like numbers into numbers.

___
Dovecot-news mailing list
Dovecot-news@dovecot.org
https://dovecot.org/mailman/listinfo/dovecot-news


v2.3.4 released

2018-11-23 Thread Timo Sirainen
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.4.tar.gz.sig
Binary packages in https://repo.dovecot.org/

 * The default postmaster_address is now "postmaster@". If username contains the @domain part, that's
   used. If not, then the server's hostname is used.
 * "doveadm stats dump" now returns two decimals for the "avg" field.

 + Added push notification driver that uses a Lua script
 + Added new SQL, DNS and connection events.
   See https://wiki2.dovecot.org/Events
 + Added "doveadm mailbox cache purge" command.
 + Added events API support for Lua scripts
 + doveadm force-resync -f parameter performs "index fsck" while opening
   the index. This may be useful to fix some types of broken index files.
   This may become the default behavior in a later version.
 - director: Kicking a user crashes if login process is very slow
 - pop3_no_flag_updates=no: Don't expunge DELEted and RETRed messages
   unless QUIT is sent.
 - auth: Fix crypt() segfault with glibc-2.28+
 - imap: Running UID FILTER script with errors assert-crashes
 - dsync, pop3-migration: POP3 UIDLs weren't added to
   dovecot.index.cache while mails were saved.
 - dict clients may have been using 100% CPU while waiting for dict
   server to finish commands.
 - doveadm user: Fixed user listing via HTTP API
 - All levels of Cassandra log messages were logged as Dovecot errors.
 - http/smtp client may have crashed after SSL handshake
 - Lua auth converted strings that looked like numbers into numbers.



Re: Panic: file istream-crlf.c: line 24 (i_stream_crlf_read_common): assertion failed: (ret != -2)

2018-10-30 Thread Timo Sirainen
On 30 Oct 2018, at 16.25, Peter Nabbefeld  wrote:
> Oct 26 07:41:20 lda(peter.nabbef...@gmx.de)<24607>: 
> Panic: file istream-crlf.c: line 24 (i_stream_crlf_read_common): assertion 
> failed: (ret != -2)
> 
> $ dovecot -n
> # 2.3.3 (dcead646b): /etc/dovecot/dovecot.conf

Are you using our packages from https://repo.dovecot.org/? 
 If so, it would be helpful if you could send the 
core dump to us (e.g. email privately to me). As for getting core dumps, see: 
https://dovecot.org/bugreport.html#coredumps



Re: VOLATILEDIR not really used?

2018-10-05 Thread Timo Sirainen
On 5 Oct 2018, at 16.42, Tom Sommer mailto:m...@tomsommer.dk>> wrote:
> 
> On 2018-10-05 11:50, Tom Sommer wrote:
>> On 2018-10-05 11:35, Timo Sirainen wrote:
>>> On 4 Oct 2018, at 17.13, Tom Sommer >> <mailto:m...@tomsommer.dk>> wrote:
>>>> On 2018-10-04 15:55, Timo Sirainen wrote:
>>>>> On 4 Oct 2018, at 14.39, Tom Sommer >>>> <mailto:m...@tomsommer.dk>> wrote:
>>>>>> Is this correct, and if so are there any plans to move dotlocks etc. to 
>>>>>> this directory?
>>>>> What dotlocks? I guess mbox and Maildir have some locks that could be
>>>>> moved there, but a better performance optimization for those
>>>>> installations would be to switch to sdbox/mdbox. Other than that, I
>>>>> don't remember there being anything important that could be moved
>>>>> there.
>>>> Maildir locks yes, switching format is not a procedure I feel comfortable 
>>>> with :)
>>>> Would it be possible to move the maildir/mbox locks to VOLATILEDIR?
>>> Sure it would be possible, but it's such a low priority for us that I
>>> doubt we'll be implementing it.
>> Might try and look into it myself.
> 
> Something like this? See attached.

Looks like it works. But could you:

1) Add it to github as merge request

2) MAILBOX_LIST_PATH_TYPE_VOLATILEDIR might be useful, but it's kind of a 
separate change of its own. To simplify, in Maildir code you could have 
box->list->set->volatile_dir != NULL ? box->list->set->volatile_dir : 
control_dir. That way it's falling back to control_dir like the previous code 
when VOLATILEDIR wasn't set.



Re: VOLATILEDIR not really used?

2018-10-05 Thread Timo Sirainen
On 4 Oct 2018, at 17.13, Tom Sommer  wrote:
> 
> 
> On 2018-10-04 15:55, Timo Sirainen wrote:
>> On 4 Oct 2018, at 14.39, Tom Sommer  wrote:
> 
>>> Is this correct, and if so are there any plans to move dotlocks etc. to 
>>> this directory?
>> What dotlocks? I guess mbox and Maildir have some locks that could be
>> moved there, but a better performance optimization for those
>> installations would be to switch to sdbox/mdbox. Other than that, I
>> don't remember there being anything important that could be moved
>> there.
> 
> Maildir locks yes, switching format is not a procedure I feel comfortable 
> with :)
> 
> Would it be possible to move the maildir/mbox locks to VOLATILEDIR?

Sure it would be possible, but it's such a low priority for us that I doubt 
we'll be implementing it.



Re: VOLATILEDIR not really used?

2018-10-04 Thread Timo Sirainen
On 4 Oct 2018, at 14.39, Tom Sommer  wrote:
> 
> Hi
> 
> According to the docs, setting VOLATILEDIR will improve I/O performance when 
> using NFS - but as far as I can see, only vsize lock-files are put here, and 
> little else?

Right. I think also autoexpunge locking.

> Is this correct, and if so are there any plans to move dotlocks etc. to this 
> directory?


What dotlocks? I guess mbox and Maildir have some locks that could be moved 
there, but a better performance optimization for those installations would be 
to switch to sdbox/mdbox. Other than that, I don't remember there being 
anything important that could be moved there.



Re: Bug reports - auth is broken in Dovecot 2.3.3

2018-10-03 Thread Timo Sirainen
On 3 Oct 2018, at 14.09, Berindeie Avram-Teodor  wrote:
> 
> 
> 
> I do not have downloaded the source from GitHub.

The patch modifies configure.ac, so unless you run autogen.sh the configure 
script isn't modified and that patch doesn't work. Or as an alternative you 
could simply manually append to config.h:

#define HAVE_CRYPT_H

> 
> On Wed, Oct 3, 2018 at 1:53 PM Timo Sirainen  <mailto:t...@iki.fi>> wrote:
> On 3 Oct 2018, at 13.22, Berindeie Avram-Teodor  <mailto:berindeie@gmail.com>> wrote:
> > 
> > I applied the patch and recompiled but nothing resolved.
> > What else can I do?
> 
> You also need to run autogen.sh & configure again.
> 



Re: Bug reports - auth is broken in Dovecot 2.3.3

2018-10-03 Thread Timo Sirainen
On 3 Oct 2018, at 13.22, Berindeie Avram-Teodor  wrote:
> 
> I applied the patch and recompiled but nothing resolved.
> What else can I do?

You also need to run autogen.sh & configure again.



Re: Delete vs. Expunge in Public Namespace

2018-10-03 Thread Timo Sirainen
On 3 Oct 2018, at 9.03, Chris  wrote:
> 
> On Tue, 2 Oct 2018 18:04:21 +0200
> Chris wrote:
> 
>> Is this because the mailbox is
>> part of public namespace?
> 
> Or are they expunged because they're duplicates? Is there any check?
> Based on Message ID?

No.. None of this makes sense. Dovecot doesn't auto-expunge \Deleted mails in 
any situation.

I see your tool says:

parser.add_option("",   "--no-close",  dest='no_close', action="store_true",
help='Do not "close" mailbox when done. Some servers 
will purge deleted messages on a close command.')

If it's sending CLOSE, that explains it. It's not just "some servers", it's all 
IMAP servers. There's UNSELECT command to close without expunging.



Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Timo Sirainen
On 2 Oct 2018, at 22.52, Helmut K. C. Tessarek  wrote:
> 
> On 2018-10-01 04:07, Timo Sirainen wrote:
>> 
>> https://wiki2.dovecot.org/Plugins/Listescape maybe?
> 
> It should be mentioned somehow that one can't just change the hierarchy
> separator on the fly (without manual changes to the fs).
> 
> If you used . as the separator, it would look this in the filesystem:
> .testfolder.sub1
> Now you change the separator to $.
> Your mail client will see the existing folders as .testfolder.sub1
> instead of:
> 
> testfolder
>   sub1

No, that's not how the listescape plugin works. You can't change the filesystem 
separator with or without listescape plugin. It's always "." with Maildir++. 
You can change the namespace's visible separator to e.g. "$" which allows you 
to start using "." in the folder names with listescape. But using "." will be 
encoded as \2e (or something) in the filesystem.

Changing the namespace separator can have other problems though. Clients 
generally don't like it much, and may redownload all mails in subfolders if 
it's changed. Some clients may become even more confused.



signature.asc
Description: Message signed with OpenPGP


Re: Help for UID THREAD and SORT optimization

2018-10-02 Thread Timo Sirainen
On 2 Oct 2018, at 12.22, Alessio Cecchi  wrote:
> 
> Hello,
> 
> we are developing a library to show last arrived messages of all threads in a 
> folder with more than 300k messages.
> 
> As per the RFC 5256, the IMAP thread command has only the option to specify 
> the grouping criteria (REFERENCES vs ORDEREDSUBJECT). So we implemented an 
> algorithm which gets the full UID SORT ordered by date in reverse order then 
> gets all thread relations and then post process all outputs to obtain the 
> final result.
> 
It sounds like what you want is REFS threading, which Dovecot supports, but it 
didn't make it into official RFC: 
https://tools.ietf.org/html/draft-gulbrandsen-imap-inthread-05 



Re: 2.3.3: Panic: file ostream-zlib.c: line 37 (o_stream_zlib_close): assertion failed

2018-10-02 Thread Timo Sirainen
On 2 Oct 2018, at 12.26, Tom Sommer  wrote:
> 
> I see this in my logs after 2.3.3:
> 
> using zlib plugin, ofc.
> 
> Oct 02 10:01:39 imap(u...@example.com)<50643>: Panic: file 
> ostream-zlib.c: line 37 (o_stream_zlib_close): assertion failed: 
> (zstream->ostream.finished || zstream->ostream.ostream.stream_errno != 0 || 
> zstream->ostream.error_handling_disabled)
> Oct 02 10:01:39 imap(u...@example.com)<50643>: Error: Raw 
> backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xce56a) [0x7f442487556a] -> 
> /usr/lib64/dovecot/libdovecot.so.0(+0xce5b1) [0x7f44248755b1] -> 
> /usr/lib64/dovecot/libdovecot.so.0(+0x3d941) [0x7f44247e4941] -> 
> /usr/lib64/dovecot/lib20_zlib_plugin.so(+0x5cdf) [0x7f44233c4cdf] -> 
> /usr/lib64/dovecot/libdovecot.so.0(+0xf4b46) [0x7f442489bb46] -> 
> /usr/lib64/dovecot/libdovecot.so.0(o_stream_destroy+0x13) [0x7f442489be83] -> 
> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_save_finish+0x173) 
> [0x7f4424b93973] -> 
> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_save_cancel+0x36) 
> [0x7f4424b68696] -> dovecot/imap [u...@example.com X.X.X.X APPEND](+0xede9) 
> [0x55a9d7e86de9] -> dovecot/imap [u...@example.com X.X.X.X 
> APPEND](command_exec+0x65) [0x55a9d7e956e5] -> dovecot/imap [u...@example.com 
> X.X.X.X APPEND](+0xf9e6) [0x55a9d7e879e6] -> 
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x55) [0x7f442488c275] -> 
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbf) 
> [0x7f442488e13f] -> 
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x55) [0x7f442488c365] 
> -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f442488c588] -> 
> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4424808053] 
> -> dovecot/imap [u...@example.com X.X.X.X APPEND](main+0x32d) 
> [0x55a9d7ea20dd] -> /lib64/libc.so.6(__libc_start_main+0x100) 
> [0x7f4424431d20] -> dovecot/imap [u...@example.com X.X.X.X APPEND](+0xe419) 
> [0x55a9d7e86419]
> 
> No clue how to reproduce

It crashes if client disconnects during APPEND, but only with Maildir and zlib 
plugin. The bug already existed in earlier v2.3.x releases though.



[Dovecot-news] v2.3.3 released

2018-10-01 Thread Timo Sirainen
https://dovecot.org/releases/2.3/dovecot-2.3.3.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.3.tar.gz.sig

 * doveconf hides more secrets now in the default output.
 * ssl_dh setting is no longer enforced at startup. If it's not set and
   non-ECC DH key exchange happens, error is logged and client is
   disconnected.

 + Added log_debug= setting.
 + Added log_core_filter= setting.
 + quota-clone: Write to dict asynchronously
 + --enable-hardening attempts to use retpoline Spectre 2 mitigations
 + lmtp proxy: Support source_ip passdb extra field.
 + doveadm stats dump: Support more fields and output stddev by default.
 + push-notification: Add SSL support for OX backend.
 - NUL bytes in mail headers can cause truncated replies when fetched.
 - director: Conflicting host up/down state changes may in some rare
   situations ended up in a loop of two directors constantly overwriting
   each others' changes.
 - director: Fix hang/crash when multiple doveadm commands are being
   handled concurrently.
 - director: Fix assert-crash if doveadm disconnects too early
 - virtual plugin: Some searches used 100% CPU for many seconds
 - dsync assert-crashed with acl plugin in some situations.
 - mail_attachment_detection_options=add-flags-on-save assert-crashed
   with some specific Sieve scripts.
 - Mail snippet generation crashed with mails containing invalid
   Content-Type:multipart header.
 - Log prefix ordering was different for some log lines.
 - quota: With noenforcing option current quota usage wasn't updated.
 - auth: Kerberos authentication against Samba assert-crashed.
 - stats clients were unnecessarily chatty with the stats server.
 - imapc: Fixed various assert-crashes when reconnecting to server.
 - lmtp, submission: Fix potential crash if client disconnects while
   handling a command.
 - quota: Fixed compiling with glibc-2.26 / support libtirpc.
 - fts-solr: Empty search values resulted in 400 Bad Request errors
 - fts-solr: default_ns parameter couldn't be used
 - submission server crashed if relay server returned over 7 lines in
   a reply (e.g. to EHLO)

___
Dovecot-news mailing list
Dovecot-news@dovecot.org
https://dovecot.org/mailman/listinfo/dovecot-news


v2.3.3 released

2018-10-01 Thread Timo Sirainen
https://dovecot.org/releases/2.3/dovecot-2.3.3.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.3.tar.gz.sig

 * doveconf hides more secrets now in the default output.
 * ssl_dh setting is no longer enforced at startup. If it's not set and
   non-ECC DH key exchange happens, error is logged and client is
   disconnected.

 + Added log_debug= setting.
 + Added log_core_filter= setting.
 + quota-clone: Write to dict asynchronously
 + --enable-hardening attempts to use retpoline Spectre 2 mitigations
 + lmtp proxy: Support source_ip passdb extra field.
 + doveadm stats dump: Support more fields and output stddev by default.
 + push-notification: Add SSL support for OX backend.
 - NUL bytes in mail headers can cause truncated replies when fetched.
 - director: Conflicting host up/down state changes may in some rare
   situations ended up in a loop of two directors constantly overwriting
   each others' changes.
 - director: Fix hang/crash when multiple doveadm commands are being
   handled concurrently.
 - director: Fix assert-crash if doveadm disconnects too early
 - virtual plugin: Some searches used 100% CPU for many seconds
 - dsync assert-crashed with acl plugin in some situations.
 - mail_attachment_detection_options=add-flags-on-save assert-crashed
   with some specific Sieve scripts.
 - Mail snippet generation crashed with mails containing invalid
   Content-Type:multipart header.
 - Log prefix ordering was different for some log lines.
 - quota: With noenforcing option current quota usage wasn't updated.
 - auth: Kerberos authentication against Samba assert-crashed.
 - stats clients were unnecessarily chatty with the stats server.
 - imapc: Fixed various assert-crashes when reconnecting to server.
 - lmtp, submission: Fix potential crash if client disconnects while
   handling a command.
 - quota: Fixed compiling with glibc-2.26 / support libtirpc.
 - fts-solr: Empty search values resulted in 400 Bad Request errors
 - fts-solr: default_ns parameter couldn't be used
 - submission server crashed if relay server returned over 7 lines in
   a reply (e.g. to EHLO)



Re: outlook idiocy - IMAP folders with /

2018-10-01 Thread Timo Sirainen
On 28 Sep 2018, at 16.44, Wojciech Puchar  wrote:
> 
> user attempts to create folders with / dovecot naturally cannot create it so 
> it returns error but outlook of course "create" it and keep data in local 
> store only. data is lost when you remove local store .pst file.
> 
> The question is - can dovecot be configured so it will automatically replace 
> slash in name with something else?

https://wiki2.dovecot.org/Plugins/Listescape 
 maybe?



Re: doveadm backup abort in imapc-client.c

2018-10-01 Thread Timo Sirainen


> On 27 Sep 2018, at 21.52, Evan Klitzke  wrote:
> 
> I am using Dovecot 2.2.36 and I am trying to use doveadm backup to migrate 
> email from Gmail to Dovecot. When I run doveadm backup, it goes for a while 
> and then eventually hits an assertion error (I've tried this a few times now 
> and it happens reliably).
> 
> The assertion failure looks like this:
> 
> dsync(e...@eklitzke.org): Panic: file imapc-client.c: line 179 
> (imapc_client_run_pre): assertion failed: (client->ioloop == NULL)
> 
> The last few lines of log output before the crash look like this: 
> https://gist.github.com/eklitzke/9a0dd77c44a6ee33e812f85d5773c24c
> 
> The GDB backtrace when the program panics looks like this: 
> https://gist.github.com/eklitzke/6b311474b08b546c462e0444c5cf479b

Looks like it crashes when:
 - you have multiple folders
 - dsync takes over 1 second between folders
 - a new mail arrives during dsync
 - you have imapc_features=gmail-migration

A simple workaround should be to comment out 
"list->refreshed_mailboxes_recently = FALSE" in imapc-sync.c. A better fix 
would probably be to remember an opened folder's mailbox_info_flags so it 
doesn't try to re-get them.



Re: Bug in conditionals to assign values to variables?

2018-10-01 Thread Timo Sirainen


> On 28 Sep 2018, at 15.25, Angel L. Mateo  wrote:
> 
> user_attrs = ...,=relpath=%{if;%{user};eq;somevalue;valuetrue;valuefalse}
> 
>   then it reports in logs:
> 
> Sep 28 14:23:22 myotis60 dovecot: auth: Error: var_expand_long(if;%{user}) 
> failed: if: requires four or five parameters, got 1
> 
>   anyway, the variable is correctly initialized, but I get the log.
> 
>   Is this a bug?
> 
> PS: I'm running dovecot 2.2.33

I can't seem to be able to reproduce this, not even with v2.2.33. It simply 
works without logging any errors.



[Dovecot-news] v2.3.3 release candidate released

2018-09-21 Thread Timo Sirainen
https://dovecot.org/releases/2.3/rc/dovecot-2.3.3.rc1.tar.gz
https://dovecot.org/releases/2.3/rc/dovecot-2.3.3.rc1.tar.gz.sig

Binary packages are also available in https://repo.dovecot.org/ in ce-2.3.3 
repository (not ce-2.3-latest).

 * doveconf hides more secrets now in the default output.
 * ssl_dh setting is no longer enforced at startup. If it's not set and
   non-ECC DH key exchange happens, error is logged and client is
   disconnected.

 + Added log_debug= setting.
 + Added log_core_filter= setting.
 + quota-clone: Write to dict asynchronously
 + --enable-hardening attempts to use retpoline Spectre 2 mitigations
 + lmtp proxy: Support source_ip passdb extra field.
 + doveadm stats dump: Support more fields and output stddev by default.
 + push-notification: Add SSL support for OX backend.
 - NUL bytes in mail headers can cause truncated replies when fetched.
 - director: Conflicting host up/down state changes may in some rare
   situations ended up in a loop of two directors constantly overwriting
   each others' changes.
 - virtual plugin: Some searches used 100% CPU for many seconds
 - dsync assert-crashed with acl plugin in some situations.
 - mail_attachment_detection_options=add-flags-on-save assert-crashed
   with some specific Sieve scripts.
 - Mail snippet generation crashed with mails containing invalid
   Content-Type:multipart header.
 - Log prefix ordering was different for some log lines.
 - quota: With noenforcing option current quota usage wasn't updated.
 - auth: Kerberos authentication against Samba assert-crashed.
 - stats clients were unnecessarily chatty with the stats server.
 - imapc: Fixed various assert-crashes when reconnecting to server.
 - lmtp, submission: Fix potential crash if client disconnects while
   handling a command.
 - quota: Fixed compiling with glibc-2.26 / support libtirpc.
 - fts-solr: Empty search values resulted in 400 Bad Request errors
 - fts-solr: default_ns parameter couldn't be used
 - submission server crashed if relay server returned over 7 lines in
   a reply (e.g. to EHLO)

___
Dovecot-news mailing list
Dovecot-news@dovecot.org
https://dovecot.org/mailman/listinfo/dovecot-news


v2.3.3 release candidate released

2018-09-21 Thread Timo Sirainen
https://dovecot.org/releases/2.3/rc/dovecot-2.3.3.rc1.tar.gz
https://dovecot.org/releases/2.3/rc/dovecot-2.3.3.rc1.tar.gz.sig

Binary packages are also available in https://repo.dovecot.org/ in ce-2.3.3 
repository (not ce-2.3-latest).

 * doveconf hides more secrets now in the default output.
 * ssl_dh setting is no longer enforced at startup. If it's not set and
   non-ECC DH key exchange happens, error is logged and client is
   disconnected.

 + Added log_debug= setting.
 + Added log_core_filter= setting.
 + quota-clone: Write to dict asynchronously
 + --enable-hardening attempts to use retpoline Spectre 2 mitigations
 + lmtp proxy: Support source_ip passdb extra field.
 + doveadm stats dump: Support more fields and output stddev by default.
 + push-notification: Add SSL support for OX backend.
 - NUL bytes in mail headers can cause truncated replies when fetched.
 - director: Conflicting host up/down state changes may in some rare
   situations ended up in a loop of two directors constantly overwriting
   each others' changes.
 - virtual plugin: Some searches used 100% CPU for many seconds
 - dsync assert-crashed with acl plugin in some situations.
 - mail_attachment_detection_options=add-flags-on-save assert-crashed
   with some specific Sieve scripts.
 - Mail snippet generation crashed with mails containing invalid
   Content-Type:multipart header.
 - Log prefix ordering was different for some log lines.
 - quota: With noenforcing option current quota usage wasn't updated.
 - auth: Kerberos authentication against Samba assert-crashed.
 - stats clients were unnecessarily chatty with the stats server.
 - imapc: Fixed various assert-crashes when reconnecting to server.
 - lmtp, submission: Fix potential crash if client disconnects while
   handling a command.
 - quota: Fixed compiling with glibc-2.26 / support libtirpc.
 - fts-solr: Empty search values resulted in 400 Bad Request errors
 - fts-solr: default_ns parameter couldn't be used
 - submission server crashed if relay server returned over 7 lines in
   a reply (e.g. to EHLO)



Re: Auth process sometimes stop responding after upgrade

2018-09-19 Thread Timo Sirainen
On 19 Sep 2018, at 11.42, Timo Sirainen  wrote:
> 
> On 19 Sep 2018, at 11.30, Timo Sirainen mailto:t...@iki.fi>> 
> wrote:
>> 
>>> 
>>> On 19 Sep 2018, at 11.11, Simone Lazzaris >> <mailto:s.lazza...@interactive.eu>> wrote:
>>> 
>>> In data mercoledì 19 settembre 2018 09:30:47 CEST, Timo Sirainen ha scritto:
>>> > On 18 Sep 2018, at 15.15, Simone Lazzaris >> > <mailto:s.lazza...@interactive.eu>> wrote:
>>> > > I've got a core dump, and here is the backtrace. Let me know if you want
>>> > > the core file.
>>> > It would be useful if we're able to use it. Could you use
>>> > https://dovecot.org/tools/core-tar.sh 
>>> > <https://dovecot.org/tools/core-tar.sh>
>>> > <https://dovecot.org/tools/core-tar.sh 
>>> > <https://dovecot.org/tools/core-tar.sh>> to send the core and the related
>>> > binaries (e.g. just email to me)? The usage is explained at the beginning
>>> > of the script. At least in theory we could then debug with the core file,
>>> > although I've had some trouble even then.
>>> > 
>>> > But just in case the core doesn't work, could you also do:
>>> > 
>>> > bt full
>>> > fr 8
>>> > p *((struct doveadm_connection *)io->context)
>>> > p *((struct doveadm_connection *)io->context)->input
>>>  
>>> I'm sending you the tarball created with core-tar; and just in case:
>> 
>> Thanks, the core worked fine. Does the attached patch (on top of the 
>> previous one) help?
>> 
>> 
> 
> Or here's a slightly different patch, although it should be basically the 
> same fix. This includes the previous patch as well.
> 
> 

No, forget about that patch. Looks like I forgot I had already fixed this 
crash, and I guess I was testing with master mainly, which is why I wasn't able 
to reproduce the crash now: 
https://github.com/dovecot/core/commit/c0583917fe760b2d901acf83387cc8edb6f99550 
<https://github.com/dovecot/core/commit/c0583917fe760b2d901acf83387cc8edb6f99550>



Re: Auth process sometimes stop responding after upgrade

2018-09-19 Thread Timo Sirainen
On 19 Sep 2018, at 11.30, Timo Sirainen <t...@iki.fi> wrote:On 19 Sep 2018, at 11.11, Simone Lazzaris <s.lazza...@interactive.eu> wrote:In data mercoledì 19 settembre 2018 09:30:47 CEST, Timo Sirainen ha scritto:> On 18 Sep 2018, at 15.15, Simone Lazzaris <s.lazza...@interactive.eu> wrote:> > I've got a core dump, and here is the backtrace. Let me know if you want> > the core file.> It would be useful if we're able to use it. Could you use> https://dovecot.org/tools/core-tar.sh> <https://dovecot.org/tools/core-tar.sh> to send the core and the related> binaries (e.g. just email to me)? The usage is explained at the beginning> of the script. At least in theory we could then debug with the core file,> although I've had some trouble even then.> > But just in case the core doesn't work, could you also do:> > bt full> fr 8> p *((struct doveadm_connection *)io->context)> p *((struct doveadm_connection *)io->context)->input I'm sending you the tarball created with core-tar; and just in case:Thanks, the core worked fine. Does the attached patch (on top of the previous one) help?Or here's a slightly different patch, although it should be basically the same fix. This includes the previous patch as well.

director2.diff
Description: Binary data


Re: Auth process sometimes stop responding after upgrade

2018-09-19 Thread Timo Sirainen
On 19 Sep 2018, at 11.11, Simone Lazzaris <s.lazza...@interactive.eu> wrote:In data mercoledì 19 settembre 2018 09:30:47 CEST, Timo Sirainen ha scritto:> On 18 Sep 2018, at 15.15, Simone Lazzaris <s.lazza...@interactive.eu> wrote:> > I've got a core dump, and here is the backtrace. Let me know if you want> > the core file.> It would be useful if we're able to use it. Could you use> https://dovecot.org/tools/core-tar.sh> <https://dovecot.org/tools/core-tar.sh> to send the core and the related> binaries (e.g. just email to me)? The usage is explained at the beginning> of the script. At least in theory we could then debug with the core file,> although I've had some trouble even then.> > But just in case the core doesn't work, could you also do:> > bt full> fr 8> p *((struct doveadm_connection *)io->context)> p *((struct doveadm_connection *)io->context)->input I'm sending you the tarball created with core-tar; and just in case:Thanks, the core worked fine. Does the attached patch (on top of the previous one) help?

director.diff
Description: Binary data


Re: Auth process sometimes stop responding after upgrade

2018-09-19 Thread Timo Sirainen
On 18 Sep 2018, at 15.15, Simone Lazzaris  wrote:
> 
> I've got a core dump, and here is the backtrace. Let me know if you want the 
> core file.

It would be useful if we're able to use it. Could you use 
https://dovecot.org/tools/core-tar.sh  
to send the core and the related binaries (e.g. just email to me)? The usage is 
explained at the beginning of the script. At least in theory we could then 
debug with the core file, although I've had some trouble even then.

But just in case the core doesn't work, could you also do:

bt full
fr 8
p *((struct doveadm_connection *)io->context)
p *((struct doveadm_connection *)io->context)->input



Re: Auth process sometimes stop responding after upgrade

2018-09-18 Thread Timo Sirainen
On 18 Sep 2018, at 13.29, Simone Lazzaris  wrote:
> 
> > Hi all, again;
> >
> > I've enabled the core dumps and let it go for some day waiting for the issue
> > to reoccur.
> >
> > Meantime I've also upgraded the poolmon script, as Sami suggested.
> >
> > It seems that the upgrade has scared the issue away, because it no longer
> > occurred.
> >
> > Maybe the problem is related to the way the old poolmon talked to the
> > director daemon? I'm not very inclined to downgrade poolmon to catch a
> > traceback, but can do if neccessary.
>  
> Well, maybe it's not necessary ;)
> I've performed some maintenance operations on the backends and that triggered 
> the crash. It seems that something goes wrong where one backend come back 
> online.

It's weird how easily you can reproduce the crash. I've ran all kinds of 
(stress) tests and I can't reproduce this crash. I was able to reproduce the 
original hang though.
 
> Unfortunately, the core was not dumped And I don't know what to do: the 
> director service was not chrooted, and ulimit -c is unlimited.

Do you have: sysctl -w fs.suid_dumpable=2



Re: Occasional crash in db-auth.c (Valgrind: Invalid read of size 4 et al.), Dovecot 2.2.27+

2018-09-14 Thread Timo Sirainen

> On 14 Sep 2018, at 12.22, Jonas Schäfer  wrote:
> 
>> On Sonntag, 28. Januar 2018 13:59:24 CEST Jonas Wielicki wrote:
>>> On Samstag, 27. Januar 2018 21:33:51 CET you wrote:
>>> Hi thank you for these, can you send doveconf -n for your minimal
>>> reproducer?
>> 
> 
> Has this been fixed in any release? I’m not sure how to figure this out, 
> unfortunately.

Not in a release yet, but it is in git master: 
https://github.com/dovecot/core/commit/90bd9600a0e38e55c02c6266c1270fdd4138c07d

Re: Auth process sometimes stop responding after upgrade

2018-09-11 Thread Timo Sirainen
On 11 Sep 2018, at 10.57, Simone Lazzaris  wrote:
> 
> Sep 11 03:25:55 imap-front4 dovecot: director: Panic: file 
> doveadm-connection.c: line 1097 (doveadm_connection_deinit): assertion 
> failed: (conn->to_ring_sync_abort == NULL)
> Sep 11 03:25:55 imap-front4 dovecot: director: Fatal: master: 
> service(director): child 4395 killed with signal 6 (core dumps disabled)

It's crashing. Can you get gdb backtrace? First enable core dumps. 
https://dovecot.org/bugreport.html#coredumps 




Re: Auth process sometimes stop responding after upgrade

2018-09-10 Thread Timo Sirainen
On 8 Sep 2018, at 15.18, Simone Lazzaris  wrote:Timo, unfortunately the patch doesn't compile;  I've moved the declaration of "conn" one line up to make it work. Oops, I guess I was too much in a hurry to even compile it. Here's a new patch that compiles and passes our director CI tests.

diff
Description: Binary data


Re: Auth process sometimes stop responding after upgrade

2018-09-07 Thread Timo Sirainen
On 7 Sep 2018, at 19.43, Timo Sirainen <t...@iki.fi> wrote:On 7 Sep 2018, at 16.50, Simone Lazzaris <s.lazza...@interactive.eu> wrote:Some more information: the issue has just occurred, again on an instance without the "service_count = 0" configuration directive on pop3-login. I've observed that while the issue is occurring, the director process goes 100% CPU. I've straced the process. It is seemingly looping: ..epoll_ctl(13, EPOLL_CTL_ADD, 78, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=149035320, u64=149035320}}) = 0epoll_ctl(13, EPOLL_CTL_DEL, 78, {0, {u32=149035320, u64=149035320}}) = 0epoll_ctl(13, EPOLL_CTL_ADD, 78, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=149035320, u64=149035320}}) = 0epoll_ctl(13, EPOLL_CTL_DEL, 78, {0, {u32=149035320, u64=149035320}}) = 0epoll_ctl(13, EPOLL_CTL_ADD, 78, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=149035320, u64=149035320}}) = 0epoll_ctl(13, EPOLL_CTL_DEL, 78, {0, {u32=149035320, u64=149035320}}) = 0Nothing else but these epoll_ctl() calls? So it's gone to some loop where it keeps calling io_add() and io_remove(). I'm guessing it's because of doveadm command handling issues, since there's some weirdness in the code. Although I couldn't figure out exactly why it would go to infinite loop there. But attached a patch that may fix it, if you're able to test. We haven't noticed such infinite looping in other installations or automated director stresstests though..

2417.patch
Description: Binary data
FD 13 is "anon_inode:[eventpoll]"What about fd 78? I guess some socket.Could you also try two more things when it happens again:ltrace -tt -e '*' -o ltrace.log -p (My guess this isn't going to be very useful, but just in case it might be..)gdb -p bt fullquitPreferably install dovecot-dbg package also so the gdb backtrace output will be better.These would still be useful to verify whether I'm even on the right track.

Re: Auth process sometimes stop responding after upgrade

2018-09-07 Thread Timo Sirainen
On 7 Sep 2018, at 16.50, Simone Lazzaris  wrote:
> 
> Some more information: the issue has just occurred, again on an instance 
> without the "service_count = 0" configuration directive on pop3-login.
>  
> I've observed that while the issue is occurring, the director process goes 
> 100% CPU. I've straced the process. It is seemingly looping:
>  
> ...
> ...
> epoll_ctl(13, EPOLL_CTL_ADD, 78, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, 
> {u32=149035320, u64=149035320}}) = 0
> epoll_ctl(13, EPOLL_CTL_DEL, 78, {0, {u32=149035320, u64=149035320}}) = 0
> epoll_ctl(13, EPOLL_CTL_ADD, 78, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, 
> {u32=149035320, u64=149035320}}) = 0
> epoll_ctl(13, EPOLL_CTL_DEL, 78, {0, {u32=149035320, u64=149035320}}) = 0
> epoll_ctl(13, EPOLL_CTL_ADD, 78, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, 
> {u32=149035320, u64=149035320}}) = 0
> epoll_ctl(13, EPOLL_CTL_DEL, 78, {0, {u32=149035320, u64=149035320}}) = 0

Nothing else but these epoll_ctl() calls? So it's gone to some loop where it 
keeps calling io_add() and io_remove(). 

> FD 13 is "anon_inode:[eventpoll]"

What about fd 78? I guess some socket.

Could you also try two more things when it happens again:

ltrace -tt -e '*' -o ltrace.log -p 
(My guess this isn't going to be very useful, but just in case it might be..)

gdb -p 
bt full
quit

Preferably install dovecot-dbg package also so the gdb backtrace output will be 
better.



Re: dsync panic

2018-07-13 Thread Timo Sirainen
On 13 Jul 2018, at 21.39, Wolfgang Rosenauer mailto:wrosena...@gmail.com>> wrote:
> 
> I think I get pretty much the same issue:
> 
> dsync(support): Panic: file mailbox-attribute.c: line 360 
> (mailbox_attribute_get_stream): assertion failed: (value_r->value != NULL || 
> value_r->value_stream != NULL)
> dsync(support): Error: Raw backtrace: 
> /usr/lib64/dovecot/libdovecot.so.0(+0xc9e06) [0x7fba8a348e06] -> 
> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) 
> [0x7fba8a348e4a] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) 
> [0x7fba8a2bd813] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x597fc) 
> [0x7fba8a64f7fc] -> dsync(dsync_mailbox_import_attribute+0x3a) 
> [0x55d72a8ac37a] -> dsync(dsync_brain_sync_mails+0x2ef) [0x55d72a8a7e9f] -> 
> dsync(dsync_brain_run+0x2b0) [0x55d72a8a3690] -> dsync(+0x29f5c) 
> [0x55d72a88af5c] -> dsync(+0x2bd97) [0x55d72a88cd97] -> dsync(+0x2c878) 
> [0x55d72a88d878] -> dsync(doveadm_mail_try_run+0x205) [0x55d72a88e1f5] -> 
> dsync(main+0x475) [0x55d72a87de85] -> 
> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fba89c8d725] -> 
> dsync(_start+0x29) [0x55d72a87e009]

Already fixed, but was scheduled for v2.3.3: 
https://github.com/dovecot/core/commit/4ff4bd024a9b6e7973b76b186ce085c2ca669d3e 


As a workaround, you could disable ACL when running dsync: doveadm -o 
plugin/acl= ...



Re: LMTP crashing heavily for my 2.2.36 installation

2018-07-11 Thread Timo Sirainen
On 11 Jul 2018, at 8.41, Wolfgang Rosenauer  wrote:
> 
> Hi,
> 
> I'm running 2.2.36 (as provided by openSUSE in their server:mail repository) 
> and at least at one of my systems LMTP is crashing regularly on certain 
> messages (apparently a lot of them).
> 
> Sometimes (but not always a backtrace is posted to the logs:
> 
> 2018-07-11T07:34:56.741848+02:00 saruman dovecot: lmtp(14690): Fatal: master: 
> service(lmtp): child 14690 killed with signal 11 (core dumps disabled)
> 2018-07-11T07:34:56.820474+02:00 saruman dovecot: lmtp(an007498): Panic: file 
> imap-bodystructure.c: line 116 (part_write_body_multipart): assertion failed: 
> (part->data != NULL)
..
> storage.so.0(mail_set_attachment_keywords+0x162) [0x7ff549773662]

Looks like it's because of the "mail_attachment_detection_options = 
add-flags-on-save" setting. What mailbox format do you use? It's currently 
broken with mbox and in v2.2.36 with Maildir (which was fixed in v2.3.2).



Re: v2.3.2.1 released

2018-07-10 Thread Timo Sirainen
On 10 Jul 2018, at 10.40, Tom Sommer  wrote:
> 
> On 2018-07-09 16:48, Timo Sirainen wrote:
>> https://dovecot.org/releases/2.3/dovecot-2.3.2.1.tar.gz
>> https://dovecot.org/releases/2.3/dovecot-2.3.2.1.tar.gz.sig
>> v2.3.2 still had a few unexpected bugs:
>> - SSL/TLS servers may have crashed during client disconnection
>> - lmtp: With lmtp_rcpt_check_quota=yes mail deliveries may have
>>   sometimes assert-crashed.
>> - v2.3.2: "make check" may have crashed with 32bit systems
> 
> Thank you for the fast release, much appreciated

Are the imap-login crashes now gone?



Re: IMAP copy stopped copying flags

2018-07-09 Thread Timo Sirainen
On 9 Jul 2018, at 16.49, Andrzej A. Filip  wrote:
> 
> Is it intended behavior?

No.

> It seems to be caused by upgrade to 1:2.3.2-2 on Debian/Testing.

What was the old version? What's your doveconf -n? How are you testing that 
it's not working?



[Dovecot-news] v2.3.2.1 released

2018-07-09 Thread Timo Sirainen
https://dovecot.org/releases/2.3/dovecot-2.3.2.1.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.2.1.tar.gz.sig

v2.3.2 still had a few unexpected bugs:

 - SSL/TLS servers may have crashed during client disconnection
 - lmtp: With lmtp_rcpt_check_quota=yes mail deliveries may have
   sometimes assert-crashed.
 - v2.3.2: "make check" may have crashed with 32bit systems

___
Dovecot-news mailing list
Dovecot-news@dovecot.org
https://dovecot.org/mailman/listinfo/dovecot-news


v2.3.2.1 released

2018-07-09 Thread Timo Sirainen
https://dovecot.org/releases/2.3/dovecot-2.3.2.1.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.2.1.tar.gz.sig

v2.3.2 still had a few unexpected bugs:

 - SSL/TLS servers may have crashed during client disconnection
 - lmtp: With lmtp_rcpt_check_quota=yes mail deliveries may have
   sometimes assert-crashed.
 - v2.3.2: "make check" may have crashed with 32bit systems



Re: 2.3.2 is still crashing (lmtp)

2018-07-06 Thread Timo Sirainen

> On 6 Jul 2018, at 11.42, Timo Sirainen  wrote:
> 
> On 6 Jul 2018, at 1.53, Martynas Bendorius  wrote:
>> 
>> Dovecot 2.3.2 still has the same issue we reported for 2.3.X:
>> 
>> Jul 05 15:10:49 lmtp(2730445): Panic: file lib-event.c: line 182 
>> (event_pop_global): assertion failed: (event != NULL)
> ...
>> #5  0x7f6502986e42 in event_pop_global (event=) at 
>> lib-event.c:182
>>   __func__ = "event_pop_global"
>> #6  0x7f6502c68f88 in mail_storage_service_io_deactivate_user_cb 
>> (user=0x7f65048e4708)
>>   at mail-storage-service.c:823
>> ---Type  to continue, or q  to quit---
>>   event = 0x0
>>   __func__ = "mail_storage_service_io_deactivate_user_cb"
> 
> Could you also run in gdb:
> 
> fr 6
> p *user
> p *user->pool
> 
> Also what's your doveconf -n?

Also, does this happen to fix it? 
https://github.com/dovecot/core/commit/231865af423b4fa4da209a0721de57840c9b91de.patch
 
<https://github.com/dovecot/core/commit/231865af423b4fa4da209a0721de57840c9b91de.patch>



Re: Benchmarks with imaptest

2018-07-06 Thread Timo Sirainen
On 4 Jul 2018, at 20.10, João Paulo Sacchetto Ribeiro Bastos 
 wrote:
> 
> Hey guys, 
> 
> Has anybody ever used imaptest to benchmark *ONLY* reads? I'm trying to run a 
> battery of tests in my new cluster and apparently no combination of 
> parameters works, because fetch instruction doesn't run. Below is the command 
> I'm using and its partial output
> 
> imaptest/src/imaptest host=$SERVER_IP port=143 user=teste%d@example%d.com 
>  pass=$TEST_PASS users=10 domains=10 clients=100 - append=0 
> select=100 fetch=100
> Logi Sele Fetc Logo 
> 100% 100% 100% 100% 
>  166  1560  156  99/100
>  223  2300  230 100/100
>  206  2030  203  97/100
> 
> Can anybody help me understand why fetch doesn't work and how to fix this?

I think the users simply don't have any mails in their INBOX. You can also add 
"rawlog" parameter and imaptest writes rawlog.* files. From them you can see 
what the IMAP traffic is. Especially if it says "* 0 EXISTS" then there are no 
mails.



Re: 2.3.2 is still crashing (lmtp)

2018-07-06 Thread Timo Sirainen
On 6 Jul 2018, at 1.53, Martynas Bendorius  wrote:
> 
> Dovecot 2.3.2 still has the same issue we reported for 2.3.X:
> 
> Jul 05 15:10:49 lmtp(2730445): Panic: file lib-event.c: line 182 
> (event_pop_global): assertion failed: (event != NULL)
...
> #5  0x7f6502986e42 in event_pop_global (event=) at 
> lib-event.c:182
>__func__ = "event_pop_global"
> #6  0x7f6502c68f88 in mail_storage_service_io_deactivate_user_cb 
> (user=0x7f65048e4708)
>at mail-storage-service.c:823
> ---Type  to continue, or q  to quit---
>event = 0x0
>__func__ = "mail_storage_service_io_deactivate_user_cb"

Could you also run in gdb:

fr 6
p *user
p *user->pool

Also what's your doveconf -n?



Re: 2.3.2 director imap-login segfaults

2018-07-06 Thread Timo Sirainen
On 5 Jul 2018, at 15.12, Tom Sommer  wrote:
> 
> My director has started segfaulting since upgradeing to 2.3.2:
> 
> #0  0x7fa19b3ec6ed in i_stream_get_root_io () from 
> /usr/lib64/dovecot/libdovecot.so.0
> No symbol table info available.
> #1  0x7fa19b3ec9b5 in i_stream_set_input_pending () from 
> /usr/lib64/dovecot/libdovecot.so.0
> No symbol table info available.
> #2  0x7fa198d48b35 in openssl_iostream_bio_sync () from 
> /usr/lib64/dovecot/libssl_iostream_openssl.so
> No symbol table info available.
> #3  0x7fa198d4920a in openssl_iostream_more () from 
> /usr/lib64/dovecot/libssl_iostream_openssl.so
> No symbol table info available.

Can you try if the attached patch fixes it?



ssl-crash-fix.diff
Description: Binary data




Re: v2.3.2 released

2018-06-29 Thread Timo Sirainen
On 29 Jun 2018, at 15.28, Tom Sommer  wrote:
> 
> On 2018-06-29 15:20, Timo Sirainen wrote:
>> On 29 Jun 2018, at 15.05, Tom Sommer  wrote:
>>> On 2018-06-29 14:51, Timo Sirainen wrote:
>>>> v2.3.2 is mainly a bugfix release. It contains all the changes in
>>>> v2.2.36, as well as a bunch of other fixes (mainly for v2.3-only
>>>> bugs). Binary packages are already in https://repo.dovecot.org/
>>> A simple "yum update" will result in a ton of these errors:
>>> Jun 29 15:02:19 stats: Error: stats: Socket supports major version 2, but 
>>> we support only 3 (mixed old and new binaries?)
>>> Should the yum update process perhaps not restart the dovecot service?
>> It sounds like you upgraded from v2.2.x to v2.3.2? The stats was
>> completely changed.
> 
> From 2.3.1 to 2.3.2:
> 
> Installed:
>  dovecot.x86_64 2:2.3.2-3dovecot-mysql.x86_64 2:2.3.2-3
> 
> Updated:
>  dovecot-pigeonhole.x86_64 2:2.3.2-3
> 
> Replaced:
>  dovecot.x86_64 2:2.3.1-1dovecot-mysql.x86_64 2:2.3.1-1

That's weird. Something appears to be connecting to the stats socket with old 
protocol version. What your doveconf -n?



Re: v2.3.2 released

2018-06-29 Thread Timo Sirainen
On 29 Jun 2018, at 15.05, Tom Sommer  wrote:
> 
> On 2018-06-29 14:51, Timo Sirainen wrote:
> 
>> v2.3.2 is mainly a bugfix release. It contains all the changes in
>> v2.2.36, as well as a bunch of other fixes (mainly for v2.3-only
>> bugs). Binary packages are already in https://repo.dovecot.org/
> 
> A simple "yum update" will result in a ton of these errors:
> 
> Jun 29 15:02:19 stats: Error: stats: Socket supports major version 2, but we 
> support only 3 (mixed old and new binaries?)
> 
> Should the yum update process perhaps not restart the dovecot service?

It sounds like you upgraded from v2.2.x to v2.3.2? The stats was completely 
changed.



[Dovecot-news] v2.3.2 released

2018-06-29 Thread Timo Sirainen
https://dovecot.org/releases/2.3/dovecot-2.3.2.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.2.tar.gz.sig

v2.3.2 is mainly a bugfix release. It contains all the changes in v2.2.36, as 
well as a bunch of other fixes (mainly for v2.3-only bugs). Binary packages are 
already in https://repo.dovecot.org/

 * old-stats plugin: Don't temporarily enable PR_SET_DUMPABLE while
   opening /proc/self/io. This may still cause security problems if the
   process is ptrace()d at the same time. Instead, open it while still
   running as root.
 + doveadm: Added mailbox cache decision commands. See
   doveadm-mailbox(1) man page for details.
 + doveadm: Added rebuild attachments command for rebuilding
   $HasAttachment or $HasNoAttachment flags for matching mails. See
   doveadm-rebuild(1) man page for details.
 + cassandra: Use fallback_consistency on more types of errors
 + lmtp proxy: Support outgoing SSL/TLS connections
 + lmtp: Add lmtp_rawlog_dir and lmtp_proxy_rawlog_dir settings.
 + submission: Add support for rawlog_dir
 + submission: Add submission_client_workarounds setting.
 + lua auth: Add password_verify() function and additional fields in
   auth request.
 - doveadm-server: TCP connections are hanging when there is a lot of
   network output. This especially caused hangs in dsync-replication.
 - Using multiple type=shared mdbox namespaces crashed
 - mail_fsync setting was ignored. It was always set to "optimized".
 - lua auth: Fix potential crash at deinit
 - SSL/TLS servers may have crashed if client disconnected during
   handshake.
 - SSL/TLS servers: Don't send extraneous certificates to client when
   alt certs are used.
 - lda, lmtp: Return-Path header without '<' may have assert-crashed.
 - lda, lmtp: Unencoded UTF-8 in email address headers may assert-crash
 - lda: -f parameter didn't allow empty/null/domainless address
 - lmtp, submission: Message size limit was hardcoded to 40 MB.
   Exceeding it caused the connection to get dropped during transfer.
 - lmtp: Fix potential crash when delivery fails at DATA stage
 - lmtp: login_greeting setting was ignored
 - Fix to work with OpenSSL v1.0.2f
 - systemd unit restrictions were too strict by default
 - Fix potential crashes when a lot of log output was produced
 - SMTP client may have assert-crashed when sending mail
 - IMAP COMPRESS: Send "end of compression" marker when disconnecting.
 - cassandra: Fix consistency=quorum to work
 - dsync: Lock file generation failed if home directory didn't exist
 - Snippet generation for HTML mails didn't ignore  inside
   blockquotes, producing strange looking snippets.
 - imapc: Fix assert-crash if getting disconnected and after
   reconnection all mails in the selected mailbox are gone.
 - pop3c: Handle unexpected server disconnections without assert-crash
 - fts: Fixes to indexing mails via virtual mailboxes.
 - fts: If mails contained NUL characters, the text around it wasn't
   indexed.
 - Obsolete dovecot.index.cache offsets were sometimes used. Trying to
   fetch a field that was just added to cache file may not have always
   found it.

___
Dovecot-news mailing list
Dovecot-news@dovecot.org
https://dovecot.org/mailman/listinfo/dovecot-news


v2.3.2 released

2018-06-29 Thread Timo Sirainen
https://dovecot.org/releases/2.3/dovecot-2.3.2.tar.gz
https://dovecot.org/releases/2.3/dovecot-2.3.2.tar.gz.sig

v2.3.2 is mainly a bugfix release. It contains all the changes in v2.2.36, as 
well as a bunch of other fixes (mainly for v2.3-only bugs). Binary packages are 
already in https://repo.dovecot.org/

 * old-stats plugin: Don't temporarily enable PR_SET_DUMPABLE while
   opening /proc/self/io. This may still cause security problems if the
   process is ptrace()d at the same time. Instead, open it while still
   running as root.
 + doveadm: Added mailbox cache decision commands. See
   doveadm-mailbox(1) man page for details.
 + doveadm: Added rebuild attachments command for rebuilding
   $HasAttachment or $HasNoAttachment flags for matching mails. See
   doveadm-rebuild(1) man page for details.
 + cassandra: Use fallback_consistency on more types of errors
 + lmtp proxy: Support outgoing SSL/TLS connections
 + lmtp: Add lmtp_rawlog_dir and lmtp_proxy_rawlog_dir settings.
 + submission: Add support for rawlog_dir
 + submission: Add submission_client_workarounds setting.
 + lua auth: Add password_verify() function and additional fields in
   auth request.
 - doveadm-server: TCP connections are hanging when there is a lot of
   network output. This especially caused hangs in dsync-replication.
 - Using multiple type=shared mdbox namespaces crashed
 - mail_fsync setting was ignored. It was always set to "optimized".
 - lua auth: Fix potential crash at deinit
 - SSL/TLS servers may have crashed if client disconnected during
   handshake.
 - SSL/TLS servers: Don't send extraneous certificates to client when
   alt certs are used.
 - lda, lmtp: Return-Path header without '<' may have assert-crashed.
 - lda, lmtp: Unencoded UTF-8 in email address headers may assert-crash
 - lda: -f parameter didn't allow empty/null/domainless address
 - lmtp, submission: Message size limit was hardcoded to 40 MB.
   Exceeding it caused the connection to get dropped during transfer.
 - lmtp: Fix potential crash when delivery fails at DATA stage
 - lmtp: login_greeting setting was ignored
 - Fix to work with OpenSSL v1.0.2f
 - systemd unit restrictions were too strict by default
 - Fix potential crashes when a lot of log output was produced
 - SMTP client may have assert-crashed when sending mail
 - IMAP COMPRESS: Send "end of compression" marker when disconnecting.
 - cassandra: Fix consistency=quorum to work
 - dsync: Lock file generation failed if home directory didn't exist
 - Snippet generation for HTML mails didn't ignore  inside
   blockquotes, producing strange looking snippets.
 - imapc: Fix assert-crash if getting disconnected and after
   reconnection all mails in the selected mailbox are gone.
 - pop3c: Handle unexpected server disconnections without assert-crash
 - fts: Fixes to indexing mails via virtual mailboxes.
 - fts: If mails contained NUL characters, the text around it wasn't
   indexed.
 - Obsolete dovecot.index.cache offsets were sometimes used. Trying to
   fetch a field that was just added to cache file may not have always
   found it.



Re: doveadm: problem listing shared mailboxes using a wildcard

2018-06-27 Thread Timo Sirainen
On 25 Jun 2018, at 16.03, Patrick Ben Koetter  wrote:
> 
> In order to do some post-login scripting foo I'd like to get a list of shared
> mailboxes the user is currently subscribed to. The doveadm-mailbox man page
> says "It's also possible to use wildcards in the mailbox name."
> 
> I'd like to use that feature to output only mailboxes from the shared
> namespace. However usind the wildcard character * doesn't output anything:
> 
> # /bin/doveadm mailbox list -u fd...@spike.test  -s 
> shared*

The wildcards traverse only within the same namespace, so in this case the 
"shared/" namespace, which doesn't have any mailboxes (only child namespaces). 
Maybe we'll change this for v2.4.



[Dovecot-news] v2.3.2 release candidate released

2018-06-18 Thread Timo Sirainen
https://dovecot.org/releases/2.3/rc/dovecot-2.3.2.rc1.tar.gz
https://dovecot.org/releases/2.3/rc/dovecot-2.3.2.rc1.tar.gz.sig 

v2.3.2 is mainly a bugfix release. It contains all the changes in v2.2.36, as 
well as a bunch of other fixes (mainly for v2.3-only bugs).

 * old-stats plugin: Don't temporarily enable PR_SET_DUMPABLE while
   opening /proc/self/io. This may still cause security problems if the
   process is ptrace()d at the same time. Instead, open it while still
   running as root.

 + doveadm: Added mailbox cache decision commands. See
   doveadm-mailbox(1) man page for details.
 + doveadm: Added rebuild attachments command for rebuilding
   $HasAttachment or $HasNoAttachment flags for matching mails. See
   doveadm-rebuild(1) man page for details.
 + cassandra: Use fallback_consistency on more types of errors
 + lmtp proxy: Support outgoing SSL/TLS connections
 + lmtp: Add lmtp_rawlog_dir and lmtp_proxy_rawlog_dir settings.
 + submission: Add support for rawlog_dir
 + submission: Add submission_client_workarounds setting.
 + lua auth: Add password_verify() function and additional fields in
   auth request.
 - doveadm-server: TCP connections are hanging when there is a lot of
   network output. This especially caused hangs in dsync-replication.
 - Using multiple type=shared mdbox namespaces crashed
 - mail_fsync setting was ignored. It was always set to "optimized".
 - lua auth: Fix potential crash at deinit
 - SSL/TLS servers may have crashed if client disconnected during
   handshake.
 - SSL/TLS servers: Don't send extraneous certificates to client when
   alt certs are used.
 - lda, lmtp: Return-Path header without '<' may have assert-crashed.
 - lda, lmtp: Unencoded UTF-8 in email address headers may assert-crash
 - lda: -f parameter didn't allow empty/null/domainless address
 - lmtp, submission: Message size limit was hardcoded to 40 MB.
   Exceeding it caused the connection to get dropped during transfer.
 - lmtp: Fix potential crash when delivery fails at DATA stage
 - lmtp: login_greeting setting was ignored
 - Fix to work with OpenSSL v1.0.2f
 - systemd unit restrictions were too strict by default
 - Fix potential crashes when a lot of log output was produced
 - SMTP client may have assert-crashed when sending mail
 - IMAP COMPRESS: Send "end of compression" marker when disconnecting.
 - cassandra: Fix consistency=quorum to work
 - dsync: Lock file generation failed if home directory didn't exist
 - Snippet generation for HTML mails didn't ignore  inside
   blockquotes, producing strange looking snippets.
 - imapc: Fix assert-crash if getting disconnected and after
   reconnection all mails in the selected mailbox are gone.
 - pop3c: Handle unexpected server disconnections without assert-crash
 - fts: Fixes to indexing mails via virtual mailboxes.
 - fts: If mails contained NUL characters, the text around it wasn't
   indexed.
 - Obsolete dovecot.index.cache offsets were sometimes used. Trying to
   fetch a field that was just added to cache file may not have always
   found it.

___
Dovecot-news mailing list
Dovecot-news@dovecot.org
https://dovecot.org/mailman/listinfo/dovecot-news


v2.3.2 release candidate released

2018-06-18 Thread Timo Sirainen
https://dovecot.org/releases/2.3/rc/dovecot-2.3.2.rc1.tar.gz
https://dovecot.org/releases/2.3/rc/dovecot-2.3.2.rc1.tar.gz.sig 

v2.3.2 is mainly a bugfix release. It contains all the changes in v2.2.36, as 
well as a bunch of other fixes (mainly for v2.3-only bugs).

 * old-stats plugin: Don't temporarily enable PR_SET_DUMPABLE while
   opening /proc/self/io. This may still cause security problems if the
   process is ptrace()d at the same time. Instead, open it while still
   running as root.

 + doveadm: Added mailbox cache decision commands. See
   doveadm-mailbox(1) man page for details.
 + doveadm: Added rebuild attachments command for rebuilding
   $HasAttachment or $HasNoAttachment flags for matching mails. See
   doveadm-rebuild(1) man page for details.
 + cassandra: Use fallback_consistency on more types of errors
 + lmtp proxy: Support outgoing SSL/TLS connections
 + lmtp: Add lmtp_rawlog_dir and lmtp_proxy_rawlog_dir settings.
 + submission: Add support for rawlog_dir
 + submission: Add submission_client_workarounds setting.
 + lua auth: Add password_verify() function and additional fields in
   auth request.
 - doveadm-server: TCP connections are hanging when there is a lot of
   network output. This especially caused hangs in dsync-replication.
 - Using multiple type=shared mdbox namespaces crashed
 - mail_fsync setting was ignored. It was always set to "optimized".
 - lua auth: Fix potential crash at deinit
 - SSL/TLS servers may have crashed if client disconnected during
   handshake.
 - SSL/TLS servers: Don't send extraneous certificates to client when
   alt certs are used.
 - lda, lmtp: Return-Path header without '<' may have assert-crashed.
 - lda, lmtp: Unencoded UTF-8 in email address headers may assert-crash
 - lda: -f parameter didn't allow empty/null/domainless address
 - lmtp, submission: Message size limit was hardcoded to 40 MB.
   Exceeding it caused the connection to get dropped during transfer.
 - lmtp: Fix potential crash when delivery fails at DATA stage
 - lmtp: login_greeting setting was ignored
 - Fix to work with OpenSSL v1.0.2f
 - systemd unit restrictions were too strict by default
 - Fix potential crashes when a lot of log output was produced
 - SMTP client may have assert-crashed when sending mail
 - IMAP COMPRESS: Send "end of compression" marker when disconnecting.
 - cassandra: Fix consistency=quorum to work
 - dsync: Lock file generation failed if home directory didn't exist
 - Snippet generation for HTML mails didn't ignore  inside
   blockquotes, producing strange looking snippets.
 - imapc: Fix assert-crash if getting disconnected and after
   reconnection all mails in the selected mailbox are gone.
 - pop3c: Handle unexpected server disconnections without assert-crash
 - fts: Fixes to indexing mails via virtual mailboxes.
 - fts: If mails contained NUL characters, the text around it wasn't
   indexed.
 - Obsolete dovecot.index.cache offsets were sometimes used. Trying to
   fetch a field that was just added to cache file may not have always
   found it.



Re: 2.3.1 Replication is throwing scary errors

2018-06-07 Thread Timo Sirainen
On 7 Jun 2018, at 11.43, Michael Grimm  wrote:
> 
> Timo Sirainen:
> 
>> Should be fixed by
>> https://github.com/dovecot/core/commit/a952e178943a5944255cb7c053d970f8e6d49336
> 
> please ignore my ignorance but shouldn't one add this commit regarding 
> src/doveadm/client-connection-tcp.c ...
> 
> https://github.com/dovecot/core/commit/2a3b7083ce4e62a8bd836f9983b223e98e9bc157
> 
> ... to a vanilla 2.3.1 source tree as well?

That's a code simplification / cleanup commit. It doesn't fix anything.



Re: Header DATE field not returned when placed after MIME section

2018-06-06 Thread Timo Sirainen
On 2 Jun 2018, at 1.03, Xavier Guerin  wrote:
> 
> Hello,
> 
> I am using Dovecot 2.2.34p0 on OpenBSD 6.3 stable.
> 
> I am writting an application using the vmime (http://www.vmime.org)
> library (x-ref issue: https://github.com/kisli/vmime/issues/199).
> 
> I noticed through testing over my own e-mail server that in some
> situations the DATE field is not returned upon ENVELOPE or
> HEADER.FIELDS request.
> 
> The e-mails at fault happen to have the DATE field at the very end of
> the header, after the MIME section.

Can you easily reproduce this with doveadm? I can't, tested with current master 
and v2.2.34. :

% cat << EOF | doveadm save -u tss
Delivered-To:
Received:
Return-Path:
Delivered-To:
DKIM-Signature:
From:
To:
Message-ID:
Subject:
MIME-Version: 1.0
Content-Type: text/html;charset=UTF-8
Content-Transfer-Encoding: 7bit
Date: Fri, 01 Jun 2018 17:44:44 -0400

body
EOF

% doveadm fetch -u tss hdr.date mailbox inbox '*'
hdr.date: Fri, 01 Jun 2018 17:44:44 -0400

% doveadm fetch -u tss imap.envelope mailbox inbox '*'
imap.envelope: "Fri, 01 Jun 2018 17:44:44 -0400" "" NIL NIL NIL NIL NIL NIL NIL 
""

I tested also with your example 2 mail, but it also worked. Maybe there's 
something else special in those mails. If you can reproduce it with the above 
commands, send me the exact mail as attachment that you used?



Re: 2.3.1 Replication is throwing scary errors

2018-06-06 Thread Timo Sirainen
Should be fixed by 
https://github.com/dovecot/core/commit/a952e178943a5944255cb7c053d970f8e6d49336 




Re: use instance-name for syslog?

2018-05-31 Thread Timo Sirainen


> On 30 May 2018, at 19.08, SATOH Fumiyasu  wrote:
> 
> Hi!
> 
> On Thu, 31 May 2018 00:44:58 +0900,
> A. Schulze wrote:
>> When running multiple instances of dovecot on the same host (or running 
>> multiple docker container),
>> it is hard to distinguish logs from different processes: the syslog entries 
>> are all prefixed with the same identifier "dovecot"
>> It is hardcoded here:
>> https://github.com/dovecot/core/blob/master/src/lib-master/master-service.c#L420
>> 
>> Would it make sense to use the already implemented instance-name as syslog 
>> ident?
>> How do others solve that problem?
> 
> I have a patchset to implement that. Please see the attachment.

> Subject: [PATCH 1/2] master: Do not prepend "dovecot-" to a process name


Why not? I'd think it would be useful to always find dovecot processes.

> - openlog(ident, options, facility);
> + static char *syslog_ident = NULL;
> +
> + i_free(syslog_ident);
> + syslog_ident = i_strdup(ident);
> +
> + openlog(syslog_ident, options, facility);


I don't think this is necessary?

> + env_put(t_strconcat("INSTANCE_NAME=", set->instance_name, 
> NULL));


Also not needed.

But yeah, I guess in general it would make sense to use instance_name for 
syslog ident.



Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-28 Thread Timo Sirainen
On 28 May 2018, at 13.28, kfx  wrote:
> 
> 
>> Especially what is in the "fts" header vs. "next uid" header? Does the UID 
>> in "fts" header keep changing every time you save a new mail? I suppose it 
>> will. 
> 
> Diff between 2 emails:
> next uid = 30104|next uid = 30105
> last_indexed_uid = 30103|last_indexed_uid = 30104

So Dovecot thinks it has indexed everything.

>> You could also monitor (e.g. tcpdump/wireshark) the network traffic between 
>> Dovecot <-> Solr what happens when a new mail arrives. I suspect Dovecot 
>> sends it to Solr, which for whatever reason just ignores the update.
> 
> ### TCPDUMP 
> POST /solr/dovecot/update HTTP/1.1
> Host: localhost:8983
> Date: Mon, 28 May 2018 10:18:05 GMT
> Transfer-Encoding: chunked
> Connection: Keep-Alive
> Content-Type: text/xml
> 
> 37581 name="box">e0c58a309323515311083ea484a8 name="user">username name="id">37581/e0c58a309323515311083ea484a8/username name="body">Search Pattern: Kai8oovi
..
> 

And Dovecot sends the mail.

> # SOLR'S RESPONSE ###
> 
> 
> 
> 
>  0
>  0
> 
> 

And Solr receives it. Your tcpdump doesn't show  being sent though. Do you see it being sent anywhere? 
Does it make the mails visible if you run it yourself? Or if you run hard 
commit? :

curl http://:8983/solr/update?commit=true



Re: Bug: subscriptions file

2018-05-24 Thread Timo Sirainen
I'd rather not add RFC-breaking settings. But there's IMAP4rev2 discussion 
going on in https://www.ietf.org/mailman/listinfo/extra 
. Someone motivated enough could 
perhaps try to suggest changing this behavior in there.

> On 23 May 2018, at 23.13, Rupert Gallagher  wrote:
> 
> Sorry for top posting, my client is still broken. 
> 
> I have never seen the ghost of a "system-alerts" or similar "well-known" mail 
> folder in the past 30 years. 
> 
> Compliance with an RFC obscure feature is compellong us all to clear 
> subscriptions fol ders by hand. 
> 
> As we meet the problem over and over again, a non-RFC configuration option 
> could solve the problem, and it would be very much appreciated...
> 
> 
> On Wed, May 23, 2018 at 11:57, Aki Tuomi  > wrote:
> 
> > On 23.05.2018 12:31, Rupert Gallagher wrote:
>>> Dovecot does not clear the subscription file from non-existent folders. 
>> 
>> Hi! 
>> 
>> Thank you for your bug report. Unfortunately this is not a BUG, but mandated 
>> behavior by RFC3501, see last two paragraphs in the excerpt. 
>> 
>> Aki Tuomi 
>> 
>> 6.3.6.  SUBSCRIBE Command 
>> 
>>Arguments:  mailbox 
>> 
>>Responses:  no specific responses for this command 
>> 
>>Result: OK - subscribe completed 
>>NO - subscribe failure: can't subscribe to that name 
>>BAD - command unknown or arguments invalid 
>> 
>>   The SUBSCRIBE command adds the specified mailbox name to the 
>>   server's set of "active" or "subscribed" mailboxes as returned by 
>>   the LSUB command.  This command returns a tagged OK response only 
>>   if the subscription is successful. 
>> 
>>   A server MAY validate the mailbox argument to SUBSCRIBE to verify 
>>   that it exists.  However, it MUST NOT unilaterally remove an 
>>   existing mailbox name from the subscription list even if a mailbox 
>>   by that name no longer exists. 
>> 
>>Note: This requirement is because a server site can 
>>choose to routinely remove a mailbox with a well-known 
>>name (e.g., "system-alerts") after its contents expire, 
>>with the intention of recreating it when new contents 
>>are appropriate. 



Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-24 Thread Timo Sirainen
On 21 May 2018, at 14.11, kada...@gmail.com wrote:
> 
> Le 21/05/2018 à 12:38, Aki Tuomi a écrit :
>> can you try turning on pluign { fts_enforced = yes } and repeat your test?
> 
> Same (wrong) result:
> 1. Send an email with "too6Ouka" in the body
> 
> 2. Search against "too6Ouka":
> # doveadm search -u username mailbox INBOX body too6Ouka
> --> No result
> 
> 3. Force re-index:
> # doveadm fts rescan -u username
> 
> 4. Search again against "too6Ouka":
> # doveadm search -u username mailbox INBOX body too6Ouka
> --> e09cce0283e8695ab76002deed92 29055
> 
> Don't know if relevant, but on a side note, if I send a second message
> with "too6Ouka" in the body, followed by:
> # doveadm -v index -u username Inbox
> --> doveadm(username): Info: INBOX: Cache is already up to date
> And a search against the pattern immediately return only one result
> instead of two.

So it looks like after "doveadm fts rescan" you can do initial indexing 
successfully. Afterwards the index isn't updated at all, unless you rebuild it 
entirely. fts_autoindex=yes appears to work as expected, because after fts 
rescan you did a search which automatically updated the index. So this doesn't 
have anything to do with fts_autoindex, just that your index updates don't seem 
to work at all.

What does it show in the header if you do:
doveadm dump /var/vmail/username/dovecot.index

Especially what is in the "fts" header vs. "next uid" header? Does the UID in 
"fts" header keep changing every time you save a new mail? I suppose it will. 
You could also monitor (e.g. tcpdump/wireshark) the network traffic between 
Dovecot <-> Solr what happens when a new mail arrives. I suspect Dovecot sends 
it to Solr, which for whatever reason just ignores the update.



[Dovecot-news] v2.2.36 released

2018-05-23 Thread Timo Sirainen
https://dovecot.org/releases/2.2/dovecot-2.2.36.tar.gz
https://dovecot.org/releases/2.2/dovecot-2.2.36.tar.gz.sig 

This is the last planned v2.2.x release. We didn't fix everything reported 
(especially build changes) to try to minimize any unexpected breakages.

v2.3.2 will be out with a lot of fixes hopefully in a few weeks. That will 
start becoming the recommended version to run then.

 * login-proxy: If ssl_require_crl=no, allow revoked certificates.
   Also don't do CRL checks for incoming client certificates.
 * stats plugin: Don't temporarily enable PR_SET_DUMPABLE while opening
   /proc/self/io. This may still cause security problems if the process
   is ptrace()d at the same time. Instead, open it while still running
   as root.

 + doveadm: Added mailbox cache decision commands. See
   doveadm-mailbox(1) man page for details.
 + doveadm: Added rebuild attachments command for rebuilding
   $HasAttachment or $HasNoAttachment flags for matching mails. See
   doveadm-rebuild(1) man page for details.
 + cassandra: Use fallback_consistency on more types of errors
 - cassandra: Fix consistency=quorum to work
 - dsync: Lock file generation failed if home directory didn't exist
 - In some configs if namespace root directory didn't yet exist, Dovecot
   failed to create mailboxes.lock when trying to create mailboxes
 - Snippet generation for HTML mails didn't ignore  inside
   blockquotes, producing strange looking snippets.
 - imapc: Fix assert-crash if getting disconnected and after
   reconnection all mails in the selected mailbox are gone.
 - pop3c: Handle unexpected server disconnections without assert-crash
 - fts: Fixes to indexing mails via virtual mailboxes.
 - fts: If mails contained NUL characters, the text around it wasn't
   indexed.
 - Obsolete dovecot.index.cache offsets were sometimes used. Trying to
   fetch a field that was just added to cache file may not have always
   found it.
 - dict-sql: Fix crash when reading NULL value from database

___
Dovecot-news mailing list
Dovecot-news@dovecot.org
https://dovecot.org/mailman/listinfo/dovecot-news


v2.2.36 released

2018-05-23 Thread Timo Sirainen
https://dovecot.org/releases/2.2/dovecot-2.2.36.tar.gz
https://dovecot.org/releases/2.2/dovecot-2.2.36.tar.gz.sig 

This is the last planned v2.2.x release. We didn't fix everything reported 
(especially build changes) to try to minimize any unexpected breakages.

v2.3.2 will be out with a lot of fixes hopefully in a few weeks. That will 
start becoming the recommended version to run then.

 * login-proxy: If ssl_require_crl=no, allow revoked certificates.
   Also don't do CRL checks for incoming client certificates.
 * stats plugin: Don't temporarily enable PR_SET_DUMPABLE while opening
   /proc/self/io. This may still cause security problems if the process
   is ptrace()d at the same time. Instead, open it while still running
   as root.

 + doveadm: Added mailbox cache decision commands. See
   doveadm-mailbox(1) man page for details.
 + doveadm: Added rebuild attachments command for rebuilding
   $HasAttachment or $HasNoAttachment flags for matching mails. See
   doveadm-rebuild(1) man page for details.
 + cassandra: Use fallback_consistency on more types of errors
 - cassandra: Fix consistency=quorum to work
 - dsync: Lock file generation failed if home directory didn't exist
 - In some configs if namespace root directory didn't yet exist, Dovecot
   failed to create mailboxes.lock when trying to create mailboxes
 - Snippet generation for HTML mails didn't ignore  inside
   blockquotes, producing strange looking snippets.
 - imapc: Fix assert-crash if getting disconnected and after
   reconnection all mails in the selected mailbox are gone.
 - pop3c: Handle unexpected server disconnections without assert-crash
 - fts: Fixes to indexing mails via virtual mailboxes.
 - fts: If mails contained NUL characters, the text around it wasn't
   indexed.
 - Obsolete dovecot.index.cache offsets were sometimes used. Trying to
   fetch a field that was just added to cache file may not have always
   found it.
 - dict-sql: Fix crash when reading NULL value from database



Re: Upgrading dovecot 2.2 to 2.3 without downtime when using proxy/director?

2018-05-15 Thread Timo Sirainen
On 15 May 2018, at 12.06, Timo Sirainen <t...@iki.fi> wrote:
> 
> If you look at .176's error log, do you see an error about 
> "director_consistent_hashing settings differ between directors"? Have you set 
> director_consistent_hashing=yes in the old directors? That is needed now, 
> because the old non-consistent-hashing method is obsoleted. Unfortunately 
> there's no easy way to upgrade directors to use the consistent hashing method 
> without stopping the entire ring. The hard way would be to build a secondary 
> director ring and start moving users to that ring in proxies.

Added https://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy#moving 
<https://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy#moving> to 
explain further the hard way.



  1   2   3   4   5   6   7   8   9   10   >