Re: mail-search backtrace

[Hugh Bragg]

On 22/05/16 05:17, Hugh Bragg wrote:



On 13/04/16 06:41, Timo Sirainen wrote:

On 09 Apr 2016, at 21:48, Hugh Bragg  wrote:

I'm repeatedly getting this error:

Apr 07 04:37:27 imap(mymail@address): Panic: file mail-search.c: 
line 84 (mail_search_arg_init): assertion failed: 
(arg->initialized.keywords == NULL)
Apr 07 04:37:27 imap(mymail@address): Error: Raw backtrace: 
/usr/lib64/dovecot/libdovecot.so.0(+0x827c2) [0x7fcb7f65e7c2] -> 
/usr/lib64/dovecot/libdovecot.so.0(+0x828ad) [0x7fcb7f65e8ad] -> 
/usr/lib64/dov
ecot/libdovecot.so.0(i_fatal+0) [0x7fcb7f605b01] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_search_arg_init+0x228) 
[0x7fcb7f91a328] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(index_search_result_updat
e_flags+0x100) [0x7fcb7f98e470] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(index_sync_search_results_update+0x52) 
[0x7fcb7f9983e2] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(index_mailbox_sync_deinit+0x185
) [0x7fcb7f998bb5] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_deinit+0x32) 
[0x7fcb7f921222] -> 
/usr/lib64/dovecot/lib20_virtual_plugin.so(virtual_storage_sync_init+0xab3) 
[0x7fcb7e9f7313] -> /usr

It's coming from virtual mailboxes.


namespace virtual {
  location = virtual:/var/mail/vhosts/%d/%n/virtual
  prefix = virtual.
  separator = .
}
What do your dovecot-virtual files contain? I guess opening one of 
those virtual mailboxes crashes always. Related to searching keywords.
It still happens once in a while. It just won't expunge old messages 
from unseen. There is no other trace or log message.
I was hoping to isolate the cause, but all I could only say for sure 
that it happens sometime after Dovecot first starts up and I have to 
restart to fix it.

dovecot-virtual files look like this:
# cat virtual/all/dovecot-virtual
*
  all
# cat virtual/Unseen/dovecot-virtual
virtual.all
  inthread refs unseen


A fresh trace:

May 21 00:28:08 imap(x@y): Panic: file mail-search.c: line 84 
(mail_search_arg_init): assertion failed: (arg->initialized.keywords 
== NULL)
May 21 00:28:08 imap(x@y): Error: Raw backtrace: 
/usr/lib64/dovecot/libdovecot.so.0(+0x85c62) [0x7f4fd8915c62] -> 
/usr/lib64/dovecot/libdovecot.so.0(+0x85d4d) [0x7f4fd8915d4d] -> 
/usr/lib64/dov
ecot/libdovecot.so.0(i_fatal+0) [0x7f4fd88ba5c1] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_search_arg_init+0x228) 
[0x7f4fd8bd4b78] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(index_search_result_updat
e_flags+0x100) [0x7f4fd8c49d00] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(index_sync_search_results_update+0x52) 
[0x7f4fd8c53ce2] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(index_mailbox_sync_deinit+0x185
) [0x7f4fd8c544b5] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_deinit+0x32) 
[0x7f4fd8bdba82] -> 
/usr/lib64/dovecot/lib20_virtual_plugin.so(virtual_storage_sync_init+0x538) 
[0x7f4fd7caa428] -> /usr
/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x3b) 
[0x7f4fd8bdb9fb] -> dovecot/imap(imap_sync_init+0x68) [0x56091d93b078] 
-> dovecot/imap(+0x1210e) [0x56091d92710e] -> dovecot/imap(+0x1234d) 
[0x56091
d92734d] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xea) 
[0x7f4fd892984a] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbb) 
[0x7f4fd892ae4b] -> /usr/lib64/dovecot/libdo
vecot.so.0(io_loop_handler_run+0x25) [0x7f4fd8929a75] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f4fd8929c18] 
-> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f4fd88c0123] -> d
ovecot/imap(main+0x328) [0x56091d922a98] -> 
/lib64/libc.so.6(__libc_start_main+0xf0) [0x7f4fd84ef580] -> 
dovecot/imap(_start+0x29) [0x56091d922c19]


Still no clue on this even with debug set on. It's become so bad I need 
to restart it or new mail is no longer reported after a few days when 
the unseen has dozens of read mails.
I've no idea why it would need the keyword when I haven't done a search 
but I suppose the virtual plugin works by using the mail-search. Still, 
this shouldn't cause an error even if it is null. I'm suppose it could 
be caused by the number of emails being so great. Perhaps something is 
corrupt but as given, my dovecot-virtual files are as recommended by the 
plugin doco and nothing else seems amiss. If there is a corrupt mail or 
something then I don't know how to trace it.


Anything anyone?


A fresh trace :
Jun 25 15:10:30 imap(x@y.z): Panic: file mail-search.c: line 84 
(mail_search_arg_init): assertion failed: (arg->initialized.keywords == 
NULL)
Jun 25 15:10:30 imap(x@y.z): Error: Raw backtrace: 
/usr/lib64/dovecot/libdovecot.so.0(+0x87102) [0x7fcb73696102] -> 
/usr/lib64/dovecot/libdovecot.so.0(+0x871ed) [0x7fcb736961ed] -> 
/usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7fcb736399e1] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_search_arg_init+0x228) 
[0x7fcb73955cc8] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(index_search_result_update_flags+0x100) 

Re: Postfix and Dovecot LDA vs. LMTP

[Jan Büren]

Hi,

> But you can easily grasp the configuration details and reverse engineer
> the technical german phrases ...
Ah well, the link:
http://www.dovecot-buch.de/buch/vorwort-timo-sirainen/
>
>
>>
>>
>>
>> Thanks much,
>>
>> Michael
>>
>>
>>
>>
>
>
> --
> kivitendo mit Schnelleinstieg zu RB-Druckvorlagen im Linux-Magazin 07
> DELUG-DVD Ausgabe
>
> Richardson & Büren GmbH
> Jan Büren
> Kölnstr. 311
> 53117 Bonn
>
> USt-IdNr. DE238288407
> Telefon: 0228 92 98 2012
>
>
> Durchwahl: 0228 92 97 8965
>
>


-- 
kivitendo mit Schnelleinstieg zu RB-Druckvorlagen im Linux-Magazin 07
DELUG-DVD Ausgabe

Richardson & Büren GmbH
Jan Büren
Kölnstr. 311
53117 Bonn

USt-IdNr. DE238288407
Telefon: 0228 92 98 2012


Durchwahl: 0228 92 97 8965

Re: Postfix and Dovecot LDA vs. LMTP

[aki . tuomi]

The most crucial difference is that LDA is intended for delivering email to a 
*real* user.

Aki

> On June 24, 2016 at 7:59 PM Jan Büren  wrote:
> 
> 
> Hi Michael,
> 
> > I'd appreciate comments from experienced users of postfix with dovecot.
> > Are
> > you using Dovecot LDA or LMTP and why?
> I have LMTP with dovecot running on Ubuntu 14.04 and Ubuntu 16.04.
> 
> LDA is the worser solution, this is best explained in chapter LTMP in
> Peers dovecot book, which is unluckily in german and more or less out of
> print.
> 
> But you can easily grasp the configuration details and reverse engineer
> the technical german phrases ...
> 
> 
> >
> >
> >
> > Thanks much,
> >
> > Michael
> >
> >
> >
> >
> 
> 
> -- 
> kivitendo mit Schnelleinstieg zu RB-Druckvorlagen im Linux-Magazin 07
> DELUG-DVD Ausgabe
> 
> Richardson & Büren GmbH
> Jan Büren
> Kölnstr. 311
> 53117 Bonn
> 
> USt-IdNr. DE238288407
> Telefon: 0228 92 98 2012
> 
> 
> Durchwahl: 0228 92 97 8965

Re: Postfix and Dovecot LDA vs. LMTP

[Jan Büren]

Hi Michael,

> I'd appreciate comments from experienced users of postfix with dovecot.
> Are
> you using Dovecot LDA or LMTP and why?
I have LMTP with dovecot running on Ubuntu 14.04 and Ubuntu 16.04.

LDA is the worser solution, this is best explained in chapter LTMP in
Peers dovecot book, which is unluckily in german and more or less out of
print.

But you can easily grasp the configuration details and reverse engineer
the technical german phrases ...


>
>
>
> Thanks much,
>
> Michael
>
>
>
>


-- 
kivitendo mit Schnelleinstieg zu RB-Druckvorlagen im Linux-Magazin 07
DELUG-DVD Ausgabe

Richardson & Büren GmbH
Jan Büren
Kölnstr. 311
53117 Bonn

USt-IdNr. DE238288407
Telefon: 0228 92 98 2012


Durchwahl: 0228 92 97 8965

Postfix and Dovecot LDA vs. LMTP

[Michael Fox]

I'm new to Dovecot and will be using it with Postfix.  I'm looking for
recommendations regarding the use of Dovecot's LDA or LMTP for virtual
mailbox delivery.

 

Many of the simple examples on the wiki use LDA.  So I've set that up
initially.  But apparently an advantage of LMTP is recipient verification.
So, as I understand it, LMTP would let Postfix know whether or not the
message was deliverable to a local virtual recipient without needing to have
a separate virtual recipients map in Postfix.  That sounds like a nice
simplification.

 

But I see in Ubuntu that the dovecot-lmtp package is not marked with the
Canonical support icon, like the pop, imap, and other packages are.  I don't
have a contract with Canonical.  But I'm wondering why they would not
support the lmtp package when they do support most of the others.  Is it
possible that the dovecot LMTP package is not as stable or reliable?

 

I'd appreciate comments from experienced users of postfix with dovecot.  Are
you using Dovecot LDA or LMTP and why?

 

Thanks much,

Michael

 

exempt local auth-client UNIX socket from failed login penalty // add to login_trusted_networks ?

[Steffen Kaiser]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

I'm using Dovecot v2.2 with  unix_listener auth-client {
} to verify passwords for a different service. However, it looks like that 
auth_failure_delay effects all connects going through that socket.


I mean:

connect /var/run/dovecot2.2/auth-client
attempt bad auth
2s penalty
NO
disconnect
==> Note, it's another connection almost immediately following each
connect /var/run/dovecot2.2/auth-client
attempt good auth
2s penalty
OK
disconnect

Can I disable auth_failure_delay for local UNIX sockets?
How do I add it to login_trusted_networks?

- -- 
Steffen Kaiser

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEVAwUBV20MbHz1H7kL/d9rAQKm1AgAiVjjSimUTapEbhqHwZzfQWLzcJlkfm2W
z5smziGbVELYb0/COPd84GK6wyUF7+3iRZOuVPhLRdljhB72PWRe+hHX3KgMWHr2
1o5WLkX+0cGEwSXMvJ2w3ee/zmxYxva2WI+PjSzkgvvhxGMtnIxO7mMglEV5zbbq
ZxJcC1Ba4T9qpUhRIw3EQ5VPRs4cnLBz3Im4IDGLduWAGJYf/Rrxh+x+k3IqMtzb
v92ErjgQtz5bN+bgEtQ8C33UehoZeZ93eA3V8o/OiwZPtWyneWL6Yqwxni4LjtLI
R4wuu0N6Ea/BbA/fsElquRer0bXH2Zkt5mckJpDG6Rbe/IO5WYXq0A==
=axyE
-END PGP SIGNATURE-

Re: auth_bind with "()" in username not working

[Matthias Lay]

Hi again,

did some more tseting on this.

I think the problem is the ldap userlookup, where "("s are evil and
have to be quoted, but these quotes should be removed for the bind
request.

I get my usernames from ldap with a filter like this

user_filter = (sAMAccountName=%Ln)

so I think in between this to steps is the problem.

For testing I hard coded the username for auth_bind and compared strace
output from the auth process


auth_bind_userdn = "spdev\\claasc (test)"


this works fine. strace output from imap login


write(26, "0+\2\1\2`&\2\1\3\4\23spdev\\claasc
(test)\200\fHubertHans99", 45) 


compared to

auth_bind_userdn = "spdev\\%Ln"

which gives

write(26, "0-\2\1\2`(\2\1\3\4\25spdev\\claasc
\\(test\\)\200\fHubertHans99", 47)

and wrong credentials


nobody else encountering similar problems? maybe the "()" are the only
chars making problems at this point


Greetz Matze

Re: Authentication Penalty with ID x-originating-ip, HAproxy

[Tobias]

A quick test confirms that HAproxy header IP information does properly 
delay the authentication failures upon successive failed login attempts 
from the same IP.


And furthermore if the webmail client is delayed on the IMAP level, this 
could potentially be exploited for DoS and as such may not be a good 
idea after all. Even with the auth_failure_delay=2 by default this is 
possible, but it's much easier to achieve the DoS if the pre-auth delay 
increases to 17 seconds (maximum delay I've observed).


Is there any other brute force / DoS mitigation option for dovecot / 
webmail interaction, short of fail2ban type IP blocking in a firewall 
(which will not work on a machine several layers deep behind e.g. a 
proxy), that isn't exclusively relying on the webmail client for such 
mitigation?


Can dovecot itself temp-ban remote IPs (as reported by HAproxy protocol, 
or IMAP ID x-originating-ip), perhaps with a notice to try again in X 
seconds, instead of delaying them?


/Tobias

On 2016-06-24 13:27, Tobias wrote:

The wiki states that anvil's authentication penalties are skipped when
IP is in login_trusted_networks.
http://wiki.dovecot.org/Authentication/Penalty

Is there a way to enable the authentication penalties for specific
advertised remote IPs, when the connecting IP is in
"login_trusted_networks", and it advertises the originating remote IP
via 'ID ("x-originating-ip", "")'?

And with regards to HAproxy, is anvil's authentication penalties by
default transparent with regards to the remote IP advertised in the
proxy protocol header?

/Tobias