Broken uids: Director lmtp proxy able to change the destination address?
Hi, Director -> Backends -> NFS setup. Problem: Broken dovecot-uidlists because of our postfix LMTP delivery using a different address than the IMAP client lookups Error: Mailbox INBOX: Broken file dovecot-uidlist line 169: Invalid data: example: LMTP delivery: lmtp(info#domain.nl@mail.local) IMAP lookup: i...@domain.nl doveadm director status info#domain.nl@mail.local Current: 10.0.10.223 (expires 2019-11-01 08:52:42) Hashed: 10.0.10.223 Initial config: 10.0.10.223 doveadm director status i...@domain.nl Current: 10.0.10.113 (expires 2019-11-01 13:47:27) Hashed: 10.0.10.113 Initial config: 10.0.10.113 On the backends these would write to the same NFS maildir: Now Director would assign these addresses to different backend servers and thus result in the expected corrupted control files. at this time i'm unable to reconfigure the Postfix LMTP delivery in a way that these use the same lookup format. Questions: 1. Disable LMTP updating dovecot-uidlist and indexes. Perhaps using Maildir:CONTROL=/dev/null :INDEX=MEMORY for lmtp-sql.conf? 2. Since i'm using director/proxy is it possible to rewrite the address using SQL in the passdb query? 3. What would be the best settings to avoid corruption in this broken setup. backend.conf Description: Binary data director.conf Description: Binary data
Re: Coredump v2.3.8 specific msg fetch, corrupted record in index cache, Broken physical size
Hi, maildir_broken_filename_sizes = yes Does prevent the crash coredump On Fri, Oct 18, 2019 at 3:36 PM Erik de Waard wrote: > Hi, i'm getting a coredump on a specific msg, i've attached the gdb. > > file on disk i noticed W= is missing. > 1571209735.M744550P1608.rwvirtual65,S=15886:2,S > > Best regards, > > > mail.log > Oct 18 14:41:39 rwvirtual10 dovecot: > imap(john...@company.nl)<15868>: > Error: Mailbox INBOX.Debug: UID=1041: read(/data/mail/ > company.nl/users/johndoe/Maildir/.Debug/cur/1571209735.M744550P1608.rwvirtual65,S=15886:2,S) > failed: Cached message size smaller than expected (15886 < 16367, > box=INBOX.Debug, UID=1041) (read reason=mail stream) > Oct 18 14:41:39 rwvirtual10 dovecot: > imap(john...@company.nl)<15868>: > Error: Corrupted record in index cache file /data/indexes/ > john...@company.nl/.Debug/dovecot.index.cache: UID 1041: Broken physical > size in mailbox INBOX.Debug: read(/data/mail/ > company.nl/users/johndoe/Maildir/.Debug/cur/1571209735.M744550P1608.rwvirtual65,S=15886:2,S) > failed: Cached message size smaller than expected (15886 < 16367, > box=INBOX.Debug, UID=1041) > Oct 18 14:41:39 rwvirtual10 dovecot: > imap(john...@company.nl)<15868>: > Panic: file istream.c: line 315 (i_stream_read_memarea): assertion failed: > (old_size <= _stream->pos - _stream->skip) > Oct 18 14:41:39 rwvirtual10 dovecot: > imap(john...@company.nl)<15868>: > Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xe5c31) > [0x7f5237d8cc31] -> /usr/lib/dovecot/libdovecot.so.0(+0xe5cd1) > [0x7f5237d8ccd1] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) > [0x7f5237ce4103] -> > /usr/lib/dovecot/libdovecot.so.0(i_stream_read_memarea+0x300) > [0x7f5237d9abd0] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x35) > [0x7f5237d9aca5] -> > /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f5237d9b64d] > -> /usr/lib/dovecot/libdovecot.so.0(+0xc4da1) [0x7f5237d6bda1] -> > /usr/lib/dovecot/libdovecot.so.0(+0xc5d4b) [0x7f5237d6cd4b] -> > /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x65) > [0x7f5237d6da05] -> > /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_body+0x31) > [0x7f5237d6dcf1] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xc19d4) > [0x7f52381099d4] -> > /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x3aa) > [0x7f5238109e9a] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x74b25) > [0x7f52380bcb25] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x70) > [0x7f52380833a0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x748d7) > [0x7f52380bc8d7] -> > /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_virtual_size+0x44) > [0x7f5238082ea4] -> dovecot/imap [john...@company.nl 10.0.10.36 > FETCH](+0x21523) [0x561f6425f523] -> dovecot/imap [john...@company.nl > 10.0.10.36 FETCH](+0x1f759) [0x561f6425d759] -> dovecot/imap [ > john...@company.nl 10.0.10.36 FETCH](imap_fetch_more+0x39) > [0x561f6425eb99] -> dovecot/imap [john...@company.nl 10.0.10.36 > FETCH](cmd_fetch+0x342) [0x561f6424ef32] -> dovecot/imap [ > john...@company.nl 10.0.10.36 FETCH](command_exec+0x7c) [0x561f6425bb0c] > -> dovecot/imap [john...@company.nl 10.0.10.36 FETCH](+0x1bb72) > [0x561f64259b72] -> dovecot/imap [john...@company.nl 10.0.10.36 > FETCH](+0x1bc04) [0x561f64259c04] -> dovecot/imap [john...@company.nl > 10.0.10.36 FETCH](client_handle_input+0x1a5) [0x561f6425a025] -> > dovecot/imap [john...@company.nl 10.0.10.36 FETCH](client_input+0x82) > [0x561f6425a6c2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x75) > [0x7f5237da6845] -> > /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x11b) > [0x7f5237da825b] -> > /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x59) [0x7f5237da6959] > Oct 18 14:41:39 rwvirtual10 dovecot: > imap(john...@company.nl)<15868>: > Fatal: master: service(imap): child 15868 killed with signal 6 (core dumps > disabled - https://dovecot.org/bugreport.html#coredumps) > > >
Coredump v2.3.8 specific msg fetch, corrupted record in index cache, Broken physical size
Hi, i'm getting a coredump on a specific msg, i've attached the gdb.
file on disk i noticed W= is missing.
1571209735.M744550P1608.rwvirtual65,S=15886:2,S
Best regards,
mail.log
Oct 18 14:41:39 rwvirtual10 dovecot:
imap(john...@company.nl)<15868>:
Error: Mailbox INBOX.Debug: UID=1041: read(/data/mail/
company.nl/users/johndoe/Maildir/.Debug/cur/1571209735.M744550P1608.rwvirtual65,S=15886:2,S)
failed: Cached message size smaller than expected (15886 < 16367,
box=INBOX.Debug, UID=1041) (read reason=mail stream)
Oct 18 14:41:39 rwvirtual10 dovecot:
imap(john...@company.nl)<15868>:
Error: Corrupted record in index cache file /data/indexes/
john...@company.nl/.Debug/dovecot.index.cache: UID 1041: Broken physical
size in mailbox INBOX.Debug: read(/data/mail/
company.nl/users/johndoe/Maildir/.Debug/cur/1571209735.M744550P1608.rwvirtual65,S=15886:2,S)
failed: Cached message size smaller than expected (15886 < 16367,
box=INBOX.Debug, UID=1041)
Oct 18 14:41:39 rwvirtual10 dovecot:
imap(john...@company.nl)<15868>:
Panic: file istream.c: line 315 (i_stream_read_memarea): assertion failed:
(old_size <= _stream->pos - _stream->skip)
Oct 18 14:41:39 rwvirtual10 dovecot:
imap(john...@company.nl)<15868>:
Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xe5c31)
[0x7f5237d8cc31] -> /usr/lib/dovecot/libdovecot.so.0(+0xe5cd1)
[0x7f5237d8ccd1] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0)
[0x7f5237ce4103] ->
/usr/lib/dovecot/libdovecot.so.0(i_stream_read_memarea+0x300)
[0x7f5237d9abd0] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x35)
[0x7f5237d9aca5] ->
/usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f5237d9b64d]
-> /usr/lib/dovecot/libdovecot.so.0(+0xc4da1) [0x7f5237d6bda1] ->
/usr/lib/dovecot/libdovecot.so.0(+0xc5d4b) [0x7f5237d6cd4b] ->
/usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x65)
[0x7f5237d6da05] ->
/usr/lib/dovecot/libdovecot.so.0(message_parser_parse_body+0x31)
[0x7f5237d6dcf1] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xc19d4)
[0x7f52381099d4] ->
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x3aa)
[0x7f5238109e9a] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x74b25)
[0x7f52380bcb25] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x70)
[0x7f52380833a0] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x748d7)
[0x7f52380bc8d7] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_virtual_size+0x44)
[0x7f5238082ea4] -> dovecot/imap [john...@company.nl 10.0.10.36
FETCH](+0x21523) [0x561f6425f523] -> dovecot/imap [john...@company.nl
10.0.10.36 FETCH](+0x1f759) [0x561f6425d759] -> dovecot/imap [
john...@company.nl 10.0.10.36 FETCH](imap_fetch_more+0x39) [0x561f6425eb99]
-> dovecot/imap [john...@company.nl 10.0.10.36 FETCH](cmd_fetch+0x342)
[0x561f6424ef32] -> dovecot/imap [john...@company.nl 10.0.10.36
FETCH](command_exec+0x7c) [0x561f6425bb0c] -> dovecot/imap [
john...@company.nl 10.0.10.36 FETCH](+0x1bb72) [0x561f64259b72] ->
dovecot/imap [john...@company.nl 10.0.10.36 FETCH](+0x1bc04)
[0x561f64259c04] -> dovecot/imap [john...@company.nl 10.0.10.36
FETCH](client_handle_input+0x1a5) [0x561f6425a025] -> dovecot/imap [
john...@company.nl 10.0.10.36 FETCH](client_input+0x82) [0x561f6425a6c2] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x75) [0x7f5237da6845] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x11b)
[0x7f5237da825b] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x59) [0x7f5237da6959]
Oct 18 14:41:39 rwvirtual10 dovecot:
imap(john...@company.nl)<15868>:
Fatal: master: service(imap): child 15868 killed with signal 6 (core dumps
disabled - https://dovecot.org/bugreport.html#coredumps)
# 2.3.8 (9df20d2db): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.8 (b7b03ba2)
# OS: Linux 4.9.0-9-amd64 x86_64 Debian 9.11
# Hostname: rwvirtual10.colo.company.nl
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_cache_ttl = 1 days
auth_username_chars =
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#"
default_client_limit = 1500
default_vsz_limit = 1 G
disable_plaintext_auth = no
info_log_path = /var/log/mail.debug
listen = *
log_timestamp = "%Y-%m-%d %H:%M:%S "
login_trusted_networks = 10.0.10.0/24
mail_debug = yes
mail_fsync = never
mail_max_userip_connections = 100
mail_prefetch_count = 10
mail_privileged_group = mail
mmap_disable = yes
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix = INBOX.
separator = .
type = private
}
passdb {
args = username_format=%{rip} /etc/dovecot/passdb
default_fields = noauthenticate=y
driver = passwd-file
}
passdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
plugin {
sieve_execute_bin_dir = /etc/dovecot/sieve-executables
sieve_global_extensions = +vnd.dovecot.execute
ITERINDEX not showing subfolders.
# 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf
Hello, we're using mail on NFS and the indexes on localdisk, with a
director setup. (18K+ users) now to get more performance i thought to use
the ITERINDEX option. however this has the side-effect on some mailboxes
that it doesn't show the subfolders under INBOX.
they seem 'disappeared'. Removing the ITERINDEX option fixes this again.
Deleting the local indexes to be recreated doesn't solve the problem.
What could cause this? and how to fix it.
# 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 ()
# OS: Linux 4.9.0-11-amd64 x86_64 Debian 9.11
# Hostname: somemail.host.domain
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_cache_ttl = 1 days
auth_username_chars =
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#"
default_client_limit = 1500
default_vsz_limit = 1 G
disable_plaintext_auth = no
info_log_path = /dev/null
listen = *
log_timestamp = "%Y-%m-%d %H:%M:%S "
login_trusted_networks = 10.0.10.0/24
mail_fsync = never
mail_max_userip_connections = 100
mail_prefetch_count = 10
mail_privileged_group = mail
mailbox_list_index_include_inbox = yes
mmap_disable = yes
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix = INBOX.
separator = .
type = private
}
passdb {
args = username_format=%{rip} /etc/dovecot/passdb
default_fields = noauthenticate=y
driver = passwd-file
}
passdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
plugin {
sieve_execute_bin_dir = /etc/dovecot/sieve-executables
sieve_global_extensions = +vnd.dovecot.execute
sieve_plugins = sieve_extprograms
}
protocols = imap lmtp
service anvil {
unix_listener anvil-auth-penalty {
mode = 0600
}
}
service auth {
user = root
}
service imap-login {
client_limit = 6000
process_limit = 4
process_min_avail = 4
service_count = 0
vsz_limit = 600 M
}
service imap {
client_limit = 1
process_limit = 1024
service_count = 50
}
service lmtp {
inet_listener lmtp {
port = 24
}
}
ssl = no
ssl_cert =
Re: 4sec latency when using director
ok, i've found out that the directors are in the doveadm penalty list.
i believe adding the following solves the issue (still testing but looking
promising.
login_trusted_networks = 10.0.10.0/24
On Tue, Mar 19, 2019 at 2:19 PM Erik de Waard wrote:
> Hi,
>
> I've been running dovecot for many years, but now i've hit a strange
> problem.
> when retrieving imap there is sometimes in 1 of 4 imap requests a 4sec
> latency.
>
> * notes:
> - connected directly to the backends this latency disappears
> - removing a director from the loadbalancer(lvs) so i'm the only connected
> to the director
> this latency disappears too
>
> I would appreciate some feedback of where to look because i tried various
> options. (client_limit,process_limit) perhaps its just as simple as to add
> more directors
> but any feedback would be welcome.
>
> Thank you.
>
> This is the setup:
>
> -- Internet -> LVS -> 3 Directors -> 9 Backends.
>
> doveadm director status
> mail server ip tag vhosts state state changed users
>
> 10.0.10.110100up- 3309
>
> 10.0.10.111100up- 3412
>
> 10.0.10.112100up- 3458
>
> 10.0.10.113100up- 3437
>
> 10.0.10.115100up- 3368
>
> 10.0.10.116100up- 3320
>
> 10.0.10.117100up- 3305
>
> 10.0.10.118100up- 3291
>
> 10.0.10.223100up- 3280
>
> #Director doveconf -n
> # Pigeonhole version 0.4.22 (22940fb7)
> # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.4
> # Hostname: server312.company.com
> auth_cache_negative_ttl = 0
> auth_cache_size = 10 M
> auth_cache_ttl = 1 days
> auth_username_chars =
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#"
> default_client_limit = 3000
> director_mail_servers = 10.0.10.223 10.0.10.110 10.0.10.111 10.0.10.112
> 10.0.10.113 10.0.10.115 10.0.10.116 10.0.10.117 10.0.10.118
> director_servers = 10.0.10.114 10.0.10.181 10.0.10.182
> director_user_expire = 1 days
> disable_plaintext_auth = no
> info_log_path = /dev/null
> lmtp_proxy = yes
> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e
> encryption=%k secured=%c
> mail_max_userip_connections = 100
> passdb {
> args = proxy=y nopassword=y
> driver = static
> }
> protocols = imap lmtp
> service director {
> fifo_listener login/proxy-notify {
> mode = 0600
> user = $default_login_user
> }
> inet_listener {
> port = 9090
> }
> unix_listener director-userdb {
> mode = 0600
> }
> unix_listener login/director {
> mode = 0666
> }
> }
> service imap-login {
> client_limit = 6000
> executable = imap-login director
> process_limit = 4
> process_min_avail = 4
> service_count = 0
> vsz_limit = 600 M
> }
> service ipc {
> unix_listener ipc {
> user = dovecot
> }
> }
> service lmtp {
> inet_listener lmtp {
> port = 24
> }
> }
> ssl_cert = ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL:!RC4:!SEED@STRENGTH
> ssl_key = # hidden, use -P to show it
> ssl_prefer_server_ciphers = yes
> protocol lmtp {
> auth_socket_path = director-userdb
> }
>
>
> #backend doveconf -n
> # 2.2.34 (874deae): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.22 (22940fb7)
> # OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.4
> # Hostname: server250.company.com
> auth_cache_negative_ttl = 0
> auth_cache_size = 10 M
> auth_cache_ttl = 1 days
> auth_username_chars =
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#"
> default_client_limit = 1500
> default_vsz_limit = 600 M
> disable_plaintext_auth = no
> info_log_path = /dev/null
> listen = *
> log_timestamp = "%Y-%m-%d %H:%M:%S "
> mail_max_userip_connections = 100
> mail_privileged_group = mail
> mmap_disable = yes
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> special_use = \Drafts
> }
> mailbox Junk {
> special_use = \Junk
> }
> mailbox Sent {
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> special_use = \Trash
> }
> prefix = INBOX.
> separator = .
> type = private
> }
> passdb {
> args = /etc/dovecot/dovecot-sql.conf
> driver = sql
> }
> plugin {
> sieve_execute_bin_dir = /etc/dovecot/sieve-executables
> sieve_global_extensions = +vnd.dovecot.execute
> sieve_plugins = sieve_extprograms
> }
> protocols = imap lmtp
> service anvil {
> unix_listener anvil-auth-penalty {
> mode = 0600
> }
> }
> service auth {
> user = root
> }
> service imap-login {
> client_limit = 6000
> process_limit = 4
> process_min_avail = 4
> service_count = 0
> vsz_limit = 600 M
> }
> service imap {
> client_limit = 1
> process_limit = 1024
> service_count = 50
> }
> service lmtp {
> inet_listener lmtp {
> port = 24
> }
> }
> ssl = no
> ssl_cert = ssl_key = # hidden, use -P to show it
> userdb {
> driver =
4sec latency when using director
Hi,
I've been running dovecot for many years, but now i've hit a strange
problem.
when retrieving imap there is sometimes in 1 of 4 imap requests a 4sec
latency.
* notes:
- connected directly to the backends this latency disappears
- removing a director from the loadbalancer(lvs) so i'm the only connected
to the director
this latency disappears too
I would appreciate some feedback of where to look because i tried various
options. (client_limit,process_limit) perhaps its just as simple as to add
more directors
but any feedback would be welcome.
Thank you.
This is the setup:
-- Internet -> LVS -> 3 Directors -> 9 Backends.
doveadm director status
mail server ip tag vhosts state state changed users
10.0.10.110100up- 3309
10.0.10.111100up- 3412
10.0.10.112100up- 3458
10.0.10.113100up- 3437
10.0.10.115100up- 3368
10.0.10.116100up- 3320
10.0.10.117100up- 3305
10.0.10.118100up- 3291
10.0.10.223100up- 3280
#Director doveconf -n
# Pigeonhole version 0.4.22 (22940fb7)
# OS: Linux 4.9.0-6-amd64 x86_64 Debian 9.4
# Hostname: server312.company.com
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_cache_ttl = 1 days
auth_username_chars =
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#"
default_client_limit = 3000
director_mail_servers = 10.0.10.223 10.0.10.110 10.0.10.111 10.0.10.112
10.0.10.113 10.0.10.115 10.0.10.116 10.0.10.117 10.0.10.118
director_servers = 10.0.10.114 10.0.10.181 10.0.10.182
director_user_expire = 1 days
disable_plaintext_auth = no
info_log_path = /dev/null
lmtp_proxy = yes
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e
encryption=%k secured=%c
mail_max_userip_connections = 100
passdb {
args = proxy=y nopassword=y
driver = static
}
protocols = imap lmtp
service director {
fifo_listener login/proxy-notify {
mode = 0600
user = $default_login_user
}
inet_listener {
port = 9090
}
unix_listener director-userdb {
mode = 0600
}
unix_listener login/director {
mode = 0666
}
}
service imap-login {
client_limit = 6000
executable = imap-login director
process_limit = 4
process_min_avail = 4
service_count = 0
vsz_limit = 600 M
}
service ipc {
unix_listener ipc {
user = dovecot
}
}
service lmtp {
inet_listener lmtp {
port = 24
}
}
ssl_cert =
