replication and .dovecot.lda-dupes
Hi list,
this question was already posted a few years ago
(https://www.dovecot.org/list/dovecot/2014-November/098585.html). I
already asked the original queriest and he told me, that he never got an
solution or workaround but it was not important enough for him.
When using replication in conjunction with sieve vacations, the
.dovecot.lda-dupes file is not synced with the other server. So when
delivering to both servers (round-robin or randomized), senders might
get more vacation mails than configured as the other server does not
know, that the first one already sent a vacation message.
Is this a bug or intentional? If it is a bug, I hereby ask for a fix,
please.
We are using Dovecot version 2.2.27 on Debian/stretch.
This is dovecot -n (hostnames anonymized):
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.76.1.amd64-smp x86_64 Debian 9.3
auth_verbose = yes
default_vsz_limit = 2 G
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
listen = *
login_log_format_elements = pid=%p user=<%u> method=%m rip=%r lip=%l
mpid=%e %c
mail_attachment_dir = /IMAP/mail/attachments
mail_attachment_fs = sis-queue /IMAP/mail/attachments/queue:posix
mail_home = /IMAP/mail/mailboxes/%u
mail_location = mdbox:~/mdbox
mail_log_prefix = "%s(%u[%p]): "
mail_max_userip_connections = 0
mail_plugins = " notify replication zlib fts fts_squat"
maildir_stat_dirs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/ldap.conf
driver = ldap
}
plugin {
fts = squat
fts_autoindex = yes
fts_squat = partial=4 full=10
mail_replica = tcp:other-server
sieve = file:~/sieve;active=~/.dovecot.sieve
zlib_save = gz
zlib_save_level = 3
}
postmaster_address = <>
protocols = " imap lmtp sieve"
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
}
unix_listener replication-notify {
mode = 0666
}
}
service anvil {
client_limit = 2250
}
service auth {
client_limit = 2447
}
service doveadm {
inet_listener doveadm-server {
port = 12345
}
}
service imap-login {
inet_listener imap {
port = 0
}
process_limit = 2047
}
service imap {
process_limit = 2047
}
service lmtp {
inet_listener lmtp {
port = 24
}
}
service pop3-login {
inet_listener pop3 {
port = 0
}
inet_listener pop3s {
port = 0
}
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl_cert = +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute fuer Informatik & Softwaresysteme
smime.p7s
Description: S/MIME Cryptographic Signature
Re: replication and .dovecot.lda-dupes
Hi list,
it's been 2 months now since my initial posting (s.b.). I wonder if I
could get at least a "still working on it" statement from the devs or
something like that?
On 22.02.2018 16:42, Patrick Cernko wrote:
Hi list,
this question was already posted a few years ago
(https://www.dovecot.org/list/dovecot/2014-November/098585.html). I
already asked the original queriest and he told me, that he never got an
solution or workaround but it was not important enough for him.
When using replication in conjunction with sieve vacations, the
.dovecot.lda-dupes file is not synced with the other server. So when
delivering to both servers (round-robin or randomized), senders might
get more vacation mails than configured as the other server does not
know, that the first one already sent a vacation message.
Is this a bug or intentional? If it is a bug, I hereby ask for a fix,
please.
We are using Dovecot version 2.2.27 on Debian/stretch.
This is dovecot -n (hostnames anonymized):
# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.76.1.amd64-smp x86_64 Debian 9.3
auth_verbose = yes
default_vsz_limit = 2 G
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
listen = *
login_log_format_elements = pid=%p user=<%u> method=%m rip=%r lip=%l
mpid=%e %c
mail_attachment_dir = /IMAP/mail/attachments
mail_attachment_fs = sis-queue /IMAP/mail/attachments/queue:posix
mail_home = /IMAP/mail/mailboxes/%u
mail_location = mdbox:~/mdbox
mail_log_prefix = "%s(%u[%p]): "
mail_max_userip_connections = 0
mail_plugins = " notify replication zlib fts fts_squat"
maildir_stat_dirs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/ldap.conf
driver = ldap
}
plugin {
fts = squat
fts_autoindex = yes
fts_squat = partial=4 full=10
mail_replica = tcp:other-server
sieve = file:~/sieve;active=~/.dovecot.sieve
zlib_save = gz
zlib_save_level = 3
}
postmaster_address = <>
protocols = " imap lmtp sieve"
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
}
unix_listener replication-notify {
mode = 0666
}
}
service anvil {
client_limit = 2250
}
service auth {
client_limit = 2447
}
service doveadm {
inet_listener doveadm-server {
port = 12345
}
}
service imap-login {
inet_listener imap {
port = 0
}
process_limit = 2047
}
service imap {
process_limit = 2047
}
service lmtp {
inet_listener lmtp {
port = 24
}
}
service pop3-login {
inet_listener pop3 {
port = 0
}
inet_listener pop3s {
port = 0
}
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl_cert =
Best regards,
--
Patrick Cernko <pcer...@mpi-klsb.mpg.de> +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute fuer Informatik & Softwaresysteme
smime.p7s
Description: S/MIME Cryptographic Signature
Re: replication dropped imap flags
Hi list, hi devs,
On 26.11.18 16:26, Patrick Cernko wrote:
Hi list,
I think, I found a bug in the replication setup, that drops (custom)
flags like Thunderbird labels. I have set up a simple 2 node setup to
reproduce and explain it:
Host adove.mpi-klsb.mpg.de and bdove.mpi-klsb.mpg.de (doveconf -n
attached) have dovecot-imapd (from repo.dovecot.org) installed on an
ext4 filesystem. They only have one account "test" via
/etc/dovecot/userdb ("test:testpassword:::Test User::/bin/bash:"). It's
mailbox contains one message, that is flagged with "$label1":
doveadm -f flow fetch -u test 'guid flags' ALL
guid=d1516a2b5c08fc5b541d5a350039 flags=\Recent $label1
Now I simulate a node reinstall on adove by uninstalling dovecot,
removing all remnants, reinstalling and configuring it again:
apt purge dovecot-core dovecot-imapd
rm -rf /var/vmail /var/lib/dovecot /etc/dovecot
apt install dovecot-imapd
# disable system auth
: > /etc/dovecot/conf.d/auth-system.conf.ext
# create /var/vmail
install -d -o nobody -g nogroup -m 700 /var/vmail
# create userdb
echo 'test:testpassword:::Test User::/bin/bash:' > /etc/dovecot/userdb
# create /etc/dovecot/local.conf
# not described here, the resulting doveconf -n is attached!
service dovecot restart
When I check the flags of the mail of user test now, the "$label1" is
disapeared on both machines, even after "doveadm replicator replicate -f
test":
doveadm -f flow fetch -u test 'guid flags' ALL
guid=d1516a2b5c08fc5b541d5a350039 flags=\Recent
Any hints what I am doing wrong here or is this really a bug?
it's been 2 weeks now, since my bug(?) report without any
reactions/answers. I wonder if anyone is working on this already or if
my report got lost or if I did something wrong on my side.
Regards,
--
Patrick Cernko +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute fuer Informatik & Softwaresysteme
smime.p7s
Description: S/MIME Cryptographic Signature
replication dropped imap flags
Hi list,
I think, I found a bug in the replication setup, that drops (custom)
flags like Thunderbird labels. I have set up a simple 2 node setup to
reproduce and explain it:
Host adove.mpi-klsb.mpg.de and bdove.mpi-klsb.mpg.de (doveconf -n
attached) have dovecot-imapd (from repo.dovecot.org) installed on an
ext4 filesystem. They only have one account "test" via
/etc/dovecot/userdb ("test:testpassword:::Test User::/bin/bash:"). It's
mailbox contains one message, that is flagged with "$label1":
doveadm -f flow fetch -u test 'guid flags' ALL
guid=d1516a2b5c08fc5b541d5a350039 flags=\Recent $label1
Now I simulate a node reinstall on adove by uninstalling dovecot,
removing all remnants, reinstalling and configuring it again:
apt purge dovecot-core dovecot-imapd
rm -rf /var/vmail /var/lib/dovecot /etc/dovecot
apt install dovecot-imapd
# disable system auth
: > /etc/dovecot/conf.d/auth-system.conf.ext
# create /var/vmail
install -d -o nobody -g nogroup -m 700 /var/vmail
# create userdb
echo 'test:testpassword:::Test User::/bin/bash:' > /etc/dovecot/userdb
# create /etc/dovecot/local.conf
# not described here, the resulting doveconf -n is attached!
service dovecot restart
When I check the flags of the mail of user test now, the "$label1" is
disapeared on both machines, even after "doveadm replicator replicate -f
test":
doveadm -f flow fetch -u test 'guid flags' ALL
guid=d1516a2b5c08fc5b541d5a350039 flags=\Recent
Any hints what I am doing wrong here or is this really a bug?
Best,
--
Patrick Cernko +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute fuer Informatik & Softwaresysteme
# 2.3.4 (0ecbaf23d): /etc/dovecot/dovecot.conf
# OS: Linux 4.14.65.1.amd64-smp x86_64 Debian 9.6
# Hostname: bdove.mpi-klsb.mpg.de
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
listen = *
mail_gid = nogroup
mail_home = /var/vmail/%u
mail_location = mdbox:~/mdbox
mail_plugins = notify replication
mail_uid = nobody
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/userdb
driver = passwd-file
}
plugin {
mail_replica = tcp:adove.mpi-klsb.mpg.de
}
protocols = " imap"
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
}
unix_listener replication-notify {
mode = 0666
}
}
service doveadm {
inet_listener doveadm-server {
port = 12345
}
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl = no
userdb {
args = /etc/dovecot/userdb
driver = passwd-file
}
# 2.3.4 (0ecbaf23d): /etc/dovecot/dovecot.conf
# OS: Linux 4.14.65.1.amd64-smp x86_64 Debian 9.6
# Hostname: adove.mpi-klsb.mpg.de
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
listen = *
mail_gid = nogroup
mail_home = /var/vmail/%u
mail_location = mdbox:~/mdbox
mail_plugins = notify replication
mail_uid = nobody
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/userdb
driver = passwd-file
}
plugin {
mail_replica = tcp:bdove.mpi-klsb.mpg.de
}
protocols = " imap"
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
}
unix_listener replication-notify {
mode = 0666
}
}
service doveadm {
inet_listener doveadm-server {
port = 12345
}
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl = no
userdb {
args = /etc/dovecot/userdb
driver = passwd-file
}
smime.p7s
Description: S/MIME Cryptographic Signature
Re: Dovecot + sis problem
Hi Michal,
On 06.01.21 00:18, Michał Kiljański wrote:
Hi,
My setup is:
mail_attachment_dir = /var/mail/attachments
mail_attachment_hash = %{sha512}
mail_attachment_min_size = 64k
mail_attachment_fs = sis posix
Something works - but I expected that is one instance of file, and this
file is linked to messages. But here in folder I have got this:
sudo ls -la /var/mail/attachments/8b/67/
-rw-rw-rw- 3 buser dovecot 5425280 Jan 5 23:13
8b67b29a73b74ecf48e17e789051602cec13aa78cc01c06c8eee1ca4be7c8e07f700cdc9e9ca0381fa3541bdd4e26289de1d7eee763233dfaa0a200a264e5258-0258913a92f2f45f0d70035f3f08
-rw-rw-rw- 1 cuser dovecot 5425280 Jan 5 23:13
8b67b29a73b74ecf48e17e789051602cec13aa78cc01c06c8eee1ca4be7c8e07f700cdc9e9ca0381fa3541bdd4e26289de1d7eee763233dfaa0a200a264e5258-28571b018ff2f45f0c70035f3f08
-rw-rw-rw- 3 buser dovecot 5425280 Jan 5 23:13
8b67b29a73b74ecf48e17e789051602cec13aa78cc01c06c8eee1ca4be7c8e07f700cdc9e9ca0381fa3541bdd4e26289de1d7eee763233dfaa0a200a264e5258-50315d3a8ef2f45f0a70035f3f08
-rw-rw-rw- 1 auser dovecot 5425280 Jan 5 23:13
8b67b29a73b74ecf48e17e789051602cec13aa78cc01c06c8eee1ca4be7c8e07f700cdc9e9ca0381fa3541bdd4e26289de1d7eee763233dfaa0a200a264e5258-586cdb378ef2f45f0770035f3f08
drwxrwsrwx 2 buser dovecot 4096 Jan 5 23:13 hashes
sudo ls -la /var/mail/attachments/8b/67/hashes/
-rw-rw-rw- 3 buser dovecot 5425280 Jan 5 23:13
8b67b29a73b74ecf48e17e789051602cec13aa78cc01c06c8eee1ca4be7c8e07f700cdc9e9ca0381fa3541bdd4e26289de1d7eee763233dfaa0a200a264e5258
What is wrog? How to setup SIS to get one instance of file?
SIS uses hard links. In your example the attachment in 8b/67/hashes/ is
referenced in 2 other files (aka attached in 2 mails) in the 8b/67/
directory. So the data is stored only once on disk but referenced in 3
directory entries. You can see the refcount in column 3 of ls -l.
Best,
--
Patrick Cernko +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute für Informatik & Softwaresysteme
smime.p7s
Description: S/MIME Cryptographic Signature
execve(/usr/bin/sieve-test) failed: Argument list too long
Hi Dovecot developers, while debugging the above error message from sieve-test, I found out, that the content of directive ssl_ca is added as env var SSL_CA by doveconf on execve and sieve-test now uses doveconf. In our setup, ssl_ca is set to ssl_ca = on our director servers. We have backend servers with certificates signed by two different CAs and to avoid problems if a backend switches to a different CA, I decided to allow all "known" CAs. The corresponding env var SSL_CA has more than 230500 bytes, which causes execve to fail with error E2BIG. I found a workaround for the problem by setting ssl_ca = Where this file contains only the two CAs used atm. However I would like to request a fix for this issue as others might also want to have all "known" CAs set for dovecot director backend connections. Best, -- Patrick Cernko +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme smime.p7s Description: S/MIME Cryptographic Signature
Re: SIS and tracing the origin of an attachment
Hi all,
On 15.03.22 22:40, doug wrote:
On 3/15/2022 3:45 PM, Oscar del Rio wrote:
On 2022-03-15 9:02 a.m., doug wrote:
On 3/8/2022 5:51 PM, doug wrote:
I'm trying to trace an attachment within an SIS subdirectory to the
email message(s) that link to it. I say messages because I'm also
using dovecot dedup. My understanding is the linked file name is the
hash value of the attachments contents concatenated with the GUID of
the email message. I have had marginal success with a message I
created myself.
Example: I generated an email with two attachments. Here are the
links in my attachment directory.
./26/c5/26c5c540d41779d83d2f5388041d05c67d720d9a-73eca8051acd27627231f2bc99a3
./65/cd/65cd73112a489ef07f17ed5740aa60358e2dd3fb-74eca8051acd27627231f2bc99a3
I keep experimenting with this and I still haven't found a reliable
way to track an attachment back to it's original message so I can
either notify the user or delete the message with doveadm. Is this
not possible? I'm using mdbox if that matters. I see a similar thread
going right now about virus scanning and deleting messages but that
is maildir and I suspect not using SIS for attachments.
The very few times I've needed to trace a SIS attachment to a mailbox,
I just grep the "storage" folders for the file hash
find username/storage -type f -exec grep
9ffa4b246589f8039d123ea909f1520e791bd880 {} +
username/storage/m.46588:X908 2409141 B72
9f/fa/9ffa4b246589f8039d123ea909f1520e791bd880-c9ee303687e13062cf740012bfe47a40
username/storage/m.46589:X1918 2409141 B72
9f/fa/9ffa4b246589f8039d123ea909f1520e791bd880-080ce71390e1306299730012bfe47a40
username/storage/m.46588:
BSent
X908 2409141 B72
9f/fa/9ffa4b246589f8039d123ea909f1520e791bd880-c9ee303687e13062cf740012bfe47a40
username/storage/m.46589:
BINBOX
X1918 2409141 B72
9f/fa/9ffa4b246589f8039d123ea909f1520e791bd880-080ce71390e1306299730012bfe47a40
-> Attachment in username's INBOX and Sent folders.
Thank you for the suggestion Oscar. My mdbox files are encrypted and
compressed, so unfortunately directly grepping them will not work.
You can use "doveadm dump" to decompress the files for grepping them,
not sure about encryption:
find path/to/userhomes/mdbox/storage -name 'm.*' | \
while read f; do
doveadm dump $f | \
grep -E '^msg.(ext-ref|orig-mailbox|guid)' | \
grep -B2 xx/yy/hash-guid || continue
echo "Match in $f"
done
The dump also contains several other fields you might want to display.
Best,
--
Patrick Cernko +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute fuer Informatik & Softwaresysteme
smime.p7s
Description: S/MIME Cryptographic Signature
Re: Am I right to assume certificate renewal with the same filename requires a dovecot reload/restart
Hi, On 14.03.19 09:33, Yassine Chaouche via dovecot wrote: On 3/14/19 9:32 AM, Yassine Chaouche via dovecot wrote: The general answere here is try and see, as you could totally test it on your own. The certificate is read at startup and put in memory for the rest of the execution time. Dovecot won't monitor the file for changes on disk, as this would waste CPU cycles and make dovecot only slower for no reason. The process (or person) that changes the file is responsible to restart dovecot to reload the new certificate in memory. Yassine. I should mention that this is also true for Apache and postfix. on our debian systems, apache reloads the certificate file with service apache2 reload I never had to use "restart" to get the new certificate online. The advantage of reload is obvious: in case of a config error the daemon stays running (with the old config) whereas with restart you get a service downtime until you fixed the error. I guess dovecot's reload mechanism (doveadm reload) also rereads the certificate file, but I did not test that yet. However I just realized that doveadm reload exists with exitcode 0 even if there is an config error. You can only see the error message in the logs. At least the service keeps running (with the old config). I cannot say anything about postfix as we use exim. At least the way we have configured exim, it neither needs reload or restart but reads the certificate file every time it has to use it. Best, -- Patrick Cernko +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme smime.p7s Description: S/MIME Cryptographic Signature
Re: dovecot-keywords are not preserved any more when moving mails between folders
Hi Timo, hi list, On 12.03.19 22:31, Timo Sirainen via dovecot wrote: On 12 Mar 2019, at 17.55, Dan Christensen via dovecot wrote: On Mar 12, 2019, Aki Tuomi via dovecot wrote: On 12.3.2019 13.46, Piper Andreas via dovecot wrote: after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords, which in my case are set by thunderbird, are not preserved any more when moving a mail between folders. We are aware of this bug, and it's being tracked as DOP-842. Could this bug also be causing flags to be lost when using dsync (as I described in some messages to this list Feb 16 to 23)? It seems like it might be a different bug, since in my experience the flags are sometimes synced and then removed later. That bug is fixed with attached patch. This patch seems to fix my "replication dropped imap flags" issue reported on this list on 2018-11-26! Thank you! Best regards, -- Patrick Cernko +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme smime.p7s Description: S/MIME Cryptographic Signature
Re: Am I right to assume certificate renewal with the same filename requires a dovecot reload/restart
Hi Yassine, hi Kostya, On 14.03.19 10:17, Kostya Vasilyev via dovecot wrote: On Thu, Mar 14, 2019, at 12:09 PM, Yassine Chaouche via dovecot wrote: On 3/14/19 9:55 AM, Patrick Cernko via dovecot wrote: [...] the way we have configured exim, it neither needs reload or restart but reads the certificate file every time it has to use it. What happens if you goof off in the middle of an opeartion, temporarily putting a wrong file instead of the new certificate, and exim starts delivering the new broken certificate right away ? or breaks ? or clients can't connect anymore with TLS ? or don't connect at all if you don't allow non-TLS connexions ? First: It happens the same if I replace the file with a wrong cert AND reload another service deamon and then get interupted. Second: I use ansible to push configurations and usually first push changes to a test system or only one machine. Third: Server administration always has the risk of human error ;-) Getting caught in the middle of a cert file or key file update should not happen -- a process that already opened a file will continue to be reading from that file, even if it gets renamed. But what if exim (or some other process) happens to read the "old" certificate file - and then the "new" private key file (or vice versa)? A race condition like this seems unlikely but technically possible. We store cert and key together in one PEM file, thus we will always exchange both cert and key in one "atomic" operation. Best, -- Patrick Cernko +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme smime.p7s Description: S/MIME Cryptographic Signature
Re: replication dropped imap flags
Update: The patch "dsync: Fix importing keywords with MAIL_TRANSACTION_SYNC flag set" mentioned in the mail from Timo Sirainen on 2019-03-12 22:31 on this list seems to fix this issue. Regards, -- Patrick Cernko +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme smime.p7s Description: S/MIME Cryptographic Signature
lmtp report permission denied on delivery to multiple recipients
Hello,
when receiving mails for multiple recipients via LMTP, I often see the
following messages on our servers:
May 22 11:06:10 sinon dovecot[44304]: lmtp(119718): Connect from $IP$
May 22 11:06:11 sinon dovecot[44304]: lmtp($USER1$)<119718><2HnmOgIR5Vym0wEA22L5Rg>:
msgid=<$MSGID$>: saved mail to INBOX
May 22 11:06:11 sinon dovecot[44304]:
lmtp($USER2$)<119718><2HnmOgIR5Vym0wEA22L5Rg:2>: Error:
stat(/IMAP/mail/mailboxes/$USER1$/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache)
failed: Permission denied (euid=$UID2$($USER2$) egid=$GID2$(rbg) missing +x perm:
/IMAP/mail/mailboxes/$USER1$, dir owned by $UID1$:$GID2$ mode=0700)
May 22 11:06:11 sinon dovecot[44304]:
lmtp($USER2$)<119718><2HnmOgIR5Vym0wEA22L5Rg:2>: Error:
open(/IMAP/mail/mailboxes/$USER1$/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache)
failed: Permission denied (euid=$UID2$($USER2$) egid=$GID2$(rbg) missing +x perm:
/IMAP/mail/mailboxes/$USER1$, dir owned by $UID1$:$GID2$ mode=0700)
May 22 11:06:11 sinon dovecot[44304]:
lmtp($USER2$)<119718><2HnmOgIR5Vym0wEA22L5Rg:2>: Error:
lstat(/IMAP/mail/mailboxes/$USER1$/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache.lock)
failed: Permission denied
May 22 11:06:11 sinon dovecot[44304]:
lmtp($USER2$)<119718><2HnmOgIR5Vym0wEA22L5Rg:2>: Error:
file_dotlock_open(/IMAP/mail/mailboxes/$USER1$/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache)
failed: Permission denied (euid=$UID2$($USER2$) egid=$GID2$(rbg) missing +x perm:
/IMAP/mail/mailboxes/$USER1$, dir owned by $UID1$:$GID2$ mode=0700)
May 22 11:06:11 sinon dovecot[44304]:
lmtp($USER2$)<119718><2HnmOgIR5Vym0wEA22L5Rg:2>: Error:
open(/IMAP/mail/mailboxes/$USER1$/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache)
failed: Permission denied (euid=$UID2$($USER2$) egid=$GID2$(rbg) missing +x perm:
/IMAP/mail/mailboxes/$USER1$, dir owned by $UID1$:$GID2$ mode=0700)
May 22 11:06:11 sinon dovecot[44304]:
lmtp($USER2$)<119718><2HnmOgIR5Vym0wEA22L5Rg:2>: 2HnmOgIR5Vym0wEA22L5Rg:2: sieve:
msgid=<$MSGID$>: stored mail into mailbox 'INBOX'
May 22 11:06:11 sinon dovecot[44304]: lmtp(119718): Disconnect from $IP$:
Successful quit
(usernames, uids, gids and IP addresses anonymized)
So far, I was not able to reproduce this issue on a simple test setup,
but I have the strong impression, that is has something to do with the
fact, if some of the affected users have a active sieve script and
others do not use sieve. In the example above $USER1$ did not have a
sieve script configured but $USER2$ had one, as you can see. As a
workaround, I configured an empty sieve script for all users with out
sieve configured. This seems to work but actually I do not want to have
users with empty sieve scripts.
This bug seems to be related to
https://dovecot.org/list/dovecot/2013-February/088540.html where Timo
stated:
"LMTP always delivers the mail to the first user. Then it tries to copy
the first mail to the second user, because in some setups this can be
done using hard links. With mbox that of course doesn't work, but looks
like instead of failing silently it logs an error. So everything is
working as it should, except there are these unnecessary errors logged.
I'll see about getting rid of them."
Is it possible to get rid of these error messages, either by a config
setting that prevents if they do not lead to a real problem or by
changing the code to avoid the behaviour described by Timo?
Attached you can find doveconf -n output and the used ldap.conf (both
anonymized), the referenced /etc/dovecot/passdb.deny and
/etc/dovecot/userdb.overrides are usually empty files and only used
during mailbox migrations.
Best regards,
--
Patrick Cernko +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute fuer Informatik & Softwaresysteme
# 2.2.36.3 (a7d78f5a2): /etc/dovecot/dovecot.conf
# Object storage plugin version 2.2.36.3 (64b24e4d)
# Pigeonhole version 0.4.24 (5a7e9e62)
# OS: Linux 4.14.99.1.amd64-smp x86_64 Debian 9.8
# Hostname: XXX
auth_verbose = yes
default_vsz_limit = 2 G
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
license_checksum = method=%m rip=%r lip=%l mpid=%e %c
mail_attachment_dir = /IMAP/mail/attachments
mail_attachment_fs = sis-queue /IMAP/mail/attachments/queue:posix
mail_home = /IMAP/mail/mailboxes/%u
mail_location = mdbox:~/mdbox
mail_log_prefix = "%s(%u)<%{pid}><%{session}>: "
mail_max_userip_connections = 0
mail_plugins = " notify replication zlib fts fts_dovecot"
maildir_stat_dirs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date index ihave duplicate
mime foreverypart extra
WARNING: using attachment_dir with plugin zlib can corrupt mails
Hello list, hello Dovecot developers,
this week, I discovered a serious bug in Dovecot, that lead to several
broken mails on our servers. The bug corrupts the first few characters
of the mail header during saving. On our setup, it was almost always
only the very first line of text, that was corrupted.
Depending on the IMAP client (they seem to request different header
fields, ... during mail access), the bug causes the imap process to hang
up the TCP connection and log errors like this:
imap(USERNAME)<4767>: Error: Corrupted record in index cache
file /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache:
UID 489113: Broken fields in mailbox Trash:
read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))): FETCH
BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs 122
In our case that finally grabbed my attention, the client was the users
iphone that did not display any new messages but his Thunderbird did.
The bug seems to be triggered by a bad "interaction" of attachment_dir
option and zlib plugin. If you use both, you most likely are affected,
too, except you only use zlib plugin for reading previously compressed
stored mails. That's also the workaround we use now: zlib plugin only
enabled in mail_plugins but no plugin/zlib_save set.
The bug occurs on very specific mails. Due to privacy reasons I could
not provide sample mails here. Storing such mails seems to trigger the
bug reproducible.
I attached a very minimal doveconf -n config, that can be used to
trigger the bug. If one of the developers is interested, I can try to
generate an "anonymized" version of such a specific mail that still
causes the issue. I discovered the bug on our productive systems,
running latest Dovecot 2.2 release, but the latest 2.3 I used during
debugging is affected, too.
During debugging, I also found one hint, that might help find the bug:
If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and
then disable the zlib plugin in mail_plugins, you can call
doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress
on test's mailbox with only that one broken mail.
This will display the beginning of the rfc822 mail text until gzip
terminates with "gzip: stdin: unexpected end of file", approximately
after twice the length of the mail HEADER. This might indicate, that
dovecot stores the uncompressed size of the header in it's data
structures although the mail is stored compressed.
I also found a very efficient way to find all affected mails in our setup:
doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \
grep -a "^[^ ]+ user=" | \
grep -avF ' hdr=Return-path: ' | \
grep -av '.* hdr=[[:print:][:space:]]*$'
(runtime for ~6M mails on our servers was 20-30min)
This can be even more optimized if you have a powerful storage system
with GNU parallel:
doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid mailbox uid
seq flags hdr' all | grep -a '^user=' | grep -avF ' hdr=Return-path: ' | grep -av '.*
hdr=[[:print:][:space:]]*$' || true"
(runtime for ~6M mails on our servers was ~4min)
The command will give you a list of mails that possibly are affected,
check the full output of
doveadm fetch -u USERNAME hdr guid GUID | less
to verify that the header is really broken.
On our systems I found 39 mails within ~12M mails.
I was able to recover these mails "manually" by reconstructing the
Return-Path header line, importing the fixed mails and expunging the
corrupt ones. Before importing, I had to disable zlib_save option obviously.
Best regards,
--
Patrick Cernko +49 681 9325 5815
Joint Administration: Information Services and Technology
Max-Planck-Institute fuer Informatik & Softwaresysteme
# 2.3.6.1 (d124cc84b): /etc/dovecot/dovecot.conf
# OS: Linux 4.14.127.1.amd64-smp x86_64 Debian 9.9
# Hostname: adove.mpi-klsb.mpg.de
listen = *
mail_attachment_dir = /var/vmail/attachments
mail_attachment_fs = posix
mail_gid = nogroup
mail_home = /var/vmail/%u
mail_location = mdbox:~/mdbox
mail_plugins = " zlib"
mail_uid = nobody
passdb {
args = /etc/dovecot/userdb
driver = passwd-file
}
plugin {
zlib_save = gz
}
protocols = imap
userdb {
args = /etc/dovecot/userdb
driver = passwd-file
}
smime.p7s
Description: S/MIME Cryptographic Signature
