"doveadm altmove -r" not working ?
Hello, I have some trouble using "doveadm altmove -r". Regular "doveadm altmove" is OK, selected mails were moved to alternate storage as expected. But I can't manage to get them back to original location, "doveadm altmove -r" has no effect. We are using Dovecot 2.3.4 (Debian Buster package). mail_location = mdbox:~/mdbox:DIRNAME=_@@_dbox-Mails_@@_:ALT=/slow%h/mdbox doveadm altmove -u myuser since 90d -> some message files are created under /slow/my/homedir/mdbox/storage doveadm altmove -r -u myuser all -> message files are still sitting under /slow/my/homedir/mdbox/storage, none were created into /my/homedir/mdbox/storage. Is there some known issues with doveadm altmove in this version ? Or am I missing something ? I can share more config details if needed. -- Benoit BRANCIARD Université Paris 1 Panthéon-Sorbonne - DSIUN-SIS B405 - Centre PMF - 90 rue de Tolbiac - 75013 Paris Tel. 01 44 07 89 68 http://dsiun.univ-paris1.fr
UPD: 2.2.27 : accessing "mdbox_deleted" content destroys indexes
Just to say this bug is still present in 2.2.27 (1:2.2.27-2~bpo8+1 jessie-backports Debian package). # dovecot --version 2.2.27 (c0f36b0) mail_location = mdbox:~/mdbox:DIRNAME=_@@_dbox-Mails_@@_ Please let me know if you want me to transmit our "doveconf -n" output. Regards, Le 25/01/2017 à 15:13, Benoit Branciard a écrit : Accessing or listing "mdbox_deleted" contents seems to destroy MDBOX indexes. Examples of commands which triggers this problem ($home being the home directory of $user, and mail_location being mdbox:~/mdbox): doveadm -o mail="mdbox_deleted:$home/mdbox" -f table mailbox status -u "$user" 'messages vsize' INBOX doveadm -v import -s -u "$user" "mdbox_deleted:$home/mdbox" restored-mail ALL The above "doveadm mailbox status" command outputs an error: doveadm(user): Error: Log synchronization error at seq=1,offset=104908 for (in-memory index): Append with UID 1, but next_uid = 5227 doveadm(user): Warning: fscking index file (in-memory index) Subsequent "doveadm mailbox status -u $user 'messages vsize'" on the active mailbox report empty folders (null messages and vsize), whereas folders actually aren't empty. Workaround: this problem is corrected by a "doveadm search -u $user all", which obviously forces indexes to be rebuilt. Vesion: 2.2.26.0 (23d1de6) (Debian jessie-backports package) We did *not* have this problem in 2.2.24 version (previous Debian jessie-backports package). We use following mail_location with explicit DIRNAME (don't know if that matters): mail_location = mdbox:~/mdbox:DIRNAME=_@@_dbox-Mails_@@_ I tested with and without appending ":DIRNAME=_@@_dbox-Mails_@@_" to mail="mdbox_deleted:$home/mdbox" with same results. -- Benoit BRANCIARD Service InfraStructures (SIS) Direction du Système d'Information et des Usages Numériques (DSIUN) Université Paris 1 Panthéon-Sorbonne Centre Pierre Mendès France 90 rue de Tolbiac - 75634 Paris cedex 13 - France Bur. B406 - Tél +33 1 44 07 89 68 - Fax +33 1 44 07 89 66 Accueil: +33 1 44 07 89 65 - assistance-ds...@univ-paris1.fr http://dsi.univ-paris1.fr
2.2.26.0 : accessing "mdbox_deleted" content destroys indexes
Accessing or listing "mdbox_deleted" contents seems to destroy MDBOX indexes. Examples of commands which triggers this problem ($home being the home directory of $user, and mail_location being mdbox:~/mdbox): doveadm -o mail="mdbox_deleted:$home/mdbox" -f table mailbox status -u "$user" 'messages vsize' INBOX doveadm -v import -s -u "$user" "mdbox_deleted:$home/mdbox" restored-mail ALL The above "doveadm mailbox status" command outputs an error: doveadm(user): Error: Log synchronization error at seq=1,offset=104908 for (in-memory index): Append with UID 1, but next_uid = 5227 doveadm(user): Warning: fscking index file (in-memory index) Subsequent "doveadm mailbox status -u $user 'messages vsize'" on the active mailbox report empty folders (null messages and vsize), whereas folders actually aren't empty. Workaround: this problem is corrected by a "doveadm search -u $user all", which obviously forces indexes to be rebuilt. Vesion: 2.2.26.0 (23d1de6) (Debian jessie-backports package) We did *not* have this problem in 2.2.24 version (previous Debian jessie-backports package). We use following mail_location with explicit DIRNAME (don't know if that matters): mail_location = mdbox:~/mdbox:DIRNAME=_@@_dbox-Mails_@@_ I tested with and without appending ":DIRNAME=_@@_dbox-Mails_@@_" to mail="mdbox_deleted:$home/mdbox" with same results. -- Benoit BRANCIARD Service InfraStructures (SIS) Direction du Système d'Information et des Usages Numériques (DSIUN) Université Paris 1 Panthéon-Sorbonne Centre Pierre Mendès France 90 rue de Tolbiac - 75634 Paris cedex 13 - France Bur. B406 - Tél +33 1 44 07 89 68 - Fax +33 1 44 07 89 66 Accueil: +33 1 44 07 89 65 - assistance-ds...@univ-paris1.fr http://dsi.univ-paris1.fr
Re: doveadm output format changes
"doveadm fetch text mailbox MyBox uid UID1,UID2" also changed... "^L"s (form-feeds) disappeared... Hmmm :-( Le 05/01/2017 à 12:38, Benoit Branciard a écrit : It appears that doveadm output format changes every now and then, without particular notice. For example, the following command: doveadm -f pager mailbox status 'messages recent' '*' -- Benoit BRANCIARD Service InfraStructures (SIS) Direction du Système d'Information et des Usages Numériques (DSIUN) Université Paris 1 Panthéon-Sorbonne Centre Pierre Mendès France 90 rue de Tolbiac - 75634 Paris cedex 13 - France Bur. B406 - Tél +33 1 44 07 89 68 - Fax +33 1 44 07 89 66 Accueil: +33 1 44 07 89 65 - assistance-ds...@univ-paris1.fr http://dsi.univ-paris1.fr
doveadm output format changes
It appears that doveadm output format changes every now and then, without particular notice. For example, the following command: doveadm -f pager mailbox status 'messages recent' '*' did output something like this until v2.2.24 : mailbox: Mailbox1 messages: 58 recent: 12 ^L mailbox: Mailbox2 messages: 128 recent: 0 but switched to that in v2.2.26 : Mailbox1 messages: 58 recent: 12 ^L Mailbox2 messages: 128 recent: 0 This seems related to the following changelog entry: 2016-10-25 20:51:36 +0300 Timo Sirainen <timo.sirai...@dovecot.fi> (5baa99e) doveadm: "pager" formatter supports now DOVEADM_PRINT_HEADER_FLAG_HIDE_TITLE M src/doveadm/doveadm-print-pager.c Some other format changes did also occur in the past. For example, "doveadm user" had a specific format in v2.0.19 (and ignored the "-f format" option), and obviously defaulted to the "-f tab" format (which has different rendering) somewhere in the v2.2.x or 2.1.x. Such changes render maintenance of wrapping scripts particularly tedious. Could it be possible to avoid such breaking changes in the future ? -- Benoit BRANCIARD Service InfraStructures (SIS) Direction du Système d'Information et des Usages Numériques (DSIUN) Université Paris 1 Panthéon-Sorbonne Centre Pierre Mendès France 90 rue de Tolbiac - 75634 Paris cedex 13 - France Bur. B406 - Tél +33 1 44 07 89 68 - Fax +33 1 44 07 89 66 Accueil: +33 1 44 07 89 65 - assistance-ds...@univ-paris1.fr http://dsi.univ-paris1.fr
dsync: loss of keywords in 2.2.13
After some tests, we found that "dsync backup" sometimes fails to copy all the IMAP "keywords" (labels) from the source mailbox to the backup one. Our Dovecot version is 2.2.13 (Debian Jessie package 1:2.2.13-12~deb8u1). We are migrating mailboxes from MBOX (with separate indexes) to MDBOX (with custom DIRNAME) format. The source mail_location is: mail_location = mbox:~/mail:INBOX=~/mail/INBOX:INDEX=/var/cache/dovecot/indexes/%16Hu/%u and the target: mail_location = mdbox:~/mdbox:DIRNAME=_@@_dbox-Mails_@@_ and we use the following dsync command to replicate (twice, a first "hot" run, and a second "cold" run with lda and user kicked off). dsync -o mail_access_groups=$mgroup -u "$login" backup "mdbox:$mdbhome/mdbox:DIRNAME=_@@_dbox-Mails_@@_" where $login is username, $mgroup is an UNIX group which has write access to the (temporary chowned/chmoded) mailbox, and $mdbhome is the new (MDBOX) user's home. It appears that some keywords get replicated, but some other do not. The only workaroud we found is to backup all keywords with "doveadm fetch 'mailbox uid flags' keywords '*'", an restore them on the new mailbox with "doveadm flags add". This solution yields no keyword loss. has this problem already been addressed ? -- Benoit BRANCIARD Service InfraStructures (SIS) Direction du Système d'Information et des Usages Numériques (DSIUN) Université Paris 1 Panthéon-Sorbonne Centre Pierre Mendès France 90 rue de Tolbiac - 75634 Paris cedex 13 - France Bur. B406 - Tél +33 1 44 07 89 68 - Fax +33 1 44 07 89 66 Accueil: +33 1 44 07 89 65 - assistance-ds...@univ-paris1.fr http://dsi.univ-paris1.fr
[Dovecot] upgrade 1.0.15 - 2.1.7: MBOX index compatibility and performance
Hi, we just upgraded our mailserver from Dovecot 1.0.15 to Dovecot 2.1.7. We use MBOX format (due to legacy compatibility), system users, PAM+GSSAPI auth, filesystem quotas, and indexes located on a separate filesystem: mail_location = mbox:~/mail:INBOX=~/mail/INBOX:INDEX=/var/cache/dovecot/indexes/%16Hu/%u The 2.1.7 configuration files have been rewritten based on default templates instead of converting it from 1.0.15. The server has ~8000 mailboxes and about ~2000 simultaneous IMAP/POP active connexions. The problem is: - indexes seem to be rebuilt: first IMAP/POP connexion for each user thows lots of error messages in the log, and the global index size decreases. Example error log: Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Cached message size larger than expected (27884 27855) Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Corrupted index cache file /var/cache/dovecot/indexes/4/myuser/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 4414 Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Cached message size larger than expected (27884 27855) Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Corrupted index cache file /var/cache/dovecot/indexes/4/myuser/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 4414 Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Cached message size larger than expected (27884 27855) Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Corrupted index cache file /var/cache/dovecot/indexes/4/myuser/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 4414 Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Cached message size larger than expected (27884 27855) Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Corrupted index cache file /var/cache/dovecot/indexes/4/myuser/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 4414 Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: copy: i_stream_read() failed: Input/output error Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Cached message size larger than expected (27884 27855) Nov 19 08:56:38 myhost dovecot: imap(myuser): Error: Corrupted index cache file /var/cache/dovecot/indexes/4/myuser/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 4414 - load average is extremely high (more than 10x the usual one), resulting from an significant increase of disk I/O, and for now (4h after the monday rush) this doesn't seem to decrease. Questions: - are 1.05 indexes supposed to be backward compatible with Dovecot 2.1.7 ? - are there some technical reasons which could explain the increase of disk I/O, apart from index rebuild ? -- Benoit BRANCIARD Service InfraStructures (SIS) - Direction du Système d'Information (DSI) Université Paris 1 Panthéon-Sorbonne Centre Pierre Mendès France B 406 - 90, rue de Tolbiac - 75634 Paris cedex 13 - France Tél : +33 1 44 07 89 68 - Fax : +33 1 44 07 89 66 Accueil tél. : +33 1 44 07 89 65 Assistance : assistance-...@univ-paris1.fr Web : http://dsi.univ-paris1.fr -- Ce message a ete verifie par MailScanner pour des virus ou des polluriels et rien de suspect n'a ete trouve.
[Dovecot] cumulative userdb ?
in Dovecot 2.0, is it possible to have kind of cumulative multiple userdb ? that is, for all users: - extract some attributes (let's say: uid, gid, home) from a first userdb (Passwd for example), - an extract some other attributes (mail for example, but overwriting those from the first userdb in case of redundancy) from a second userdb (LDAP for example) ? This is *different* from the multiple databases setup described in http://wiki2.dovecot.org/Authentication/MultipleDatabases, where it is meant as failover: the second database is looked up only if the user isn't found in the first database. -- Benoit BRANCIARD Service InfraStructures (SIS) - Direction du Système d'Information (DSI) Université Paris 1 Panthéon-Sorbonne Centre Pierre Mendès France B 406 - 90, rue de Tolbiac - 75634 Paris cedex 13 - France Tél : +33 1 44 07 89 68 - Fax : +33 1 44 07 89 66 Accueil tél. : +33 1 44 07 89 65 Assistance : assistance-...@univ-paris1.fr Web : http://dsi.univ-paris1.fr -- Ce message a ete verifie par MailScanner pour des virus ou des polluriels et rien de suspect n'a ete trouve.
Re: [Dovecot] outlook2007 shows frequent imap disconnect no matterwhat outlook-idle setting in dovecot.conf
Timo Sirainen a écrit : On Mar 31, 2008, at 8:34 AM, Kielbasiewicz, Peter wrote: I updated to the latest rev from atrpms.net which is 1.0.13 and I see the same behaviour. No matter if I set imap_client_workarounds = outlook-idle or imap_client_workarounds = I get those disconnect popups. It seems that the setting has no effect. Is there someone who had success by applying the setting with outlook 2007? .. dovecot: 2008-03-26 15:16:01 Info: IMAP(user): Connection closed: Connection reset by peer Connection reset by peer means the connection was closed by the client (or something in the network), not Dovecot. Such disconnects may be due to a router or a firewall (or something like an ADSL modem/router) which sets a timeout on idle TCP connexions (reflexive ACL, NAT mapping...). It is frequent to have no more than 5 minutes timeout. Workaround is to force the client or the server to keep the connexion alive by sending some packets at an interval less than 5 minutes, for example by polling new mail every 3 minutes. -- Ce message a ete verifie par MailScanner pour des virus ou des polluriels et rien de suspect n'a ete trouve.
Re: [Dovecot] Security issue #5: mail_extra_groups setting is often used insecurely
Timo Sirainen a écrit : 2a) mbox: Any files/directories under mail group-writable directories can be created/deleted/renamed by symlinking the directory under ~/mail/. For example ln -s /var/mail ~/mail/var, DELETE var/root will happily delete root's mailbox. This I hadn't thought about before. Not if /var/mail is set sticky, which is the case on all good modern Unix systems: Right. That's why it was included in the workarounds. :) Anyway I also thought that /var/mail would be sticky in at least some systems. I couldn't find a single one. CentOS 5, Debian, FreeBSD 6.2, Solaris 10 none have it sticky by default. All our Debian Sarge and Etch systems (with Sendmail and procmail packages) have /var/mail sticky by default, we didn't modify it ourselves. -- Ce message a ete verifie par MailScanner pour des virus ou des polluriels et rien de suspect n'a ete trouve.
Re: [Dovecot] Just verifying upgrade procedure?
Bjørn T Johansen a écrit : When upgrading from source, it's ok to just run make install and every files will be overwritten with the new version? Nothing else one should do? Pay attention to the fact that the dovecot daemon dynamically starts the imap and pop server executables whenever inboud connexions are requested. If you upgrade without stopping the master dovecot, and clients request connexions, dovecot will try to run the new imap and pop executables, which are not the same version as the master, and problems may arise. The good solution is to stop the master dovecot *before* make install, and restart it afterwards; or to make sure no clients try to connect before you completed the upgrade process. -- Ce message a ete verifie par MailScanner pour des virus ou des polluriels et rien de suspect n'a ete trouve.
Re: [Dovecot] got too little data ?? [SOLVED]
Benoit Branciard a écrit : I just switched in a hurry one of our mailbox servers from UW-imapd to Dovecot in an attempt to address performance and stability problems. [...] The bad new is that we get lots or errors like this in log: IMAP(username): FETCH for mailbox INBOX UID 23862 got too little data: 3186 vs 3206 We are running 1.0.9 (which includes Timo's patch) for about one day without the bug showing up anymore. So the fix seems OK. I believe Dovecot is the only software for which bugs are fixed in less time I need to complete the bug report :-) -- Ce message a ete verifie par MailScanner pour des virus ou des polluriels et rien de suspect n'a ete trouve.
[Dovecot] got too little data ??
I just switched in a hurry one of our mailbox servers from UW-imapd to Dovecot in an attempt to address performance and stability problems. This server hosts ~5500 accounts, ~2000 are daily used, ~600 simultaneous IMAP connexions in rush hours, and lots of POP ones. Some accounts may be accessed simultaneously with IMAP and POP. The system is : - Debian Sarge (x86 with amd64 kernel) - MBOX files mailboxes - indexes migrated out of filesystem quotas - Sendmail MTA - Procmail local delivery agent (not easily replaceable due to extensive use of procmailrc filters) - Both Procmail and Dovecot use [dotlock, fcntl] as mailbox locking scheme - No other processes have access to mailboxes in normal use - Dovecot 1.0.8 hand-compiled from tarball The good new is that it seems to perform better than UW-imapd. The bad new is that we get lots or errors like this in log: IMAP(username): FETCH for mailbox INBOX UID 23862 got too little data: 3186 vs 3206 Such accidents happen about 4-5 times a day, and does not correct themselves, even with a mailbox expunge or reopen. However manual deletion of the faulty mailbox index seems to clear the problem, until another one occurs. On such errors some mail clients (Thunderbird) retry operation in infinite loop, so log gets really cluttered. In imap-fetch-body.c it is stated : We shouldn't really ever get here. One reason is if mail was deleted from NFS server while we were reading it. Another is some temporary disk error. Both reasons do not seem to apply in our case: mail partition is not NFS-mounted, and disks (a RAID5 array) seem healthy. Furthermore, affected accounts are not over quota. So may we have found a bug ? Or may we have a locking problem ? Or an index handling one ? I tried to disable mbox_lazy_writes without success. I will try to disable disk indexes totally, which should help since problems seem index-related; but performance may suffer. Any clues appreciated... Dovecot -n says : # 1.0.8: /usr/local/etc/dovecot.conf protocols: imap imaps pop3 pop3s listen: [::] ssl_cert_file: /usr/local/etc/ssl/private/our-certificate.pem ssl_key_file: /usr/local/etc/ssl/private/our-certificate.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login first_valid_uid: 200 first_valid_gid: 200 mail_location: mbox:~/mail:INBOX=~/mail/INBOX:INDEX=/espace/dovecot/indexes/%u fsync_disable: yes mbox_lazy_writes: no mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle tb-extra-mailbox-sep delay-newmail imap_client_workarounds(imap): outlook-idle tb-extra-mailbox-sep delay-newmail imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xv%08Xu pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: passdb: driver: pam args: * userdb: driver: passwd plugin: quota: fs -- Ce message a ete verifie par MailScanner pour des virus ou des polluriels et rien de suspect n'a ete trouve.
