how is the "Total number of known users" reported by doveadm replicator calculted?
Hello, I have a question regarding the "Total number of known users" displayed by doveadm replicator status. How is it calculated? Shouldn't it match the number of users reported by doveadm user '*' ? I have 3 servers being replicated, the "total number of known users" a lot higher than the number of users reports by doveadm user '*' why is that? Arnaud -- Arnaud Abélard Responsable Service Infrastructures DSIN Université de Nantes - smime.p7s Description: S/MIME Cryptographic Signature
Re: no full syncs after upgrading to dovecot 2.3.18
Ah, I'm now getting errors in the logs, that would explains the
increasing number of failed sync requests:
dovecot: imap(x)<2961235>: Error:
Mailbox INBOX: /vmail/l/i/x/dovecot.index reset, view is now
inconsistent
And sure enough:
# dovecot replicator status x
x none 00:02:54 07:11:28 -y
What could explain that error?
Arnaud
On 25/04/2022 15:13, Arnaud Abélard wrote:
Hello,
On my side we are running Linux (Debian Buster).
I'm not sure my problem is actually the same as Paul or you Sebastian
since I have a lot of boxes but those are actually small (quota of
110MB) so I doubt any of them have more than a dozen imap folders.
The main symptom is that I have tons of full sync requests awaiting but
even though no other sync is pending the replicator just waits for
something to trigger those syncs.
Today, with users back I can see that normal and incremental syncs are
being done on the 15 connections, with an occasional full sync here or
there and lots of "Waiting 'failed' requests":
Queued 'sync' requests 0
Queued 'high' requests 0
Queued 'low' requests 0
Queued 'failed' requests 122
Queued 'full resync' requests 28785
Waiting 'failed' requests 4294
Total number of known users 42512
So, why didn't the replicator take advantage of the weekend to replicate
the mailboxes while no user were using them?
Arnaud
On 25/04/2022 13:54, Sebastian Marske wrote:
Hi there,
thanks for your insights and for diving deeper into this Paul!
For me, the users ending up in 'Waiting for dsync to finish' all have
more than 256 Imap folders as well (ranging from 288 up to >5500; as per
'doveadm mailbox list -u | wc -l'). For more details on my
setup please see my post from February [1].
@Arnaud: What OS are you running on?
Best
Sebastian
[1] https://dovecot.org/pipermail/dovecot/2022-February/124168.html
On 4/24/22 19:36, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:
Question having similiar replication issues
pls read everything below and advise the folder counts on the
non-replicated users?
i find the total number of folders / account seems to be a factor and
NOT the size of the mail box
ie i have customers with 40G of emails no problem over 40 or so folders
and it works ok
300+ folders seems to be the issue
i have been going through the replication code
no errors being logged
i am assuming that the replication --> dhclient --> other server is
timing out or not reading the folder lists correctly (ie dies after X
folders read)
thus i am going through the code patching for log entries etc to find
the issues.
see
[13:33:57] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
# ll
total 86
drwxr-xr-x 2 root wheel uarch 4B Apr 24 11:11 .
drwxr-xr-x 4 root wheel uarch 4B Mar 8 2021 ..
-rw-r--r-- 1 root wheel uarch 73B Apr 24 11:11 instances
-rw-r--r-- 1 root wheel uarch 160K Apr 24 13:33 replicator.db
[13:33:58] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
#
replicator.db seems to get updated ok but never processed properly.
# sync.users
n...@elirpa.com high 00:09:41 463:47:01 - y
ke...@elirpa.com high 00:09:23 463:45:43 - y
p...@scom.ca high 00:09:41 463:46:51 - y
e...@scom.ca high 00:09:43 463:47:01 - y
ed.ha...@dssmgmt.com high 00:09:42 463:46:58 - y
p...@paulkudla.net high 00:09:44 463:47:03 580:35:07
y
so
two things :
first to get the production stuff to work i had to write a script that
whould find the bad sync's and the force a dsync between the servers
i run this every five minutes or each server.
in crontab
*/10 * * * * root /usr/bin/nohup
/programs/common/sync.recover > /dev/null
python script to sort things out
# cat /programs/common/sync.recover
#!/usr/local/bin/python3
#Force sync between servers that are reporting bad?
import os,sys,django,socket
from optparse import OptionParser
from lib import *
#Sample Re-Index MB
#doveadm -D force-resync -u p...@scom.ca -f INBOX*
USAGE_TEXT = '''\
usage: %%prog %s[options]
'''
parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')
parser.add_option("-m", "--send_to", dest="send_to", help="Send Email
To")
parser.add_option("-e", "--email", dest="email_box", help="Box to
Index")
parser.add_option("-d", "--detail",action='store_true',
dest="detail",default =False, help="Detailed report")
parser.add_option("-i", "--index",action='store_true',
dest="index",default =False, help="Index")
options, args = parser.parse_args()
print (options.email_box)
print (options.send_to)
print (options.detail)
Re: no full syncs after upgrading to dovecot 2.3.18
ely the issue - not the mbox pyhsical size as thought origionally. if someone can point me in the right direction, it seems either the replicator is not picking up on the number of folders to replicat properly or it has a hard set limit like 256 / 512 / 65535 etc and stops the replication request thereafter. I am mainly a machine code programmer from the 80's and have concentrated on python as of late, 'c' i am starting to go through just to give you a background on my talents. It took 2 months to finger this out. this issue also seems to be indirectly causing the duplicate messages supression not to work as well. python programming to reproduce issue (loops are for last run started @ 200 - fyi) : # cat mbox.gen #!/usr/local/bin/python2 import os,sys from lib import * user = 'p...@paulkudla.net' """ for count in range (0,600) : box = 'INBOX/folder-%s' %count print count command = '/usr/local/bin/doveadm mailbox create -s -u %s %s' %(user,box) print command a = commands.getoutput(command) print a """ for count in range (0,600) : box = 'INBOX/folder-0/sub-%' %count print count command = '/usr/local/bin/doveadm mailbox create -s -u %s %s' %(user,box) print command a = commands.getoutput(command) print a #sys.exit() Happy Sunday !!! Thanks - paul Paul Kudla Scom.ca Internet Services <http://www.scom.ca> 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3 Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 On 4/24/2022 10:22 AM, Arnaud Abélard wrote: Hello, I am working on replicating a server (and adding compression on the other side) and since I had "Error: dsync I/O has stalled, no activity for 600 seconds (version not received)" errors I upgraded both source and destination server with the latest 2.3 version (2.3.18). While before the upgrade all the 15 replication connections were busy after upgrading dovecot replicator dsync-status shows that most of the time nothing is being replicated at all. I can see some brief replications that last, but 99,9% of the time nothing is happening at all. I have a replication_full_sync_interval of 12 hours but I have thousands of users with their last full sync over 90 hours ago. "doveadm replicator status" also shows that i have over 35,000 queued full resync requests, but no sync, high or low queued requests so why aren't the full requests occuring? There are no errors in the logs. Thanks, Arnaud -- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes - smime.p7s Description: S/MIME Cryptographic Signature
no full syncs after upgrading to dovecot 2.3.18
Hello, I am working on replicating a server (and adding compression on the other side) and since I had "Error: dsync I/O has stalled, no activity for 600 seconds (version not received)" errors I upgraded both source and destination server with the latest 2.3 version (2.3.18). While before the upgrade all the 15 replication connections were busy after upgrading dovecot replicator dsync-status shows that most of the time nothing is being replicated at all. I can see some brief replications that last, but 99,9% of the time nothing is happening at all. I have a replication_full_sync_interval of 12 hours but I have thousands of users with their last full sync over 90 hours ago. "doveadm replicator status" also shows that i have over 35,000 queued full resync requests, but no sync, high or low queued requests so why aren't the full requests occuring? There are no errors in the logs. Thanks, Arnaud -- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes - smime.p7s Description: S/MIME Cryptographic Signature
Re: replication fails with "Error: sync: Unknown user in remote" but user shows up in doveadm user "*"
Hello Christian, I actually found why I had the problem. While I thought the ldap filters were the same, there was a sligth difference (damn crazy long filters!) Sorry about the noise. Arnaud On 22/04/2022 10:08, Christian Mack wrote: Hello Am 21.04.22 um 18:00 schrieb Arnaud Abélard: Hello, I've been trying to replicate a production server (debian buster, dovecot 2.3.4.1). But I nothing is actually being replicated and for each attempted user the message "Error: sync: Unknown user in remote" is being logged. The ldap settings are actually the same on both server (source and destnation) and the "unknown user" is actually showing up in doveadm user "*" on the destination server. I had already replicated 2 servers and used the same settings. Am I missing something obvious here? Can you da a doveadm user for that specific user? Are her/his attributes OK? Kind regards, Christian Mack -- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes - smime.p7s Description: S/MIME Cryptographic Signature
replication fails with "Error: sync: Unknown user in remote" but user shows up in doveadm user "*"
Hello, I've been trying to replicate a production server (debian buster, dovecot 2.3.4.1). But I nothing is actually being replicated and for each attempted user the message "Error: sync: Unknown user in remote" is being logged. The ldap settings are actually the same on both server (source and destnation) and the "unknown user" is actually showing up in doveadm user "*" on the destination server. I had already replicated 2 servers and used the same settings. Am I missing something obvious here? Thanks, Arnaud -- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes - smime.p7s Description: S/MIME Cryptographic Signature
Re: [SOLVED] after replication with compression quotas are wrong
Ah! At last, after comparing all default values using doveconf I found the culprit: mailbox_list_index. Default is "no" on dovecot 2.2.7, but "yes" on dovecot 2.3.4 (on debian, at least). Switching mailbox_list_index to "no" fixes my quota miscalculation problem. Arnaud On 12/11/2021 20:09, Arnaud Abélard wrote: On 10/11/2021 15:52, Arnaud Abélard wrote: I've just removed dummy-c-1's mailbox completely on the target server, removed the quota plugin, forced a sync for dummy-c-1 only and once it was done reactivated the quota plugins: double usage again. I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server in order to have both server running the same version, without improvement. Actually, I got that wrong, test server is running dovecot 2.3.4 and production server dovecot 2.2.7. I copied the dovecot.conf file from the production server to the test one, then rsync'ed the dummy-c-1 mailbox from the production server to the test server, restarted dovecot and the quota is still 115% after recalc. The only difference now is dovecot's version. Were there any changes related to quota on dovecot 2.3? Thanks, Arnaud I'm puzzled. Is dovecot storing anything outside of the user's mailbox? Like a cache, a sqlite database of some kind somewhere? Arnaud On 08/11/2021 11:48, Arnaud Abélard wrote: On the target, I enabled the replication service without mail_replica and: doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none - - - - It only knows of dummy-c-1, no trace of his evil twin dummy-...@univ-nantes.fr. On the target, I do have the same number of files find . -type f |wc -l 8705 which is around half of what the quota is reporting (plus de index files, etc): ~# doveadm -D quota get -u dummy-c-1 Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=10S doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: plugin/quota_rule=*:backend=10S doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, home=/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur backend=maildir args= doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=* bytes=10 messages=0 doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=INBOX.Trash bytes=+104857600 messages=0 doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur bytes=1 (10%) doveadm(dummy-c-1): Debug: replication: No mail_replica setting - replication disabled doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt= doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, co
Re: after replication with compression quotas are wrong
On 10/11/2021 15:52, Arnaud Abélard wrote: I've just removed dummy-c-1's mailbox completely on the target server, removed the quota plugin, forced a sync for dummy-c-1 only and once it was done reactivated the quota plugins: double usage again. I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server in order to have both server running the same version, without improvement. Actually, I got that wrong, test server is running dovecot 2.3.4 and production server dovecot 2.2.7. I copied the dovecot.conf file from the production server to the test one, then rsync'ed the dummy-c-1 mailbox from the production server to the test server, restarted dovecot and the quota is still 115% after recalc. The only difference now is dovecot's version. Were there any changes related to quota on dovecot 2.3? Thanks, Arnaud I'm puzzled. Is dovecot storing anything outside of the user's mailbox? Like a cache, a sqlite database of some kind somewhere? Arnaud On 08/11/2021 11:48, Arnaud Abélard wrote: On the target, I enabled the replication service without mail_replica and: doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none - - - - It only knows of dummy-c-1, no trace of his evil twin dummy-...@univ-nantes.fr. On the target, I do have the same number of files find . -type f |wc -l 8705 which is around half of what the quota is reporting (plus de index files, etc): ~# doveadm -D quota get -u dummy-c-1 Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=10S doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: plugin/quota_rule=*:backend=10S doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, home=/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur backend=maildir args= doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=* bytes=10 messages=0 doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=INBOX.Trash bytes=+104857600 messages=0 doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur bytes=1 (10%) doveadm(dummy-c-1): Debug: replication: No mail_replica setting - replication disabled doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt= doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(dummy-c-1): Debug: quota: quota_over_flag check: quota_over_script unset - skipping Quota name Type Value Limit % Quota Utilisateur STORAGE 1126751 976563 115 Quota Uti
Re: after replication with compression quotas are wrong
I've just removed dummy-c-1's mailbox completely on the target server, removed the quota plugin, forced a sync for dummy-c-1 only and once it was done reactivated the quota plugins: double usage again. I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server in order to have both server running the same version, without improvement. I'm puzzled. Is dovecot storing anything outside of the user's mailbox? Like a cache, a sqlite database of some kind somewhere? Arnaud On 08/11/2021 11:48, Arnaud Abélard wrote: On the target, I enabled the replication service without mail_replica and: doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none - - - - It only knows of dummy-c-1, no trace of his evil twin dummy-...@univ-nantes.fr. On the target, I do have the same number of files find . -type f |wc -l 8705 which is around half of what the quota is reporting (plus de index files, etc): ~# doveadm -D quota get -u dummy-c-1 Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=10S doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: plugin/quota_rule=*:backend=10S doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, home=/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur backend=maildir args= doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=* bytes=10 messages=0 doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=INBOX.Trash bytes=+104857600 messages=0 doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur bytes=1 (10%) doveadm(dummy-c-1): Debug: replication: No mail_replica setting - replication disabled doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt= doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(dummy-c-1): Debug: quota: quota_over_flag check: quota_over_script unset - skipping Quota name Type Value Limit % Quota Utilisateur STORAGE 1126751 976563 115 Quota Utilisateur MESSAGE 16686 - 0 The debug mode isn't much of any help here. Arnaud On 08/11/2021 11:30, Aki Tuomi wrote: Recalculation won't fix replication mistakes. Did you ensure on the *target* server that it has only dummy-c-1 in replication and that the file count on the target server matches source server? You might also get something useful out of `doveadm -D quota recalc|get -u user` Aki On 08/11/2021 12:22 Arnaud Abélard wrote: On 08/11/2021 10:39, Aki Tuomi wrote:
Re: after replication with compression quotas are wrong
On the target, I enabled the replication service without mail_replica and: doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none - - -- It only knows of dummy-c-1, no trace of his evil twin dummy-...@univ-nantes.fr. On the target, I do have the same number of files find . -type f |wc -l 8705 which is around half of what the quota is reporting (plus de index files, etc): ~# doveadm -D quota get -u dummy-c-1 Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=10S doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: plugin/quota_rule=*:backend=10S doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, home=/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur backend=maildir args= doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=* bytes=10 messages=0 doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=INBOX.Trash bytes=+104857600 messages=0 doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur bytes=1 (10%) doveadm(dummy-c-1): Debug: replication: No mail_replica setting - replication disabled doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt= doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(dummy-c-1): Debug: quota: quota_over_flag check: quota_over_script unset - skipping Quota nameType Value Limit % Quota Utilisateur STORAGE 1126751 976563 115 Quota Utilisateur MESSAGE 16686 - 0 The debug mode isn't much of any help here. Arnaud On 08/11/2021 11:30, Aki Tuomi wrote: Recalculation won't fix replication mistakes. Did you ensure on the *target* server that it has only dummy-c-1 in replication and that the file count on the target server matches source server? You might also get something useful out of `doveadm -D quota recalc|get -u user` Aki On 08/11/2021 12:22 Arnaud Abélard wrote: On 08/11/2021 10:39, Aki Tuomi wrote: Try doveadm replicator remove dummy-...@univ-nantes.fr I did so on the source server, checked indeed that dummy-...@univ-nantes.fr wasn't showing up in the doveadm replicator status list and it was indeed the case, so far so good. On the destination server, dummy-c-1's mailbox was still showing a 115% quota so I tried to make dovecot recalculate the quota but it did not change anything. I ended up deleting the user's mailbox and forced a replication and, to my surprise, the new mailbox still has a 115% quota. Arnaud Aki On
Re: after replication with compression quotas are wrong
On 08/11/2021 10:39, Aki Tuomi wrote:
Try doveadm replicator remove dummy-...@univ-nantes.fr
I did so on the source server, checked indeed that
dummy-...@univ-nantes.fr wasn't showing up in the doveadm replicator
status list and it was indeed the case, so far so good. On the
destination server, dummy-c-1's mailbox was still showing a 115% quota
so I tried to make dovecot recalculate the quota but it did not change
anything. I ended up deleting the user's mailbox and forced a
replication and, to my surprise, the new mailbox still has a 115% quota.
Arnaud
Aki
On 08/11/2021 11:32 Arnaud Abélard wrote:
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use
the domain part anymore and added "auth_username_format = %Ln" just in
case (and reloaded dovecot). So I shouldn't have any more new
user@domain users but for all the existing users, doveadm replicator
status still show user@domain clone and on the new server mails are
still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't
think it's coming from there.
Is there a local cache that still has a list of those user@domain
usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote:
This sounds like you are not normalizing usernames properly.
Either use
auth_username_format = %Ln
as global setting, or return `user` attribute in both userdb and passdb lookups.
Aki
On 07/11/2021 20:31 Arnaud Abélard wrote:
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm
replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*'
username priority fast sync full sync
success sync failed
dummy-c-1 none 01:13:19 01:13:19
01:13:19 -
dummy-...@univ-nantes.fr none 00:15:28 00:15:28
00:15:28 -
That'd explain why mails are counted twice when replicated on the new
server but where does this come from since I don't have this quota
problem on the production server?
From the logs, it seems that postfix uses usern...@univ-nantes.fr when
calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr'
mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using
the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*'
dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote:
Hello,
We are very long time happy dovecot users (since 2008 at least). We have
around 90k mailboxes and since we had to move away from our NAS storage
to a ceph storage I jumped on the opportunity to enable compression with
the zlib plugin and dovecot's replication mecanism. We are using
debian's dovecot 2.2.27 packages on production and our new server is
running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server,
replicated mailboxes' quota is all wrong:
on production:
# doveadm quota get -u dummy-c-1
Quota name Type Value Limit %
Quota Utilisateur STORAGE 660026 976563 67
Quota Utilisateur MESSAGE 8651 - 0
on new server:
doveadm quota get -u dummy-c-1
Quota name Type Value Limit %
Quota Utilisateur STORAGE 1125251 976563 115
Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly
the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F','
'{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
675865938
And I have exactly the same amountof mails on the two server, the
replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm
quota recalc -u , but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.17 (054dddfa)
# OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
# Hostname: gromel-test
auth_cache_size = 10 k
auth_verbose = yes
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
hostname = gromel1.univ-nantes.prive
lda_mailbox_autosubscribe = yes
listen = *
log_timestamp = "%Y-%m-%d %H:%M:%S "
login_trusted_networks = (...)
mail_gid = 5000
mail_location = maildir:%h
mail_plugins = quota zlib notify replication
mail_privileged_group = vmail
mail_uid = 5000
maildir_stat_dirs = yes
maildir_very_dirty_syncs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character subaddress comparator-i;ascii-numeric relational regex
imap4flags copy include va
Re: after replication with compression quotas are wrong
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use
the domain part anymore and added "auth_username_format = %Ln" just in
case (and reloaded dovecot). So I shouldn't have any more new
user@domain users but for all the existing users, doveadm replicator
status still show user@domain clone and on the new server mails are
still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't
think it's coming from there.
Is there a local cache that still has a list of those user@domain
usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote:
This sounds like you are not normalizing usernames properly.
Either use
auth_username_format = %Ln
as global setting, or return `user` attribute in both userdb and passdb lookups.
Aki
On 07/11/2021 20:31 Arnaud Abélard wrote:
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm
replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*'
username priority fast sync full sync
success sync failed
dummy-c-1 none 01:13:19 01:13:19
01:13:19 -
dummy-...@univ-nantes.fr none 00:15:28 00:15:28
00:15:28 -
That'd explain why mails are counted twice when replicated on the new
server but where does this come from since I don't have this quota
problem on the production server?
From the logs, it seems that postfix uses usern...@univ-nantes.fr when
calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr'
mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using
the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*'
dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote:
Hello,
We are very long time happy dovecot users (since 2008 at least). We have
around 90k mailboxes and since we had to move away from our NAS storage
to a ceph storage I jumped on the opportunity to enable compression with
the zlib plugin and dovecot's replication mecanism. We are using
debian's dovecot 2.2.27 packages on production and our new server is
running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server,
replicated mailboxes' quota is all wrong:
on production:
# doveadm quota get -u dummy-c-1
Quota name Type Value Limit %
Quota Utilisateur STORAGE 660026 976563 67
Quota Utilisateur MESSAGE 8651 - 0
on new server:
doveadm quota get -u dummy-c-1
Quota name Type Value Limit %
Quota Utilisateur STORAGE 1125251 976563 115
Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly
the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F','
'{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
675865938
And I have exactly the same amountof mails on the two server, the
replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm
quota recalc -u , but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.17 (054dddfa)
# OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
# Hostname: gromel-test
auth_cache_size = 10 k
auth_verbose = yes
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
hostname = gromel1.univ-nantes.prive
lda_mailbox_autosubscribe = yes
listen = *
log_timestamp = "%Y-%m-%d %H:%M:%S "
login_trusted_networks = (...)
mail_gid = 5000
mail_location = maildir:%h
mail_plugins = quota zlib notify replication
mail_privileged_group = vmail
mail_uid = 5000
maildir_stat_dirs = yes
maildir_very_dirty_syncs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character subaddress comparator-i;ascii-numeric relational regex
imap4flags copy include variables body enotify environment mailbox date
index ihave duplicate mime foreverypart extracttext
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf
driver = ldap
}
plugin {
quota = maildir:Quota Utilisateur
quota_exceeded_message = Cet utilisateur a dépassé son quota, votre
message n'a pu lui être livré.
quota_full_tempfail = yes
quota_rule = *:storage=1000M
quota_rule2 = INBOX.Trash:storage=+100M
sieve = ~/dovecot.sieve
sieve_dir = ~/sieve
sieve_extensions = -vacation
sieve_global_di
Re: after replication with compression quotas are wrong
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm
replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*'
username priority fast sync full sync
success sync failed
dummy-c-1 none 01:13:19 01:13:19
01:13:19 -
dummy-...@univ-nantes.fr none 00:15:28 00:15:28
00:15:28 -
That'd explain why mails are counted twice when replicated on the new
server but where does this come from since I don't have this quota
problem on the production server?
From the logs, it seems that postfix uses usern...@univ-nantes.fr when
calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr'
mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using
the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*'
dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote:
Hello,
We are very long time happy dovecot users (since 2008 at least). We have
around 90k mailboxes and since we had to move away from our NAS storage
to a ceph storage I jumped on the opportunity to enable compression with
the zlib plugin and dovecot's replication mecanism. We are using
debian's dovecot 2.2.27 packages on production and our new server is
running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server,
replicated mailboxes' quota is all wrong:
on production:
# doveadm quota get -u dummy-c-1
Quota name Type Value Limit %
Quota Utilisateur STORAGE 660026 976563 67
Quota Utilisateur MESSAGE 8651 - 0
on new server:
doveadm quota get -u dummy-c-1
Quota name Type Value Limit %
Quota Utilisateur STORAGE 1125251 976563 115
Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly
the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F','
'{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
675865938
And I have exactly the same amountof mails on the two server, the
replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm
quota recalc -u , but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.17 (054dddfa)
# OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
# Hostname: gromel-test
auth_cache_size = 10 k
auth_verbose = yes
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
hostname = gromel1.univ-nantes.prive
lda_mailbox_autosubscribe = yes
listen = *
log_timestamp = "%Y-%m-%d %H:%M:%S "
login_trusted_networks = (...)
mail_gid = 5000
mail_location = maildir:%h
mail_plugins = quota zlib notify replication
mail_privileged_group = vmail
mail_uid = 5000
maildir_stat_dirs = yes
maildir_very_dirty_syncs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character subaddress comparator-i;ascii-numeric relational regex
imap4flags copy include variables body enotify environment mailbox date
index ihave duplicate mime foreverypart extracttext
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf
driver = ldap
}
plugin {
quota = maildir:Quota Utilisateur
quota_exceeded_message = Cet utilisateur a dépassé son quota, votre
message n'a pu lui être livré.
quota_full_tempfail = yes
quota_rule = *:storage=1000M
quota_rule2 = INBOX.Trash:storage=+100M
sieve = ~/dovecot.sieve
sieve_dir = ~/sieve
sieve_extensions = -vacation
sieve_global_dir = /var/lib/dovecot/sieve/global/
sieve_max_redirects = 1
zlib_save = gz
zlib_save_level = 6
}
postmaster_address = postmaster@
protocols = imap pop3 sieve
replication_max_conns = 50
service auth {
client_limit = 49452
unix_listener auth-userdb {
group = vmail
mode = 0600
user = vmail
}
user = root
}
service dict {
unix_listener dict {
mode = 0600
user = vmail
}
}
service doveadm {
inet_listener {
port = 12345
}
}
service imap-login {
process_min_avail = 8
service_count = 0
user = mail
}
service imap {
executable = imap
process_limit = 16384
vsz_limit = 2 G
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
process_min_avail = 8
service_count = 0
user = mail
vsz_limit = 2 G
}
service managesieve {
drop_priv_before_exec = yes
process_limit = 16384
}
se
after replication with compression quotas are wrong
Hello,
We are very long time happy dovecot users (since 2008 at least). We have
around 90k mailboxes and since we had to move away from our NAS storage
to a ceph storage I jumped on the opportunity to enable compression with
the zlib plugin and dovecot's replication mecanism. We are using
debian's dovecot 2.2.27 packages on production and our new server is
running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server,
replicated mailboxes' quota is all wrong:
on production:
# doveadm quota get -u dummy-c-1
Quota nameType Value Limit %
Quota Utilisateur STORAGE 660026 97656367
Quota Utilisateur MESSAGE 8651 - 0
on new server:
doveadm quota get -u dummy-c-1
Quota nameType Value Limit %
Quota Utilisateur STORAGE 1125251 976563 115
Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly
the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F','
'{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
675865938
And I have exactly the same amountof mails on the two server, the
replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm
quota recalc -u , but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.17 (054dddfa)
# OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
# Hostname: gromel-test
auth_cache_size = 10 k
auth_verbose = yes
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
hostname = gromel1.univ-nantes.prive
lda_mailbox_autosubscribe = yes
listen = *
log_timestamp = "%Y-%m-%d %H:%M:%S "
login_trusted_networks = (...)
mail_gid = 5000
mail_location = maildir:%h
mail_plugins = quota zlib notify replication
mail_privileged_group = vmail
mail_uid = 5000
maildir_stat_dirs = yes
maildir_very_dirty_syncs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character subaddress comparator-i;ascii-numeric relational regex
imap4flags copy include variables body enotify environment mailbox date
index ihave duplicate mime foreverypart extracttext
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf
driver = ldap
}
plugin {
quota = maildir:Quota Utilisateur
quota_exceeded_message = Cet utilisateur a dépassé son quota, votre
message n'a pu lui être livré.
quota_full_tempfail = yes
quota_rule = *:storage=1000M
quota_rule2 = INBOX.Trash:storage=+100M
sieve = ~/dovecot.sieve
sieve_dir = ~/sieve
sieve_extensions = -vacation
sieve_global_dir = /var/lib/dovecot/sieve/global/
sieve_max_redirects = 1
zlib_save = gz
zlib_save_level = 6
}
postmaster_address = postmaster@
protocols = imap pop3 sieve
replication_max_conns = 50
service auth {
client_limit = 49452
unix_listener auth-userdb {
group = vmail
mode = 0600
user = vmail
}
user = root
}
service dict {
unix_listener dict {
mode = 0600
user = vmail
}
}
service doveadm {
inet_listener {
port = 12345
}
}
service imap-login {
process_min_avail = 8
service_count = 0
user = mail
}
service imap {
executable = imap
process_limit = 16384
vsz_limit = 2 G
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
process_min_avail = 8
service_count = 0
user = mail
vsz_limit = 2 G
}
service managesieve {
drop_priv_before_exec = yes
process_limit = 16384
}
service pop3-login {
process_min_avail = 8
service_count = 0
user = mail
}
service pop3 {
drop_priv_before_exec = yes
process_limit = 16384
}
ssl = no
userdb {
args = /etc/dovecot/dovecot-ldap.conf
driver = ldap
}
verbose_proctitle = yes
protocol imap {
mail_max_userip_connections = 50
mail_plugins = quota zlib notify replication imap_quota zlib
}
protocol pop3 {
mail_plugins = quota zlib notify replication
}
protocol sieve {
mail_max_userip_connections = 10
}
protocol lda {
mail_plugins = quota zlib notify replication sieve zlib
}
--
Arnaud Abélard
Responsable pôle Système et Stockage
Service Infrastructures
DSIN Université de Nantes
-
Re: [Dovecot] Lot of connections IMAP
On 09/26/2013 09:47 AM, voy...@sbt.net.au wrote: On Thu, September 26, 2013 5:41 pm, Davide wrote: Hi to all, i have dovecot 2.2.5 when i digit doveadm who i see a lot of connections IMAP for single user liker the example below x.yyy...@mail.cgilfe.it 9 imap (20572 20614 19120 20653 19136 20655 19138 20661 20471) (192.168.x.xxx) Why so many IMAP? I think(?), it's that IMAP mail clients keep connections to each IMAP folder they access... Exactly and even 9 imap processes for one user isn't that bad. If most of our users use around 5 processess it's not exceptionnel to have have 20 processes for one user. We have users who check their mail on their smartphones and on their desktop, some of those have around 80 imap process just for them... Arnaud -- Arnaud Abélard (jabber: arnaud.abel...@univ-nantes.fr) Administrateur Système - Responsable Services Web Direction des Systèmes d'Informations Université de Nantes - ne pas utiliser: trapem...@univ-nantes.fr
Re: [Dovecot] Mail deduplication
On 04/30/2013 08:05 AM, Angel L. Mateo wrote: El 30/04/13 03:28, Tim Groeneveld escribió: Hi Guys, I am wondering about mail deduplication. I am looking into the possibility of seperating out all of the message bodies with multiple parts inside mail that is recived from `dovecot` and hashing them all. The idea is that by hashing all of the parts inside the email, I will be able to ensure that each part of the email will only be saved once. This means that attachments common parts of the body will only be saved once inside the storage. How achievable would this be with the current state of dovecot? Would it even be worth doing? I asked the same question recently. As Timo responsed at http://kevat.dovecot.org/list/dovecot/2013-March/089072.html it seems that this feature is production stable in recent versions of dovecot. And I think it is worth. My estimations (with just about 10 users of my organization, they are no accurate) is that you can save more than 30% of total mail storage. To configure it you need to use options: * mail_attachment_dir * mail_attachement_min_size * mail_attachment_fs * mail_attachment_hash Hello, Is it just working or is it working in a optimal way? back in October 2011 we noticed that the deduplication wasn't working as well as we were expecting as some files weren't properly deduplicated (http://markmail.org/message/ymfdwng7un2mj26z). Timo did you ever hit that bug and got it fixed if there was anything to fix on your side? Since we are very interrested in this feature I am very eager to hear about admins using it on a similar scale (around 80,000 mailboxes). Thanks, Arnaud -- Arnaud Abélard (jabber: arnaud.abel...@univ-nantes.fr) Administrateur Système - Responsable Services Web Direction des Systèmes d'Informations Université de Nantes - ne pas utiliser: trapem...@univ-nantes.fr
[Dovecot] autocreate mailbox but no sub folders
Hello, We have been using lda_mailbox_autocreate to avoid to intialize empty mailboxes upon account creations. This does the job allright except that dovecot-lda is also allowed to autocreate sub folders which is a problem when using addresses extensions (adress+fol...@domain.tld). It's easy for a malicious user to flood a mailbox with tons of subfolders. Is there a way to tell dovecot-lda it can only create the initial mailbox and maybe some special folders but nothing else? Thanks, Arnaud -- Arnaud Abélard jabber: arnaud.abel...@univ-nantes.fr / twitter: ArnY Administrateur Système DSI Université de Nantes -
[Dovecot] Out of memory after upgrading to dovecot 2.0.21
Hello, I've upgraded our servers and now dovecot seems to be running out of memory when accessing my own mailbox (and only mine, which is in a way pretty fortunate): dovecot: imap(abelard-a): Error: mmap() failed with file /vmail/a/b/abelard-a/dovecot.index.cache: Cannot allocate memory imap(abelard-a): Fatal: pool_system_realloc(131072): Out of memory /vmail/a/b/abelard-a/dovecot.index.cache is 224MB big and my mailbox is 72% full of 8GB. any idea of what is happening? Thanks in advance, Arnaud -- Arnaud Abélard jabber: arnaud.abel...@univ-nantes.fr / twitter: ArnY Administrateur Système DSI Université de Nantes -
Re: [Dovecot] Out of memory after upgrading to dovecot 2.0.21
Hrm... exactly when I pressed sent, dovecot started giving me more
information:
master: Error: service(imap): child 22324 returned error 83 (Out of
memory (service imap { vsz_limit=256 MB }, you may need to increase it))
Is that why the imap processes were running out of memory or is that
just another symptom of the same problem? How come we never had that
problem with the older version of dovecot 2.0 (sorry, i can't remember
which one we were running...).
Arnaud
On 02/05/2013 06:33 PM, Arnaud Abélard wrote:
Hello,
I've upgraded our servers and now dovecot seems to be running out of
memory when accessing my own mailbox (and only mine, which is in a way
pretty fortunate):
dovecot: imap(abelard-a): Error: mmap() failed with file
/vmail/a/b/abelard-a/dovecot.index.cache: Cannot allocate memory
imap(abelard-a): Fatal: pool_system_realloc(131072): Out of memory
/vmail/a/b/abelard-a/dovecot.index.cache is 224MB big and my mailbox is
72% full of 8GB.
any idea of what is happening?
Thanks in advance,
Arnaud
--
Arnaud Abélard
jabber: arnaud.abel...@univ-nantes.fr / twitter: ArnY
Administrateur Système
DSI Université de Nantes
-
Re: [Dovecot] Out of memory after upgrading to dovecot 2.0.21
On 02/05/2013 06:44 PM, Reindl Harald wrote:
hard to say without config-details
shared prcoess or one per connection?
One per connection, around 1000 connections right now.
in case of shared 256 MB is really really low
I changed the vsz_limit to 512MB and it seems better, but I'm still
surprised my mailbox actually hit the memory limit since I doubt it's
most used one.
Arnaud
Am 05.02.2013 18:38, schrieb Arnaud Abélard:
Hrm... exactly when I pressed sent, dovecot started giving me more information:
master: Error: service(imap): child 22324 returned error 83 (Out of memory
(service imap { vsz_limit=256 MB }, you
may need to increase it))
Is that why the imap processes were running out of memory or is that just
another symptom of the same problem? How
come we never had that problem with the older version of dovecot 2.0 (sorry, i
can't remember which one we were
running...).
Arnaud
On 02/05/2013 06:33 PM, Arnaud Abélard wrote:
Hello,
I've upgraded our servers and now dovecot seems to be running out of
memory when accessing my own mailbox (and only mine, which is in a way
pretty fortunate):
dovecot: imap(abelard-a): Error: mmap() failed with file
/vmail/a/b/abelard-a/dovecot.index.cache: Cannot allocate memory
imap(abelard-a): Fatal: pool_system_realloc(131072): Out of memory
/vmail/a/b/abelard-a/dovecot.index.cache is 224MB big and my mailbox is
72% full of 8GB.
any idea of what is happening?
--
Arnaud Abélard
jabber: arnaud.abel...@univ-nantes.fr / twitter: ArnY
Administrateur Système
DSI Université de Nantes
-
Re: [Dovecot] Out of memory after upgrading to dovecot 2.0.21
On 02/05/2013 09:25 PM, Stan Hoeppner wrote:
On 2/5/2013 12:15 PM, Arnaud Abélard wrote:
I changed the vsz_limit to 512MB and it seems better, but I'm still
surprised my mailbox actually hit the memory limit since I doubt it's
most used one.
According to the wiki, vsz_limit only affects login processes, not IMAP
processes, which is odd given the second error you posted, which seems
to indicate a relationship between vsz_limit and imap.
Actually, service { vsz_limit } replaced mail_process_size (at least
according to what dovecot said upon restart...) so service imap {
vsz_limit=512MB } which i added earlier actually does what you suggested.
dovecot: imap(abelard-a): Error: mmap() failed with file
/vmail/a/b/abelard-a/dovecot.index.cache: Cannot allocate memory
imap(abelard-a): Fatal: pool_system_realloc(131072): Out of memory
This error suggests your mail_process_size variable is set too low. If
it is currently set to 128MB, increase it to 512MB. The bulk of this
memory is used for mmap()ing files, and is virtual, not physical. Thus
you don't actually need 51GB of RAM to support 100 users.
Also, a dovecot.index.cache file of size 224MB seems rather large. My
largest list mail folder is 150MB, contains 17,647 msgs, and has a
dovecot.index.cache file of only 16MB. I use mbox. This would seem to
suggest you may have a single folder with tens of thousands of msgs in
it. Or maybe indexes are handled differently for your mailbox
format--I've not used any others. If the former, I suggest you cull
your large folders to get them down to manageable size.
Yep I have a huge INBOX folder right now, I haven't archived my 2011
mail yet (I keep the previous year in my INBOX).
Thanks,
Arnaud
--
Arnaud Abélard
jabber: arnaud.abel...@univ-nantes.fr / twitter: ArnY
Administrateur Système
DSI Université de Nantes
-
[Dovecot] sieve_max_redirects=0 not working as documented
Hello
I'm trying to keep my users from using the redirect action with
pigeonhole from dovecot 2.0.21. As documented in
http://wiki2.dovecot.org/Pigeonhole/Sieve/Configuration I configured
sieve like this:
plugin {
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_max_redirects = 0
}
(and doveconf indeed shows the new setting for sieve_max_redirects)
Unfortunately I can still use the redirect action:
lda(du...@univ-nantes.fr): Debug: sieve: executing script from
/vmail/d/u/dummy/.dovecot.svbin
lda(du...@univ-nantes.fr): sieve:
msgid=510a758d.1030...@univ-nantes.fr: forwarded to x...@x.com
what could I have missed?
Thanks,
Arnaud
--
Arnaud Abélard (jabber: arnaud.abel...@univ-nantes.fr)
Administrateur Système - Responsable Services Web
Direction des Systèmes d'Informations
Université de Nantes
-
ne pas utiliser: trapem...@univ-nantes.fr
Re: [Dovecot] what best for anti-spam filter?
On 07/24/2012 06:49 AM, Noel Butler wrote: On Tue, 2012-07-24 at 11:58 +0800, fy wrote: what anti-spam for you used ? dspam?spammassian? amavisd-new ? what is best ? amavisd-new with spamassassin and anti virus scanner, clamav with sanesecurity rules use enforcing rules in mail server, like block hosts with no DNS/rDNS Enforce SPF, publish SPF with hardfail use DNSBL's in your mail server, spamhaus, spamcop, spam.sorbs, and more etc. milter regex to stop dynamic/suspect hosts And first of all, even if this is not dovecot related, use a greylisting solution. There is no one solution, the solution, is a box of many tricks You might get the odd false blocking, but if system opers can not be bothered running a compliant network with standardised naming conventions for servers, then it is not my problem, and we have had very very very few complaints about this type of policy in over a decade. If someoine sooks to you, educate them, dont whitelist them. Indeed! Fighting spam is a continuous task. While greylisting will cut down the amount of spam by more than 50%, the remaining 50% will give you the hardest time and will keep changing to bypass your rules. You'll need to keep an eye on the flow of false negatives or false positive you are getting... We (72,000 mailboxes) are currently using amavisd-new with spamassassin and CRM114 via a custom plugin instead of the default bayesian filter. Also like Noel, we're using DNSBLs, SPF (although we had to publish a permissive record since some of our users are using their ISP smtp instead of our own). Arnaud -- Arnaud Abélard (jabber: arnaud.abel...@univ-nantes.fr) Administrateur Système - Responsable Services Web Direction des Systèmes d'Informations Université de Nantes - ne pas utiliser: trapem...@univ-nantes.fr
Re: [Dovecot] what best for anti-spam filter?
On 07/24/2012 10:49 AM, Morten Stevens wrote: On 24.07.2012 09:16, Arnaud Abélard wrote: And first of all, even if this is not dovecot related, use a greylisting solution. No, greylisting is really a bad solution. It is not RFC compliant and delays the mail traffic. I would prefer a pre-queue content-filtering solution like MIMEDefang or amavisd-new. Best regards, Morten As far as I know greylisting is RFC compliant it even has its own RFC: http://tools.ietf.org/html/rfc6647 Greylisting was prefered for a simple reason If a mail is accepted on our servers we have to deliver it to the user (unless it has a virus) - With greylisting we aren't rejecting potentially spammy mails, we are rejecting misbehaving servers. That's important, legally speaking. We could be in trouble if we rejected an important mail by mistake when our server actually accepted it. - Our users already complain they are getting too much spams (we mark them with [SPAM] in the subject). Without greylist and for the reason I stated above, even if we were able to detect the bot spam as spams, we wouldn't want to reject it and our users would be flooded. Greylisting isn't a perfect solution. Just a good one. Depends on what you want, I guess. Arnaud -- Arnaud Abélard (jabber: arnaud.abel...@univ-nantes.fr) Administrateur Système - Responsable Services Web Direction des Systèmes d'Informations Université de Nantes - ne pas utiliser: trapem...@univ-nantes.fr
Re: [Dovecot] what best for anti-spam filter?
On 07/24/2012 10:38 AM, Stan Hoeppner wrote: On 7/24/2012 2:16 AM, Arnaud Abélard wrote: And first of all, even if this is not dovecot related, use a greylisting solution. Greylisting only stops bots. It is resource intensive, and causes delivery delays. There exist bot spam killing solutions that are just as effective, with less downside. Two are Postfix' postscreen daemon, and fqrdns.pcre, which rejects based on consumer/dynamic looking rDNS. Some users have modified the latter for use on HELO strings instead of client rDNS strings, with good success. Either combined with CBL/ZEN should kill all your bot spam much more efficiently. I'm surprised you're using greylisting (Postgrey?) with 72k mailboxes. Greylisting only stops bots. Exactly. That's the whole point! We have been using sqlgrey for now 5 years and we only had one problem last month with OVH smtp infrastructure which sucks and we're happy to see mails bouncing from them, hoping their customers will complain. But I can understand why you would think greylist is trouble. It depends on how you set it up. One mail delayed per domain and per month is really nothing compared to hundred thousands of bot spams we are rejecting. dynamic/consumer ip range DNSBL are dangerous since they are rarely up to date, I can painfully remember that. I guess it all depends on what kind of smtp traffic you get. As a large university we aren't getting the same traffic as a big corporate company which will mostly communicate with other business. We are getting tons of individual mails from local ISPs, lot of geeks hosting their servers at home (a lot of ppl do that here...), etc. Indeed! Fighting spam is a continuous task. Unfortunately... We (72,000 mailboxes) are currently using amavisd-new with spamassassin and CRM114 via a custom plugin instead of the default bayesian filter. Also like Noel, we're using DNSBLs, SPF (although we had to publish a permissive record since some of our users are using their ISP smtp instead of our own). Which of your countermeasures blocks spam from Orange/France Telecom VPS/colo sources? Ahah.. that's a good question! since we are a french university we are also getting tons of clean mails from Orange/FT. But the problem isn't as bad as it used to be since Orange is now blocking direct outgoing traffic on port 25 for a few years now. Back then the DNSRBL were a good solution for spams coming from them. Now the new pain in the ass is OVH, the largest european hosting company which also has the worst smtp infrastructure that will not play well with greylist (tons of smtp servers, each on a different ip range so you can't even whitelist them by their networks). Arnaud -- Arnaud Abélard (jabber: arnaud.abel...@univ-nantes.fr) Administrateur Système - Responsable Services Web Direction des Systèmes d'Informations Université de Nantes - ne pas utiliser: trapem...@univ-nantes.fr
Re: [Dovecot] what best for anti-spam filter?
On 07/24/2012 02:06 PM, Markus Schönhaber wrote: 24.07.2012 11:57, Arnaud Abélard: - With greylisting we aren't rejecting potentially spammy mails, we are rejecting misbehaving servers. That's important, legally speaking. We could be in trouble if we rejected an important mail by mistake when our server actually accepted it. That's something which is not greylisting-specific at all. You must not accept mail you are unwilling or unable to deliver - ever! That's my point. Greylisting screens bad behaviored servers away and if a mail is accepted it will be delivered. If it's detected as a potential spam, it will still be delivered to the end user with a proper tag in the subject. From what I just read, it seems that indeed postscreen could be an alternative for that purpose. But screening solutions aren't enough since a lot of unwanted mails are sent from legit RFC compliant servers: - newsletters from sites the users provided their email to and forgot they did. - digital prospecting which is legal if properly done (in France, it must be related to your professionnal field of activity and an unsubscribe link must be provided) - phishing and scams sent from stolen webmail accounts. Greylisting and DNSBL aren't really useful for any of those, only content analysis will catch them and that's the hard part. Bayesian and markovian filters need training and corrections. Spamassassin rules needs to be added every few weeks, etc. I kind of like how pyzor and razor work but those are rather slow and tend to use too much CPU. Anyone here who had a good experience with those? Arnaud Creating bounces will make you a source of backscatter and get you blacklisted, eventually. (Outgoing mail is a different matter, of course) But that doesn't mean that greylisting is the only means for fighting spam that is compliant to the above rule. It's, for example, not uncommon to have things like milters or pre-queue filters pipe the incoming mail through a spam checker and accept or reject the mail - during the SMTP dialogue - depending on the result of the check. -- Arnaud Abélard (jabber: arnaud.abel...@univ-nantes.fr) Administrateur Système - Responsable Services Web Direction des Systèmes d'Informations Université de Nantes - ne pas utiliser: trapem...@univ-nantes.fr
Re: [Dovecot] Feature request
On 07/07/2012 04:35 PM, Reindl Harald wrote: Am 07.07.2012 16:26, schrieb Malloc Kilobyte: Appreciating all Dovecot rich features, I lack just one. And this is the ability to customize the quota exceeded, message rejected message. I know I can set it's default content using quota_exceeded_message parameter, but i would also like to have ability to set: - to whom the message is sent. Some mail systems send copy not only to the sender, but also to the recipient. It makes sense in case rejected message has big size. - it's subject - if the rejected message is being attached ( with or without it's own attachments ) or not Moreover, I think it would make sense if the rejected messege contained information about size of rejected message and current mailbox space usage. hmm - dovecot doe snot send any mail so there is not sobject and to whom the message is sen messages are typically only REJECTED eith status code and status message the mail itself is based on this jerecht code/message from the delivering MTA Then the cool feature could be to be able to make the quota plugin call an external script/command which could do whatever the admins wish: Send an email to a backup account, modify a database, set off an alarm, phone the president, slap the user in the face or whatever. As far as I know that feature could already exist since the idea just popped to me and didn't bother checking the documentation yet. So if it is so, my bad. Arnaud -- Arnaud Abélard jabber: arnaud.abel...@univ-nantes.fr / twitter: ArnY Administrateur Système DSI Université de Nantes -
[Dovecot] RECENT status always 0
Hello, we are using dovecot 2.0.13 with maildir++ (we migrated away from courrier just a few months ago) and the RECENT status doesn't seem to be working: . STATUS INBOX (MESSAGES UNSEEN RECENT) * STATUS INBOX (MESSAGES 35106 RECENT 0 UNSEEN 10) then 2 minutes later: . STATUS INBOX (MESSAGES UNSEEN RECENT) * STATUS INBOX (MESSAGES 35106 RECENT 0 UNSEEN 11) How can the UNSEEN count change without the RECENT count changing accordingly? Thanks in advance, Arnaud -- Arnaud Abélard (jabber: arnaud.abel...@univ-nantes.fr) Administrateur Système - Responsable Services Web Direction des Systèmes d'Informations Université de Nantes - ne pas utiliser: trapem...@univ-nantes.fr
[Dovecot] keywords/flags questions
Hello, I am currently using dovecot 2.0.13 and I have been working on keywords handling our webmail and I have a few questions about how dovecot handles them. First, if I am not mistaken keywords neeed to be UTF-7 encoded. That means I need to encode special caracters using values between and - chars. UTF-7 encoding is case sensitive, AOA- isn't the same chars as aoa-. But docevot save keywords in lowercase or am I mistaken? For example: . STORE 1:1 flags AOA-_refaire * 1 FETCH (FLAGS (aoa-_refaire)) This makes retrieving the keyword properly impossible. What did I miss? My other question is about the permanent flags being displayed upon selecting a mailbox. I'm trying to understand why unused keywords are still showing up in there. Is there a way to force the definitive removal of a keyword from a mailbox? are old keywords kept undefinitely? Thanks in advance, Arnaud -- Arnaud Abélard jabber: arnaud.abel...@univ-nantes.fr / twitter: ArnY Administrateur Système DSI Université de Nantes -
