ok a little bit more info ....
My servers all run under FreeBSD 12.xx
which was also the base for the apple operating system origionally.
setting default_vsz_limit = 0 i seem to remember trying with no so great
results
setting to zero can cause memory over runs (espically with replication)
etc i found that when i used the config i sent eariler (vsz_limit is
defaulted (not set )) everything worked
I generally find that tweaking the memory alloted to the individual
services a more balanced approach but it does take a lot of trial and error
also note memory in the system is also a factor my mail servers have 32G
dedicated to them which is what the settings were based on that seem to
work pretty good at the moment.
I am running without any setting thus the default
I got this info from :
https://doc.dovecot.org/configuration_manual/service_configuration/
vsz_limit
Limit the process’s address space (both RLIMIT_DATA and RLIMIT_AS if
available). When the space is reached, some memory allocations may start
failing with “Out of memory”, or the kernel may kill the process with
signal 9. This setting is mainly intended to prevent memory leaks from
eating up all of the memory, but there can be also legitimate reasons
why the process reaches this limit. For example a huge mailbox may not
be accessed if this limit is too low. The default value
(18446744073709551615=2^64-1) sets the limit to default_vsz_limit, while
0 disables the limit entirely.
There are 3 types of services that need to be optimized in different ways:
Master services (e.g. auth, anvil, indexer, director, log):
Currently there isn’t any easy way to optimize these. If these
become a bottleneck, typically you need to run another Dovecot server.
In some cases it may be possible to create multiple master processes and
have each one be responsible for only specific users/processes, although
this may also require some extra development.
Services that do disk I/O or other blocking operations (e.g. imap,
pop3, lmtp):
These should have client_limit=1, because any blocking
operation will block all the other clients and cause unnecessary delays
and even timeouts. This means that process_limit specifies the maximum
number of available parallel connections.
Services that have no blocking operations (e.g. imap-login,
pop3-login):
For best performance (but a bit less safety), these should have
process_limit and process_min_avail set to the number of CPU cores, so
each CPU will be busy serving the process but without unnecessary
context switches. Then client_limit needs to be set high enough to be
able to serve all the needed connections (max connections=process_limit
* client_limit). service_count is commonly set to unlimited (0) for
these services. Otherwise when the service_count is beginning to be
reached, the total number of available connections will shrink. With
very bad luck that could mean that all the processes are simply waiting
for the existing connections to die away before the process can die and
a new one can be created. Although this could be made less likely by
setting process_limit higher than process_min_avail, but that’s still
not a guarantee since each process could get a very long running
connection and the process_limit would be eventually reached.
Happy Saturday !!!
Thanks - paul
Paul Kudla
Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3
Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca
On 1/6/2023 5:20 PM, Gerben Wierda wrote:
How problematic is it to have
default_vsz_limit = 0
in dovecot.conf? macOS+MacPorts had this as a requirement even.
Gerben
On 6 Jan 2023, at 16:49, Paul Kudla <p...@scom.ca
<mailto:p...@scom.ca>> wrote:
i ran into this as well
here is the full config for mine with replication
# cat dovecot.conf
# 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 12.1-RELEASE amd64
# Hostname: mail18.scom.ca <http://mail18.scom.ca>
auth_debug = no
auth_debug_passwords = no
default_process_limit = 16384
mail_debug = no
#lock_method = dotlock
#mail_max_lock_timeout = 300s
#mbox_read_locks = dotlock
#mbox_write_locks = dotlock
mmap_disable = yes
dotlock_use_excl = no
mail_fsync = always
mail_nfs_storage = no
mail_nfs_index = no
auth_mechanisms = plain login
auth_verbose = yes
base_dir = /data/dovecot/run/
debug_log_path = syslog
disable_plaintext_auth = no
dsync_features = empty-header-workaround
info_log_path = syslog
login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_location = maildir:~/
mail_plugins = " virtual notify replication fts fts_lucene "
mail_prefetch_count = 20
protocols = imap pop3 lmtp sieve
protocol lmtp {
mail_plugins = $mail_plugins sieve
postmaster_address =
}
service lmtp {
process_limit=1000
vsz_limit = 512m
client_limit=1
unix_listener /usr/home/postfix.local/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
protocol lda {
mail_plugins = $mail_plugins sieve
}
service lda {
process_limit=1000
vsz_limit = 512m
}
service imap {
process_limit=4096
vsz_limit = 2g
client_limit=1
}
service pop3 {
process_limit=1000
vsz_limit = 512m
client_limit=1
}
namespace inbox {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = /
}
passdb {
args = /usr/local/etc/dovecot/dovecot-pgsql.conf
driver = sql
}
doveadm_port = 12345
doveadm_password = secretxxxx
service doveadm {
process_limit = 0
process_min_avail = 0
idle_kill = 0
client_limit = 1
user = vmail
inet_listener {
port = 12345
}
}
service config {
unix_listener config {
user = vmail
}
}
dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
#dsync_remote_cmd = doveadm sync -d -u%u
replication_dsync_parameters = -d -N -l 300 -U
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags
push_notification_driver = dlog
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
#sieve = ~/.dovecot.sieve
sieve_duplicate_default_period = 1h
sieve_duplicate_max_period = 1h
sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
sieve_global_dir = /usr/local/etc/dovecot/sieve
sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
mail_replica = tcp:10.221.0.19:12345
#mail_replica = remote:vmail@10.221.0.19 <mailto:vmail@10.221.0.19>
#replication_sync_timeout = 2
fts = lucene
fts_lucene = whitespace_chars=@.
fts_autoindex = yes
fts_languages = en
}
#sieve_extensions = vnd.dovecot.duplicate
#sieve_plugins = vnd.dovecot.duplicate
service anvil {
process_limit = 1
client_limit=5000
vsz_limit = 512m
unix_listener anvil {
group = vmail
mode = 0666
}
}
service indexer-worker {
vsz_limit = 2g
}
service auth {
process_limit = 1
client_limit=5000
vsz_limit = 1g
unix_listener auth-userdb {
mode = 0660
user = vmail
group = vmail
}
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
}
service stats {
process_limit = 1000
vsz_limit = 1g
unix_listener stats-reader {
group = vmail
mode = 0666
}
unix_listener stats-writer {
group = vmail
mode = 0666
}
}
userdb {
args = /usr/local/etc/dovecot/dovecot-pgsql.conf
driver = sql
}
protocol imap {
mail_max_userip_connections = 50
mail_plugins = $mail_plugins notify replication
}
protocol pop3 {
mail_max_userip_connections = 50
mail_plugins = $mail_plugins notify replication
}
protocol imaps {
mail_max_userip_connections = 25
mail_plugins = $mail_plugins notify replication
}
protocol pop3s {
mail_max_userip_connections = 25
mail_plugins = $mail_plugins notify replication
}
service managesieve-login {
process_limit = 1000
vsz_limit = 1g
inet_listener sieve {
port = 4190
}
}
verbose_proctitle = yes
replication_max_conns = 100
replication_full_sync_interval = 1d
service replicator {
client_limit = 0
drop_priv_before_exec = no
idle_kill = 4294967295s
process_limit = 1
process_min_avail = 0
service_count = 0
vsz_limit = 8g
unix_listener replicator-doveadm {
mode = 0600
user = vmail
}
vsz_limit = 8192M
}
service aggregator {
process_limit = 1000
#vsz_limit = 1g
fifo_listener replication-notify-fifo {
user = vmail
group = vmail
mode = 0666
}
}
service pop3-login {
process_limit = 1000
client_limit = 100
vsz_limit = 512m
}
service imap-urlauth-login {
process_limit = 1000
client_limit = 1000
vsz_limit = 1g
}
service imap-login {
process_limit=1000
client_limit = 1000
vsz_limit = 1g
}
protocol sieve {
managesieve_implementation_string = Dovecot Pigeonhole
managesieve_max_line_length = 65536
}
#Addition ssl config
!include sni.conf
with sni cert support (examples)
# cat sni.conf
#sni.conf
ssl = yes
verbose_ssl = yes
ssl_dh =</usr/local/etc/dovecot/dh-4096.pem
ssl_prefer_server_ciphers = yes
#ssl_min_protocol = TLSv1.2
#Default *.scom.ca <http://scom.ca>
ssl_key =</usr/local/etc/dovecot/scom.pem
ssl_cert =</usr/local/etc/dovecot/scom.pem
ssl_ca =</usr/local/etc/dovecot/scom.pem
local_name .scom.ca <http://scom.ca> {
ssl_key =</usr/local/etc/dovecot/scom.pem
ssl_cert =</usr/local/etc/dovecot/scom.pem
ssl_ca =</usr/local/etc/dovecot/scom.pem
}
local_name mail.clancyca.com <http://mail.clancyca.com> {
ssl_key =</usr/local/etc/dovecot/cert/mail.clancyca.com
<http://mail.clancyca.com>
ssl_cert =</usr/local/etc/dovecot/cert/mail.clancyca.com
<http://mail.clancyca.com>
ssl_ca =</usr/local/etc/dovecot/cert/mail.clancyca.com
<http://mail.clancyca.com>
}
local_name mail.paulkudla.net <http://mail.paulkudla.net> {
ssl_key =</usr/local/etc/dovecot/cert/mail.paulkudla.net
<http://mail.paulkudla.net>
ssl_cert =</usr/local/etc/dovecot/cert/mail.paulkudla.net
<http://mail.paulkudla.net>
ssl_ca =</usr/local/etc/dovecot/cert/mail.paulkudla.net
<http://mail.paulkudla.net>
}
local_name mail.ekst.ca <http://mail.ekst.ca> {
ssl_key =</usr/local/etc/dovecot/cert/mail.ekst.ca <http://mail.ekst.ca>
ssl_cert =</usr/local/etc/dovecot/cert/mail.ekst.ca <http://mail.ekst.ca>
ssl_ca =</usr/local/etc/dovecot/cert/mail.ekst.ca <http://mail.ekst.ca>
}
local_name mail.hamletdevelopments.ca
<http://mail.hamletdevelopments.ca> {
ssl_key =</usr/local/etc/dovecot/cert/mail.hamletdevelopments.ca
<http://mail.hamletdevelopments.ca>
ssl_cert =</usr/local/etc/dovecot/cert/mail.hamletdevelopments.ca
<http://mail.hamletdevelopments.ca>
ssl_ca =</usr/local/etc/dovecot/cert/mail.hamletdevelopments.ca
<http://mail.hamletdevelopments.ca>
}
pg sql support supporting replication
# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname=scom_billing user=pgsql
password=
default_pass_scheme = PLAIN
password_query = SELECT username as user, password FROM email_users
WHERE username = '%u' and password <> 'alias' and status = True and
destination = '%u'
user_query = SELECT home, uid, gid FROM email_users WHERE username =
'%u' and password <> 'alias' and status = True and destination = '%u'
#iterate_query = SELECT user, password FROM email_users WHERE username
= '%u' and password <> 'alias' and status = True and destination = '%u'
iterate_query = SELECT "username" as user, domain FROM email_users
WHERE status = True and alias_flag = False
Happy Friday !!!
Thanks - paul
Paul Kudla
Scom.ca <http://Scom.ca> Internet Services <http://www.scom.ca
<http://www.scom.ca>>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3
Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca <mailto:p...@scom.ca>
On 1/6/2023 5:32 AM, Gerben Wierda wrote:
On 6 Jan 2023, at 08:53, Aki Tuomi <aki.tu...@open-xchange.com
<mailto:aki.tu...@open-xchange.com>> wrote:
On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda
<gerben.wie...@rna.nl <mailto:gerben.wie...@rna.nl>> wrote:
One step further in my quest to create a replacement mail server.
I now have my old mail server (2.3.19.1, macOS + MacPorts) and my
new (2.3.20, Alpine Linux, Docker, apk package). When I turn on
replication it works, but, after a while I see:
Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when
allocating 268435496 bytes
Jan 06 00:50:32 replicator: Fatal: master: service(replicator):
child 133 killed with signal 6 (core dumped)
Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning:
replication(sysbh): Sync failure:
Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning:
replication(sysbh): Remote sent invalid input: -
I've removed synchronous operation for now (found a message on the
net suggesting that) but is this known and what does it mean?
Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda
<https://www.linkedin.com/in/gerbenwierda>>)
R&A IT Strategy <https://ea.rna.nl/ <https://ea.rna.nl/>> (main site)
Book: Chess and the Art of Enterprise Architecture
<https://ea.rna.nl/the-book/ <https://ea.rna.nl/the-book/>>
Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/
<https://ea.rna.nl/the-book-edition-iii/>>
Dovecot default memory limit is 256M. You should probably set
service replicator {
vsz_limit = 2G
}
because replicator might have to use more memory, especially for
larger indexes.
Aki
That is a good tip as well.
I had followed this bit of experience from someone else:
https://marc.info/?l=dovecot&m=164438199727640
<https://marc.info/?l=dovecot&m=164438199727640>, haven't seen any
err message since. But that might be because they are in sync now and
both sides are aware. Can I trigger full replication again so I can test?
Gerben
--
This message has been scanned for viruses and
dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
believed to be clean.
--
This message has been scanned for viruses and
dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
believed to be clean.
