Re: mmap_disable=yes not honored always

[Timo Sirainen]

On 10/15/2014 06:31 PM, Peter Benko wrote:
>  garbled>
> 
> Hi all,
>  
> I'm experimenting with having the mail store on a 9p file system that lacks 
> mmap() functionality. So I disabled it in dovecot:
>  
> mmap_disable = yes
>  
> However, I keep getting the following error messages in my log:
>  
> Oct 15 16:55:00 computer-name dovecot: imap u...@domain.com[192.168.1.3] 
> Error: mmap() failed with file 
> /mail/mailboxes/domain.com/user/indexpvt-accounts/other-user/mailboxes/INBOX/dovecot.index.pvt.log:
>  Invalid argument
>  
> I only get those messages for the shared folder's private indexes. This setup 
> works perfectly on an ext2/3/4 fs.
>  
> Any opinions, is this a bug? Is mmap_disable not mandatory for all operations?

Took a while, but fixed finally:
http://hg.dovecot.org/dovecot-2.2/rev/00817b01020a

Re: Dovecot cluster using GlusterFS

[Filip Pytloun]

Hello Alessio and Gordon, thank you for answers.

Dsync-based architecture looks promising, but I would preffer to stay
with GlusterFS for now as I also use it as a storage for other
components.

So director is the way to go, I don't want to setup more than two nodes
to keep this setup as simple as possible - so I will probably update to
2.2.19 and have director and backend on the same servers (and Dovecot
instance).

I asked about poolmon, because I think that Dovecot should have some
internal mechanism on how to recognize broken backend by default.
But if it works nicely, I am going to use it as well :-)

> At the moment, I cannot recognize the requirement for using lmtp over
> the directors. When using postfix for delivering e-mails to the
> backend, do this directly with an corresponding MX record.

I have two MX records of the same weight with postfix using dovecot-lmtp
for delivery. So that's why I wanted to use LMTP over directors.
Using lower weight for second MX is an option, but not truly
master-master setup :-)

Filip

On 2015/12/06 02:31, Alessio Cecchi wrote:
> Il 05.12.2015 10:42 Filip Pytloun ha scritto:
> >Hello,
> >
> >I have recently setup mailserver solution using 2-node master-master
> >setup (mainly based on MySQL M-M replication and GlusterFS with 2
> >replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
> >
> >Unfortunately even with shared-storage-aware setting:
> >
> > mail_nfs_index = yes
> > mail_nfs_storage = yes
> > mail_fsync = always
> > mmap_disable = yes
> 
> With only these setting you don't solve the problem of shared storage.
> 
> >..I have hit strange issues pretty soon especially when user was
> >manipulating same mailbox from multiple devices at the same time.
> >
> >Most issues was about corrupted indexes which was solved easily by just
> >putting them on local storage of each node:
> >
> > mail_location =
> >maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
> >
> >But I still hit issues like this one:
> >
> > dovecot: lmtp(6276, u...@example.com): Error: Broken file
> >/srv/mail/example.com/u...@example.com/dovecot-uidlist line 8529: UIDs
> >not ordered (8527 >= 8527)
> >
> >Which I am not sure how serious it is or if it's possible to solve or
> >workaround?
> 
> You need Director for POP/IMAP and also LMTP so you can solve all "Broken
> file" and "corrupted indexes" problems.
> 
> >
> >Anyway because of the above and high possibility of GlusterFS
> >split-brains, I have decided to setup Dovecot Director according to the
> >docs [1] but I have a couple of questions:
> >
> >- is custom monitoring still required? Poolmon [2] is 4 year old so I
> >  would suppose there's some progress since that?
> 
> For me poolmon works fine.
> 
> >- it's not possible to have same backends and directors in Dovecot
> >  <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is
> >  not an issue, but..
> 
> Yes is possibile (also with < 2.2.17), create two instances, like dovecot
> and director, two config directory /etc/dovecot/ and /etc/director/ and bind
> on differents IPs.
> 
> >- documentation states that it still doesn't work for LMTP [3]?
> >  Which is probably important for my setup, because both Postfix servers
> >  are using dovecot-lmtp for mail delivery so there can be still some
> >  issues (but probably less frequent?) when both servers will deliver
> >  new mails for one user at once.
> >  So do I really have to split directors from backends?
> 
> I'm running Director and backend on the same server for POP/IMAP, and in
> another configuration and Director for LMTP is on the same server (but with
> 2.2.19).
> 
> >Anyone has experience with clustered Dovecot setup?
> >Why is Dovecot behaving so bad when it pretends to be shared storage
> >friendly? Are these issues only specific for older Dovecot?
> >Or is there something wrong in my architecture design?
> 
> You need Director, Dovecot has not problems with shared storage, big
> installation are always using shared storage (like NFS).
> -- 
> Alessio Cecchi
> Postmaster AT http://www.qboxmail.it
> http://www.linkedin.com/in/alessice


signature.asc
Description: Digital signature

Re: LIST MANAGEMENT BROKEN

[Timo Sirainen]

> On 06 Dec 2015, at 05:05, Edwardo Garcia  wrote:
> 
> Timo
> I have for two days try to unsubscribe from this list using email mailman.
> The list server does not send me confirmation request.
> Please fix your server and remove me

Your mail kept getting stuck at spamassassin for some reason. I guess it would 
need some reconfiguration.

Re: quota_full_tempfail = yes broken with lmtp_rcpt_check_quota (was: Re: lmtp_rcpt_check_quota not working)

[Timo Sirainen]

> On 29 Jan 2015, at 14:53, Bernhard Schmidt  wrote:
> 
> - In 2.2.9 and 2.2.13 lmtp_rcpt_check_quota is ignored
> - In 2.2.15 lmtp_rcpt_check_quota works, but quota_full_tempfail is
>  ignored at the RCPT TO stage. It still works at the DATA stage, so
>  disabling lmtp_rcpt_check_quota is a workaround falling back to 2.2.13
>  behaviour

Finally fixed in upcoming v2.2.20.

Re: [PATCH] [dovecot 2.2.9] Quota warnings ignored with FS quotas

[Timo Sirainen]

On 23 Mar 2015, at 22:32, Michał Giżyński  wrote:
> 
> 
> On 17.02.2015 20:03, Michał Giżyński wrote:
>> 
>> On 21.11.2014 20:04, Grzegorz Nosek wrote:
>>> Hi all,
>>> 
>>> I noticed that in some circumstances quota warnings are ignored. The bug 
>>> arises when both of the following are used:
>>> 
>>> 1. percentage-based quota warnings, i.e.:
>>> 
>>>  quota_warning = storage=1%% quota-warning 1 %u
>>> 
>>> 2. filesystem quota backend (and probably others, except for quotas 
>>> configured directly in dovecot config)
>>> 
>>> Percentage-based quota warnings have rule.bytes_limit recalculated based on 
>>> root_set->default_rule.bytes_limit, however this value is zero when FS 
>>> quotas are in use. Real quota values (from quotactl) are fetched very late, 
>>> in quota_warnings_execute() but at that point no recalculation happens. As 
>>> the warning rules have bytes_limit==0, they're effectively ignored.
>>> 
>>> The patch below enables quota warnings to be sent when using filesystem 
>>> (and possibly maildirsize-based) quotas.
>>> 
>>> Based and tested on Ubuntu 14.04's dovecot 2.2.9.
>>> 
>>> Best regards,
>>> Grzegorz Nosek
>>> 
>>> 
>>> diff --git a/src/plugins/quota/quota.c b/src/plugins/quota/quota.c
>>> index adbd70d..8e4d7e0 100644
>>> --- a/src/plugins/quota/quota.c
>>> +++ b/src/plugins/quota/quota.c
>>> @@ -1163,6 +1163,8 @@ static void quota_warnings_execute(struct 
>>> quota_transaction_context *ctx,
>>>   _current, _limit) < 0)
>>>return;
>>> 
>>> +   quota_root_recalculate_relative_rules(root->set, bytes_limit, 
>>> count_limit);
>>> +
>>>bytes_before = bytes_current - ctx->bytes_used;
>>>count_before = count_current - ctx->count_used;
>>>for (i = 0; i < count; i++) {
>> Hi,
>> this patch realy fix the bug in quota_warning. Is it possible to add this 
>> patch to the next release ?
>> 
> Hi,
> I saw that new dovecot was released. Is the quota_warning has been improved ?

That patch is doing a bit too many recalculations, which also always trigger 
debug message logging if mail_debug=yes. The attached patch probably fixes the 
problem? I'll add it after v2.2.20 is released (because it's affecting so many 
different platforms that there's too big of a chance of it breaking one of them 
due to some typo).



quota-fs-rules-recalc.diff
Description: Binary data

Re: Dovecot cluster using GlusterFS

[Timo Sirainen]

On 05 Dec 2015, at 11:42, Filip Pytloun  wrote:
> 
> Anyway because of the above and high possibility of GlusterFS
> split-brains, I have decided to setup Dovecot Director according to the
> docs [1] but I have a couple of questions:
> 
> - is custom monitoring still required? Poolmon [2] is 4 year old so I
>  would suppose there's some progress since that?

I think it's always going to be a separate script. In different environments 
people may want to do it slightly differently.

> - documentation states that it still doesn't work for LMTP [3]?
>  Which is probably important for my setup, because both Postfix servers
>  are using dovecot-lmtp for mail delivery so there can be still some
>  issues (but probably less frequent?) when both servers will deliver
>  new mails for one user at once.
>  So do I really have to split directors from backends?

You can run director and backend in the same servers, but they'd have to be in 
different config files (so two dovecot instances). Or you could also do it with 
a single instance:

> [3] "LMTP however doesn't currently support mixing recipients to both
> being proxied and store locally."

This is a problem only if there are multiple recipients in the same LMTP 
session. So if you configure your MTA to restrict the recipient limit to 1, 
this mixing can't happen and a single Dovecot instance can work.

v2.3 will solve this problem completely.

Re: v2.2.20 release candidate released

[Gerhard Wiesinger]

On 06.12.2015 13:10, Timo Sirainen wrote:

On 05 Dec 2015, at 11:32, Gerhard Wiesinger  wrote:

Is it possible to configure the secure session caching mechanism?
e.g. like in nginx: https://bjornjohansen.no/optimizing-https-nginx

I remember hearing about various security vulnerabilities in that earlier.. I 
guess they're fixed now then, unless people find more ways to exploit it. 
Anyway I'm not sure how useful it would actually even be for most IMAP/POP3 
servers, because most clients don't connect all that often. Or I guess it might 
help some clients that create multiple connections immediately.

Then again, we are planning on adding some HTTP(S)-based services to Dovecot 
and there it would likely be more useful. So I guess it gets implemented at 
some point.




Session tickets are broken by DESIGN as they violate PFS (Perfect 
Forward Secrecy). If you can steal one AES key (all session tickets are 
encrypted for server lifetime with only one key) you can decrypt ALL 
sessions ever made with session tickets for the future. This violates 
PFS (Perfect Forward Secrecy) as only server side "parameters" are 
relevant from now on.


Yes, session caching should reduce server load on multiple connections.

See e.g. https://community.qualys.com/thread/15768

Therefore it would be great if you could implement the secure session 
caching mechanism.


As Gedalya mentioned OCSP would be great, too.

Ciao,
Gerhard

Re: Dovecot 2.2.18.2: Panic: file fs-api.c: line 756 (fs_copy): assertion failed: (src->fs == dest->fs)

[Timo Sirainen]

On 27 Oct 2015, at 14:26, Frank Mehrtens  wrote:
> 
> Hello,
> 
> a colleague found some strange messages in the logs:
> 
> Oct 27 13:08:43 mail dovecot: lmtp(mailarc...@domain.de): Panic: file 
> fs-api.c: line 756 (fs_copy): assertion failed: (src->fs == dest->fs)

Same mail with attachments was probably attempted to be delivered to multiple 
users within the same LMTP session. 
http://hg.dovecot.org/dovecot-2.2/rev/183576574417 should fix this.

Re: ACL Troubles

[Timo Sirainen]

> On 05 Dec 2015, at 00:17, Bobber  wrote:
> 
> Using dovecot version 2.2.19.
> 
> In the config file I have the following to set up acls:
> 
> > mail_plugins = $mail_plugins acl
> > protocol imap {
> >   mail_plugins = $mail_plugins imap_acl
> > }
> 
> > plugin {
> >  acl = vfile:/usr/local/etc/dovecot/acls:cache_secs=300
> > }
> 
> And here's my acl file:
> > *  anyone rl
> 
> I am getting the following error in the dovecot log file:
> > Error: Global ACL file /usr/local/etc/dovecot/acls line 1: Unknown ID 
> > 'anyonerl'
> 
> Also, my subfolders in the shared mailbox area are no longer visible or 
> accessible.
> 
> Any ideas what I am doing wrong?

I think you have a TAB between "anyone"and "rl" instead of a space, which 
Dovecot expects.

Re: v2.2.20 release candidate released

[Timo Sirainen]

On 05 Dec 2015, at 11:32, Gerhard Wiesinger  wrote:
> 
> Is it possible to configure the secure session caching mechanism?
> e.g. like in nginx: https://bjornjohansen.no/optimizing-https-nginx

I remember hearing about various security vulnerabilities in that earlier.. I 
guess they're fixed now then, unless people find more ways to exploit it. 
Anyway I'm not sure how useful it would actually even be for most IMAP/POP3 
servers, because most clients don't connect all that often. Or I guess it might 
help some clients that create multiple connections immediately.

Then again, we are planning on adding some HTTP(S)-based services to Dovecot 
and there it would likely be more useful. So I guess it gets implemented at 
some point.

Re: SIS attachments compressed

[Timo Sirainen]

> On 03 Jun 2015, at 18:29, Juan Carlos Sanchez  
> wrote:
> 
> Hello:
> 
> In case it is defined to use zlib compression, does SIS save attachments 
> compressed?
> 
> I have found this question replied in the past (2012) and the answer was NO. 
> I ask again just to know if there have been changes or if it is in the 
> roadmap to add compression.
> 
> I have been testing to migrate from maildir+zlib to mdbox+zlib+sis and the 
> results, in terms of space used, and worst than using just mdbox+zlib 
> (without SIS), and suppose the reason in the lack of compression in 
> attachments.

Pretty late reply, but nowadays there's fs-compress plugin. So something like 
this should work (untested):

mail_attachment_fs = sis compress:gz:6:posix

It's a bit dangerous to enable this for existing attachments though, because if 
there are any gziped attachments Dovecot will try to gunzip them before sending 
to client, which causes problems. (So they should be double-gziped to avoid 
this problem.)

Re: mailbox_list_index and maildir_very_dirty_syncs are in conflicts?

[Timo Sirainen]

On 05 Dec 2015, at 20:40, Alessio Cecchi  wrote:
> 
> Hi,
> 
> I’m running Dovecot 2.2.19 with Maildir as storage and LDA for delivery.
> 
> I noticed that if I set mailbox_list_index=yes and 
> maildir_very_dirty_syncs=yes when I login via IMAP the STATUS command don’t 
> “see” new messages in sub-folders (like Spam).
..
> An importante note, my dovecot LDA configuration (on MX servers) don’t update 
> index files:
> 
> protocol lda {
>  mail_location = maildir:~/Maildir:INDEX=MEMORY
>  mail_plugins = quota acl expire fts fts_solr zlib sieve
> }

This works as intended.. Fixing it would slow down the normal way people use 
Dovecot.

> These because I need to filter incoming email via Sieve but since I cannot 
> use LMTP (and Director) on MX (but I have Director for POP/IMAP access) the 
> only way for not corrupting dovecot.index files is not update their on 
> delivery emails.

I don't understand why you couldn't use LMTP+director. That's how it's normally 
done.

> But reading http://wiki2.dovecot.org/MailLocation/Maildir this shouldn’t be a 
> problem (Optimizations “maildir_very_dirty_syncs=yes” … It's still safe to 
> deliver new mails to new/ …) since MX deliver new emails in new/.

I updated the wiki.

Re: Dovecot cluster using GlusterFS

[l...@airstreamcomm.net]

We ran a load test using glusterfs and were able to deliver mail (I can't 
remember specifically how much per second, maybe 100 messages per second?) 
without any issues.  We did use the glusterfs fuse client and not nfs, and used 
regular maildir.  We developed a mail bot cluster that would deliver mail, and 
simultaneously receive and delete it with pop and IMAP and we ran into zero 
issues.  We even had the replicas stretched between two datacenters.  Not sure 
what the difference here is but it can be done.



> On Dec 5, 2015, at 3:42 AM, Filip Pytloun  wrote:
> 
> Hello,
> 
> I have recently setup mailserver solution using 2-node master-master
> setup (mainly based on MySQL M-M replication and GlusterFS with 2
> replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
> 
> Unfortunately even with shared-storage-aware setting:
> 
> mail_nfs_index = yes
> mail_nfs_storage = yes
> mail_fsync = always
> mmap_disable = yes
> 
> ..I have hit strange issues pretty soon especially when user was
> manipulating same mailbox from multiple devices at the same time.
> 
> Most issues was about corrupted indexes which was solved easily by just
> putting them on local storage of each node:
> 
> mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
> 
> But I still hit issues like this one:
> 
> dovecot: lmtp(6276, u...@example.com): Error: Broken file 
> /srv/mail/example.com/u...@example.com/dovecot-uidlist line 8529: UIDs not 
> ordered (8527 >= 8527)
> 
> Which I am not sure how serious it is or if it's possible to solve or
> workaround?
> 
> Anyway because of the above and high possibility of GlusterFS
> split-brains, I have decided to setup Dovecot Director according to the
> docs [1] but I have a couple of questions:
> 
> - is custom monitoring still required? Poolmon [2] is 4 year old so I
>  would suppose there's some progress since that?
> 
> - it's not possible to have same backends and directors in Dovecot
>  <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is
>  not an issue, but..
> 
> - documentation states that it still doesn't work for LMTP [3]?
>  Which is probably important for my setup, because both Postfix servers
>  are using dovecot-lmtp for mail delivery so there can be still some
>  issues (but probably less frequent?) when both servers will deliver
>  new mails for one user at once.
>  So do I really have to split directors from backends?
> 
> 
> Anyone has experience with clustered Dovecot setup?
> Why is Dovecot behaving so bad when it pretends to be shared storage
> friendly? Are these issues only specific for older Dovecot?
> Or is there something wrong in my architecture design?
> 
> Thanks for any help,
> Filip
> 
> 
> ---
> [1] http://wiki2.dovecot.org/Director
> [2] https://github.com/brandond/poolmon/
> [3] "LMTP however doesn't currently support mixing recipients to both
> being proxied and store locally."
> 
> ---
> BTW if someone is interested in SaltStack, here are Salt formulas for
> Dovecot + Postfix + GlusterFS + Roundcube + Mailman setup which we are
> using:
> 
> https://github.com/tcpcloud/salt-formula-dovecot
> https://github.com/tcpcloud/salt-formula-postfix
> https://github.com/tcpcloud/salt-formula-roundcube
> https://github.com/tcpcloud/salt-formula-glusterfs

Re: Dovecot cluster using GlusterFS

[Gordon Grubert]

On 12/06/2015 10:16 AM, Filip Pytloun wrote:

At the moment, I cannot recognize the requirement for using lmtp over
the directors. When using postfix for delivering e-mails to the
backend, do this directly with an corresponding MX record.


I have two MX records of the same weight with postfix using dovecot-lmtp
for delivery. So that's why I wanted to use LMTP over directors.
Using lower weight for second MX is an option, but not truly
master-master setup :-)


We're using postfix for delivering to the dovecot backends via lmtp on
the base of an MX record with the same weight. Here, postfix can
directly access the dovecot backends without using the dovecot
directors. It a really master-master setup ;-)

Best regards,
Gordon

Re: Dovecot cluster using GlusterFS

[Michael Schwartzkopff]

Am Sonntag, 6. Dezember 2015, 20:33:32 schrieb l...@airstreamcomm.net:
> We ran a load test using glusterfs and were able to deliver mail (I can't
> remember specifically how much per second, maybe 100 messages per second?)
> without any issues.  We did use the glusterfs fuse client and not nfs, and
> used regular maildir.  We developed a mail bot cluster that would deliver
> mail, and simultaneously receive and delete it with pop and IMAP and we ran
> into zero issues.  We even had the replicas stretched between two
> datacenters.  Not sure what the difference here is but it can be done.
> > On Dec 5, 2015, at 3:42 AM, Filip Pytloun  wrote:
> > 
> > Hello,
> > 
> > I have recently setup mailserver solution using 2-node master-master
> > setup (mainly based on MySQL M-M replication and GlusterFS with 2
> > replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
> > 
> > Unfortunately even with shared-storage-aware setting:
> > 
> > mail_nfs_index = yes
> > mail_nfs_storage = yes
> > mail_fsync = always
> > mmap_disable = yes
> > 
> > ..I have hit strange issues pretty soon especially when user was
> > manipulating same mailbox from multiple devices at the same time.
> > 
> > Most issues was about corrupted indexes which was solved easily by just
> > putting them on local storage of each node:
> > 
> > mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
> > 
> > But I still hit issues like this one:
> > 
> > dovecot: lmtp(6276, u...@example.com): Error: Broken file
> > /srv/mail/example.com/u...@example.com/dovecot-uidlist line 8529: UIDs
> > not ordered (8527 >= 8527)
> > 
> > Which I am not sure how serious it is or if it's possible to solve or
> > workaround?

hi,

I did experiments with glusterfs and dovecot about one year ago and ran into 
the same trouble. It was quite easy to corrupt users mailboxes delivering mail 
somultaniously on different nodes.

I tried a lot of different configurations (see NFS recomendations, ...) but 
nothig really solved the problem.

Also from the list I did not get any useful comments.

So I decided that plain dovecot / glusterfs is not usable.

I would be glad to hear of any other experiance and configuration tweaks.

Mit freundlichen Grüßen,

Michael Schwartzkopff

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64, +49 (162) 165 0044
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

signature.asc
Description: This is a digitally signed message part.