Re: overview zlib efficiency? Summary and added note

[Haravikk]

Not sure how you’re seeing such a high ratio; I tried the same commands on my 
system (thanks for these btw) and my savings from compression are around 5% =D

That said I’m dealing with a much smaller volume (3gb) and I’ve only identified 
a half dozen or so attachments that don’t have some kind of compression 
already; most modern mail programs will compress common types like images by 
default, and many modern file-formats have compression built in, and can give 
better results than zlib anyway.

My biggest savings are on mailing list messages (I filter these into their own 
mailbox) since they tend to be longer than typical messages, especially with 
auto-quoting, they also tend to be very busy mailboxes, but I also don’t keep 
them forever.

As an experiment I also tried moving my (uncompressed) messages to a 
compressing file-system (ZFS using lz4) but the savings were similarly small; I 
assume they were probably a bit better, but the extra overhead of the 
file-system eroded it since the savings are so small in my case. I think if 
you’re serious about compression then a compressing file-system is the way to 
go though, but in my case I’m on virtual hosting so there’s not much point in 
layering a ZFS volume on top of shared storage (since it’s ZFS based already 
for integrity/redundancy).

I just thought I’d mention my experience since people are quoting big savings 
that I haven’t seen; I wouldn’t consider my usage all that unusual, maybe some 
of you are receiving a lot more newsletter type traffic (these messages can be 
quite large), uncompressed document type files, or are less selective in which 
messages are retained forever? Just a caution that people looking at 
compression may not see the same savings depending upon their actual content.

Spam is another bad category for compression I’ve found; at least in my case 
the messages are usually very short, and/or contain randomised junk to try to 
confound filters, though I’m pretty aggressive about clearing them (I discard 
messages outright above a certain threshold, and use a script to expunge spam 
messages so that I can expunge messages with higher spam ratings faster (so 
possible false positives stick around longer so they can be caught).

> On 16 Mar 2016, at 09:48, Harald Leithner  wrote:
> 
> Hi,
> 
> use "doveadm" to get all real message
> 
> doveadm -f table fetch -A "size.physical" ALL | awk 
> '{s+=$2}END{printf("%.2fMB\n", s/1024/1024);}'
> 
> 189247.67MB .. 185G
> 
> use "du" to get size on disc:
> 
> In my case
> with deduplication:
> 
> /srv/stroage/# du -s -h *
> 53G vmail
> 75G vmail_sis
> 
> without deduplication
> 
> /srv/stroage/# du -s -h -l *
> 53G vmail
> 209Gvmail_sis
> 
> j4i, SIS can't use the zlib plugin so the 75G in my case are not compressed 
> (I haven't a filesystem that I trust and has a compression feature). Anyway 
> it has a 3:1 ratio in my case.
> 
> Maybe I interpret the SIS wrong and SIS couldn't be counted with du -l (count 
> links).
> 
> But if someone doesn't have SIS this values should be point you into the 
> right direction.
> 
> bye
> 
> Harald
> 
> Am 16.03.2016 um 08:50 schrieb Götz Reinicke - IT Koordinator:
>> Am 15.03.16 um 16:01 schrieb Götz Reinicke - IT Koordinator:
>>> Hi,
>>> 
>>> may be someone has already done that: Do you have a script(?) tool which
>>> shows the efficiency of the mail compression if zlib is used?
>>> 
>>> Something that shows the uncompressed size vrs. the compressed.
>> 
>> Hi,
>> 
>> maybe my question was a bit misleading. But anyway thanks for your
>> feedback regarding your experiences and compression rates.
>> 
>> We already thought about the benefit of less IO and more CPU power,
>> which is no concern.
>> 
>> The mailboxes I checked also go with 40-60% compression rate.
>> 
>> But what I was looking for was a tool or way to see what volume would be
>> used if we where not using compression.
>> 
>> e.g. "du -hs --without-zlib"
>> 
>> Our management would like to see a graph one day which shows the volume
>> uncompressed and compressed ...
>> 
>> Adding zlib with mdbox or maildir - as we do it currently - is from my
>> POV if you have the CPU power a MUST :)
>> 
>>  happy dovecoting - Götz
>> 
>> 
>> 
> 
> -- 
> Harald Leithner
> 
> ITronic
> Wiedner Hauptstraße 120/5.1, 1050 Wien, Austria
> Tel: +43-1-545 0 604
> Mobil: +43-699-123 78 4 78
> Mail: leith...@itronic.at | itronic.at

[Feature Request] doveadm option to return number of messages acted upon

[Haravikk]

So I have a script for handling my specific archive and expunge needs, but it’d 
be nice to be able to track how many messages are being affected.

Currently I’m doing it by firing the same search queries into doveadm search 
and counting the lines of the result with wc -l, but that’s not a very pretty 
solution. While I’m mostly doing it out of interest on a personal server, I 
can’t imagine it’s a very scalable way to do it if you wanted to gather some 
kind of metrics for example.

What I think would make sense would be for relevant doveadm commands such as 
move, expunge (probably purge too) and others I haven’t thought of to have a 
new option that, if enabled, will cause the command to output the number of 
messages affected as the final line of output (most of these commands don’t 
have any output anyway). A doveadm count command could also be a convenient, 
more efficient, alternative to doveadm search | wc -l.

Re: autoexpunge - different settings for different users

[Haravikk]

> On 7 Feb 2016, at 02:50, Tom Johnson  wrote:
> 
> I have some users where I'd like to expunge their mail at different times.
> 
> For example, I may have some users who I want to expunge everything from 
> every folder at 30 days.  
> 
> Others where I'd like to expunge everything at 90 days.
> 
> And still others where I only want to expunge mail from the Trash and Junk 
> folders, and nowhere else.
> 
> Is there a way to do some sort of lookup for this?  Or is this beyond the 
> ability of the autoexpunge feature?  Should I just use a cron job and use 
> "doveadm expunge" on a user-by-user basis?
> 
> Thanks-

I think you’ll need a script for this. At the most basic level you could just 
create a bash script, add a bunch of expunge commands and then run it every day 
(or week or whatever) via crontab. For example:

doveadm expunge -u some...@yourdomain.tld 
 \( mailbox Trash or mailbox Junk \) savedBefore 
30d
doveadm expunge -u someone_e...@youdomain.tld 
 savedBefore 90d

And so-on. Also, don’t forget to add a purge command if you’re using the mdbox 
format, in my case I’d add this to the start of the script, so messages aren’t 
purged right away (i.e- expunged messages are not purged until the next time 
the script runs), but this only applies to that format.

More complex logic will require a more complex script, I’ve been working on 
something of my own but I don’t know if I’d call it ready yet, and I’m not 
aware of anything else pre-made that will do it, but hopefully others can weigh 
in if there is.

Re: autoexpunge - different settings for different users

[Haravikk]

> On 7 Feb 2016, at 10:17, Haravikk <dove...@haravikk.me> wrote:
> 
> 
>> On 7 Feb 2016, at 02:50, Tom Johnson <t...@terramar.net> wrote:
>> 
>> I have some users where I'd like to expunge their mail at different times.
>> 
>> For example, I may have some users who I want to expunge everything from 
>> every folder at 30 days.  
>> 
>> Others where I'd like to expunge everything at 90 days.
>> 
>> And still others where I only want to expunge mail from the Trash and Junk 
>> folders, and nowhere else.
>> 
>> Is there a way to do some sort of lookup for this?  Or is this beyond the 
>> ability of the autoexpunge feature?  Should I just use a cron job and use 
>> "doveadm expunge" on a user-by-user basis?
>> 
>> Thanks-
> 
> I think you’ll need a script for this. At the most basic level you could just 
> create a bash script, add a bunch of expunge commands and then run it every 
> day (or week or whatever) via crontab. For example:
> 
>   doveadm expunge -u some...@yourdomain.tld 
> <mailto:some...@yourdomain.tld> \( mailbox Trash or mailbox Junk \) 
> savedBefore 30d
>   doveadm expunge -u someone_e...@youdomain.tld 
> <mailto:someone_e...@youdomain.tld> savedBefore 90d
> 
> And so-on. Also, don’t forget to add a purge command if you’re using the 
> mdbox format, in my case I’d add this to the start of the script, so messages 
> aren’t purged right away (i.e- expunged messages are not purged until the 
> next time the script runs), but this only applies to that format.
> 
> More complex logic will require a more complex script, I’ve been working on 
> something of my own but I don’t know if I’d call it ready yet, and I’m not 
> aware of anything else pre-made that will do it, but hopefully others can 
> weigh in if there is.

Just a note, but please disregard the extra e-mail in the angle brackets, OS 
X’s Mail seems to have added these automatically, the sample commands should 
look like:

doveadm expunge -u some...@yourdomain.tld \( mailbox Trash or mailbox 
Junk \) savedBefore 30d
doveadm expunge -u someone_e...@youdomain.tld savedBefore 90d

Re: Disable Client Certificate Authentication for Unencrypted Connections?

[Haravikk]

So I still haven’t found a way to require client certificates only for port 
993/IMAPS while leaving unencrypted IMAP open for local, trusted, services.

Is there really no way to do this? I just found out how to do the same thing 
for postfix (turns out it’s fairly easy, just a matter of adding the settings 
in the right parts of master.cf instead of main.cf), allowing me to restrict 
client certificate verification to port 587 (I don’t accept port 465) without 
interfering with incoming mail server connections on port 25.

> On 29 Jan 2016, at 13:00, Haravikk <dove...@haravikk.me> wrote:
> 
>> 
>> On 27 Jan 2016, at 21:55, Axel Luttgens <axel.luttg...@skynet.be> wrote:
>> 
>> Hello Haravikk,
>> 
>> Perhaps could you try to devise an exception based on one (or more) "remote" 
>> section(s), as in:
>> 
>>  remote ip.of.webmail.server {
>>  ssl_verify_client_cert = no
>>  [other settings, if needed]
>>  }
>> 
>> But I guess you would need to combine this with inner protocol blocks, and 
>> probably to replace the "protocol !smtp" block with less general settings.
>> 
>> HTH,
>> Axel
> 
> Thanks for the suggestion!
> 
> Unfortunately the problem seems to be auth_ssl_require_client_cert; it can 
> only be added to protocol blocks not to local or remote ones. Turning off 
> ssl_verify_client_cert doesn’t seem to prevent dovecot from requiring a 
> certificate if auth_ssl_require_client_cert is enabled (it may even force 
> ssl_verify_client_cert to on implicitly, I’m not sure).
> 
> It’s annoying because at present it seems like my only option would be to 
> limit client certificates to POP3 and use that in my mail clients, allowing 
> me to disable client certificates for IMAP to keep it free for Roundcube to 
> use exclusively, but that’s not really an option.

Re: Disable Client Certificate Authentication for Unencrypted Connections?

[Haravikk]

That definitely seems promising! Though I’m a little uncomfortable about having 
to apply a patch to make it work.

However it’s got me curious; could I do something similar by defining a second 
IMAP protocol? Since the protocol !smtp option can enable client certificates 
for only some protocols, it seems like if I could copy (or symlink) the IMAP 
protocol into a second IMAPS protocol then I could possibly create the 
exception using protocol !imap !smtp, could something like that work? Of course 
I don’t know enough about how dovecot’s protocol modularity works, so I’m not 
sure what this would involve exactly, but if it could avoid the need for 
patching it might be a nice way to achieve the same goal.

> On 2 Feb 2016, at 12:44, Timo Sirainen <t...@iki.fi> wrote:
> 
> 
>> On 02 Feb 2016, at 14:19, Timo Sirainen <t...@iki.fi> wrote:
>> 
>> On 02 Feb 2016, at 13:09, Haravikk <dove...@haravikk.me> wrote:
>>> 
>>> So I still haven’t found a way to require client certificates only for port 
>>> 993/IMAPS while leaving unencrypted IMAP open for local, trusted, services.
>>> 
>>> Is there really no way to do this? I just found out how to do the same 
>>> thing for postfix (turns out it’s fairly easy, just a matter of adding the 
>>> settings in the right parts of master.cf instead of main.cf), allowing me 
>>> to restrict client certificate verification to port 587 (I don’t accept 
>>> port 465) without interfering with incoming mail server connections on port 
>>> 25.
>> 
>> If you really want to do it, you could do it something like this (not tested 
>> - probably needs some additions/changes):
> 
> Oh, except now you have two auth master processes, so the final userdb lookup 
> would have to go to the right auth process. This requires the attached patch. 
> After that you can do:
> 
> # no 993 port for this imap-login
> service imap-login {
> inet_listener imaps {
>   port = 0
> }
> }
> 
> # create a duplicate 993 service
> service imap-login-ssl {
> executable = imap-login -P imap-ssl login-ssl
> chroot = login
> user = $default_login_user
> # add other settings similarly as to service imap-login
> inet_listener imaps {
>   port = 993
> }
> }
> 
> service imap-ssl {
>  executable = imap -S auth-master-ssl
>  unix_listener login/imap-ssl {
>user = $default_login_user
>  }
> }
> 
> # create a separate auth master process for port 993
> service auth-ssl {
> executable = auth -o auth_ssl_require_client_cert=yes
> # add other settings similarly as to service auth
> unix_listener login/login-ssl {
>   mode = 0666
> }
> unix_listener auth-master-ssl {
> }
> }
> 
> 

Re: Disable Client Certificate Authentication for Unencrypted Connections?

[Haravikk]

> On 27 Jan 2016, at 21:55, Axel Luttgens <axel.luttg...@skynet.be> wrote:
> 
> Hello Haravikk,
> 
> Perhaps could you try to devise an exception based on one (or more) "remote" 
> section(s), as in:
> 
>   remote ip.of.webmail.server {
>   ssl_verify_client_cert = no
>   [other settings, if needed]
>   }
> 
> But I guess you would need to combine this with inner protocol blocks, and 
> probably to replace the "protocol !smtp" block with less general settings.
> 
> HTH,
> Axel

Thanks for the suggestion!

Unfortunately the problem seems to be auth_ssl_require_client_cert; it can only 
be added to protocol blocks not to local or remote ones. Turning off 
ssl_verify_client_cert doesn’t seem to prevent dovecot from requiring a 
certificate if auth_ssl_require_client_cert is enabled (it may even force 
ssl_verify_client_cert to on implicitly, I’m not sure).

It’s annoying because at present it seems like my only option would be to limit 
client certificates to POP3 and use that in my mail clients, allowing me to 
disable client certificates for IMAP to keep it free for Roundcube to use 
exclusively, but that’s not really an option.

Correct Way to Run expunge for All?

[Haravikk]

So as part of my daily mailbox maintenance I have the following command:

doveadm expunge -A DELETED

i.e- I’m only expunging messages that have been specifically marked as deleted. 
I also only do this *after* running doveadm purge, as this means that there is 
at least a 24 hour window between a message being marked as deleted, and it 
being gone forever.

However I get the following error message:

Fatal: expunge: To avoid accidents, search query must contain MAILBOX 
in all search branches

While I can see the logic in this, it isn’t an accidental use of this command, 
so who do I force it to proceed?

Going forward I’d like to recommend that expunge -A DELETED should be allowed, 
as it doesn’t seem like an especially dangerous form of the command, 
particularly with mdboxes where the messages won’t be removed until the purge 
step. Either that or there should be a flag to force the command when you know 
it isn’t an accident; anyone adding this to all their queries deserves the 
damage they inflict =)

Re: Setting Recent Flag on New Messages?

[Haravikk]

> On 29 Jan 2016, at 14:29, Timo Sirainen  wrote:
> 
> \Recent flag is special and you can't add/remove it. It's not even internally 
> stored like regular flags.

Are you certain? The following commands seem to work just fine:

doveadm flags add -A ‘\Recent’ OLD UNSEEN
doveadm search -A NEW

Before I ran the first command, the second wouldn’t return anything, so \Recent 
does seem to be set via doveadm flags, doveadm fetch flags also seems to 
confirm this.

> I think what you want instead is:
> 
> doveadm expunge -A old seen savedbefore 7d mailbox Trash

While you’re right that I should probably be using saved before, I’m trying to 
find an alternative to this specific command because it seems like it would be 
possible for a message that’s 7 days old to potentially to be opened by the 
user such that is flagged as seen, but is actually still being read when the 
command is run, allowing it to be expunged before the user has a chance to do 
anything with it. Or is the saved before value updated when flags are changed?

smime.p7s
Description: S/MIME cryptographic signature

Setting Recent Flag on New Messages?

[Haravikk]

So I’m looking at setting up some house-keeping for my dovecot mailboxes, 
specifically deleting messages in trash, spam etc., by deleting Seen messages 
older than X days.

In order to protect against the edge case of a user reading a message right 
before the deletion task however I’d like to do something like the following:

doveadm expunge -A OLD SEEN BEFORE 7d MAILBOX Trash
doveadm flags -A remove '\Recent' SEEN BEFORE 7d MAILBOX Trash
(except I’m actually doing this per-user with a script and per-user settings 
for flexibility, but you get the idea)

Basically the expunge step only deletes messages without the Recent flag (and 
which haven’t been marked as unseen again). It then removes this flag from 
messages which have been seen, so that they will be deleted next time the 
expunge command runs. In other words, if this command runs once a day, there 
will be a minimum delay of 24 hours between a message being seen, and then 
being deleted.

However I can’t for the life of me figure out how to set the Recent flag except 
via doveadm flags. While I could periodically run doveadm -A flags add 
‘\Recent’ SEEN or such this doesn’t seem like a great way to do it (adding the 
flag after the messages are stored), really I’d rather do it as they arrive. 
I’ve tried sieve, but it can’t set this flag.

Have I managed to completely miss an option for setting this when new messages 
arrive?

Re: Setting Recent Flag on New Messages?

[Haravikk]

> On 28 Jan 2016, at 16:55, Robert Schetterer <r...@sys4.de> wrote:
> 
> Am 28.01.2016 um 16:51 schrieb Haravikk:
>> I’ve tried sieve, but it can’t set this flag.
> 
> imapflags muss be added in 90-sieve.conf

Oh I can set other flags, but \Recent seems to be specifically disallowed in 
the specification that you linked to:

"Note that it is not possible to use this extension to set or clear the 
\Recent flag or any other special system flag that is not settable in [IMAP].”

Or does the deprecated imapflags allow it where imap4flags does not? It doesn’t 
seem like I should really have to rely on a deprecated sieve extension though.

[Bug/Enhancement] doveadm user ignores -f option when performing a wildcard search

[Haravikk]

The title mostly says it all on this one, but it seems that doveadm user has 
unique behaviour when a wildcard username is given, causing it to return a list 
of matching usernames in place of regular output, ignoring other command line 
options such as the -f (field) option.

For example, I had hoped to use the following command in a script to easily get 
a list of all user home locations (excluding local users):

doveadm user -f home *@*

However the results are a list of usernames, rather than a list of home 
locations as I had hoped.

I’ve tagged this thread as Bug/Enhancement as outputting a list of matched 
usernames is clearly the intended behaviour, but it could certainly be 
clarified, as could the fact that the -f option is ignored in this special case.

I think that the correct “fix” here is to clarify in the documentation that 
wildcard matching is a separate use-case of this command in which the -f (and 
possibly other?) options are ignored and the results are always given as a list 
of matching usernames. It needs to be clear that doveadm user u...@domain1.tld 
u...@domain2.tld and doveadm user *@* are not equivalent, even if those are the 
only two users, as this differs from many other unix commands where wildcards 
are permitted, e.g- using rm with a wildcard is equivalent to adding all 
matching files to the command yourself.

However, in future I’d like to see support for the -f option with wildcard 
matches as well, with -f user simply being the default. This would be much 
easier for a script inspecting user home directories, as for now my alternative 
is to use the wildcard match to get a list of names, then call dovecot user -f 
home  for each one to get their home location, which isn’t nearly as nice 
=)

- Haravikk

Disable Client Certificate Authentication for Unencrypted Connections?

[Haravikk]

I’m using dovecot to provide encrypted IMAP e-mail support for remote clients 
and it’s working great. However, I also need to set up a webmail front-end 
(Roundcube), which I’m hoping to have use unencrypted IMAP on port 143 (as only 
port 993 is available externally).

The problem I’m running into is that I want to require client certificate 
authentication on port 993, but dovecot is apparently requiring a certificate 
on all connections, which is preventing Roundcube from connecting.

Since dovecot is also providing authentication to postfix I’ve already created 
an exemption from the client certificate requirement for SMTP connections by 
doing the following:

protocol !smtp {
ssl_ca =