Re: lazy-load SNI?

[Aki Tuomi]

On 11.11.2016 01:02, Felipe Gasper wrote:

Hello,

We’re rolling out large SNI deployments for our mail servers. Each 
domain gets an entry like this in the config:

local_name mail.foo.com {
 ssl_cert = 

Unfortunately it's not possible now, it has been asked before though. We 
have this feature request in our list but cannot give any date when it 
would be available.


Aki Tuomi

Dovecot oy

Re: post-delivery virus scan

[Dmitry Melekhov]

10.11.2016 16:47, Frank Elsner пишет:

On Wed, 9 Nov 2016 15:36:33 -0600 Brad Koehn wrote:

   [ ... ]


To help detect and remove the infected messages after they’ve been delivered to 
users’ mailboxes, I created a small script that iterates the INBOX and Junk 
mailbox directories, scans recent messages for viruses, and deletes them if 
found. The source of my script (run via cron) is here: 
https://gitlab.koehn.com/snippets/9

Bad idea. The user may already taken the action needed for infection. And what 
about legal aspects?

Is it legal to redistribute malware in Germany? :-D

In my country (Germany), information suppression would be punishable.


I guess main problem here is that this is not on access scan, i.e. even 
if virus can be already detected by newer virus database, it can be 
accessed by user before it rescanned.

Re: How does one mark all messages as read (imap4flag "seen") with sieve?

[Ben Johnson]

On 11/10/2016 3:46 AM, Bill Shirley wrote:
> I don't use the Anti-Spam plugin; I just fire off a BASH script every
> four hours with
> crontab which iterates thru the vmail email accounts and trains
> Spamassassin 'per-user'
> accounts.  If the script sounds interesting I can post it here.  It
> probably could use a little polish
> though.
> 
> Bill

Thanks, Bill!

Sure, please do share the script, if it's not too much trouble.

For my specific use-case, I've been maintaining a "corpus" of known
ham/spam messages, and enjoy being able to hand
classify/re-classify/ignore if necessary.

But I do see the appeal of training with a single script that iterates
through each user's mailbox.

Heretofore, my thinking has been that combining all "submitted" spam,
which is piped into the training mailbox automatically, whenever a user
drags from Inbox -> Spam (or vice versa), I have a much broader sample
of the the ham and spam out there.

And yes, a "shared" corpus among all users does seem to "dilute"
specific individuals' would-be training preferences a bit, but the
trade-off seems worthwhile.

Interesting quandary... I would love to see the script! No problem if
it's a bit "rough around the edges"; the overall concept and approach
are what's important to me.

-Ben

Re: How does one mark all messages as read (imap4flag "seen") with sieve?

[Ben Johnson]

On 11/10/2016 2:02 AM, Aki Tuomi wrote:
> Hi!
> 
> Can you provide bt full from gbd?
> 
> Install debug symbols and acquire core file
> 
> Run gdb /path/to/bin /path/to/core
> 
> Issue bt ful
> 
> Send it to list.
> 
> 
> Aki Tuomi
> 
> Dovecot oy 
> 
> 
> 
> On November 10, 2016 at 2:42 AM Larry Rosenman 
> wrote:
> 
> looks to me from the coredump (although you pointed to the wrong binary)
> that
> deliver was PANIC()'ing with
> 
> io_add(0x%x) called twice fd=%d, callback=%p -> %p
> 
> I'm not sure what that message means, but maybe one of the dovecot folks
> does.
> 
> Are all the packages built together?
> 
> Are you averse to compiling stuff yourself?



Thanks so much for the assistance here, Larry and Aki.

Aki, you had asked me to send this core-dump to the list when I asked
about it back in September; I did post the "bt full" output at that
time, but there were no further replies:

http://www.dovecot.org/list/dovecot/2016-September/105428.html

And for convenience, here is the "bt full" output I posted back then:

http://pastebin.com/4xdGNXa6

So, this is where I'm confused: Larry, you mentioned "although you
pointed to the wrong binary", which is a concern I had asked about back
in September:

http://www.dovecot.org/list/dovecot/2016-September/105424.html

I still don't understand how this is pointing to the wrong binary:

# gdb /usr/lib/dovecot/dovecot-lda
/var/vmail/tmp/core-deliver-6-5000-5000-29125-1473732949

Is /usr/lib/dovecot/dovecot-lda not the binary that segfaulted here?
Sure, the gdb output says, "Core was generated by
`/usr/lib/dovecot/deliver ...", but on the system in question, that is a
symlink pointing to /usr/lib/dovecot/dovecot-lda:

# ls -lah /usr/lib/dovecot/deliver
lrwxrwxrwx 1 root root 11 Sep 21 10:29 /usr/lib/dovecot/deliver ->
dovecot-lda

What am I missing here? The mismatch message does say something specific
about this:

warning: the debug information found in "/lib64/ld-2.23.so" does not
match "/lib64/ld-linux-x86-64.so.2" (CRC mismatch).

Is this the result of pointing to the wrong executable when calling
"gdb"? If so, where is the correct executable to pass as the fist argument?

Again, for convenience, the pipe script I'm using:

http://pastebin.com/zXzBDcvG

And the debug output from said pipe script:

http://pastebin.com/rz2f4S4G

My full "doveconf -n":

http://pastebin.com/hCgpA009

Thanks!

-Ben

lazy-load SNI?

[Felipe Gasper]

Hello,

We’re rolling out large SNI deployments for our mail servers. Each 
domain gets an entry like this in the config:

local_name mail.foo.com {
ssl_cert = 

Re: service doveadm : ssl problems

[nerbrume]

- Mail original -
> De: "Tobi" 
> À: dovecot@dovecot.org
> Envoyé: Jeudi 10 Novembre 2016 16:35:56
> Objet: Re: service doveadm : ssl problems
> 
> Have you specified the path to ca-certificates?
> On Debian it's normally something like that
> 
> #10-ssl.conf
> ssl_client_ca_dir = /etc/ssl/certs

Yup, I did exactly that, sorry I forgot to include that part in the excerpt 
from my ssl config.
However, as far as I understood, this is of no impact when I test with openssl, 
right ?
(for the record, I also tried to manually add the intermediate ca (Let’s 
Encrypt Authority X3) in the /etc/ssl/certs dir, without any luck)

N.

> see http://wiki.dovecot.org/Replication#SSL
> 
> 
> Am 10.11.2016 um 16:09 schrieb nerbr...@free.fr:
> > Hello,
> > 
> > I'm using dovecot 2.2.13 on Debian stable.
> > My users are authenticated through PAM, and stored in an LDAP
> > backend
> > I'm trying to set-up replication with ssl, following (mainly) this
> > : http://wiki2.dovecot.org/Replication
> > 
> > 1) I only diverted from the instructed setup by not setting
> > "doveadm_port = 12345", as it would give me errors of the like:
> >> Fatal: /var/run/dovecot/auth-userdb: Configured passdbs don't
> >> support crentials lookups (to see if user is proxied, because
> >> doveadm_port is set)
> > but rather specifying the port in the mail_replica setting :
> > "mail_replica = tcps:my.domain.com:1465"
> > (following a mail from here :
> > http://www.dovecot.org/list/dovecot/2016-September/105356.html)
> > So far, this seems to be working for me.
> > 
> > 2) However, I'm having ssl problems. I have a let's encrypt
> > certificate, and have concatened the CA cert and my server cert in
> > a fullchain.pem.
> > Excerpt from my ssl config :
> >> ssl = yes
> >> ssl_cert =  >> ssl_key =  > 
> > doveadm return me these errors (sudo -u dovecot doveadm -v sync -u
> > user tcps:my.domain.com:12345) :
> >> doveadm(casoli): Info: Received invalid SSL certificate: unable to
> >> get local issuer certificate: /CN=my.domain.com
> >> doveadm(casoli): Error: doveadm server disconnected before
> >> handshake: Received invalid SSL certificate: unable to get local
> >> issuer certificate: /CN=my.domain.com
> >> doveadm(casoli): Fatal: Disconnected from remote: Received invalid
> >> SSL certificate: unable to get local issuer certificate:
> >> /CN=my.domain.com
> > 
> > Which I can reproduce with openssl (openssl s_client -showcerts
> > -CApath /etc/ssl/certs -connect my.domain.com:12345) :
> >> (...)
> >> Verify return code: 21 (unable to verify the first certificate)
> > Indeed, in this case, dovecot only returns the local part of the
> > certificate (my.domain.com), and not the full chain (with the
> > intermediate CA).
> > 
> > While testing regular IMAPS with openssl is ok (openssl s_client
> > -showcerts -CApath /etc/ssl/certs -connect my.domain.com:993)
> >> (...)
> >> Verify return code: 0 (ok)
> > And I can see the full chain.
> > 
> > 
> > So, it's seems to me that doveadm is somehow wrongly serving my
> > certificate, truncating it, but I can't see why, and if this is a
> > misconfiguratin on my part.
> > I can post more config files or message outputs if needed, I kept
> > them redacted here for the sake of brevity.
> > 
> > Regards,
> > N
> > 
> 

Enterprise Edition: Any known access issues with the repo? Have existing accounts been expired?

[deoren]

Hi,

I'm getting errors when attempting to run apt-get update on an Ubuntu 
14.04 box where I've had an existing EE installation for some time:


> W: Failed to fetch 
https://apt.dovecot.fi/stable-2.2/ubuntu/trusty/dists/trusty/main/binary-amd64/Packages 
 HttpError401

>
> W: Failed to fetch 
https://apt.dovecot.fi/stable-2.2/ubuntu/trusty/dists/trusty/main/binary-i386/Packages 
 HttpError401

>
> E: Some index files failed to download. They have been ignored, or 
old ones used instead.


I'm running 2.2.25.5 now and when I looked at the announcement forum[1] 
I see a posting[2] for 2.2.26.1, but nothing about repo changes.


For what it is worth, I use the EE credentials on only a single node so 
I am hopefully not triggering any abuse thresholds.


I tried accessing the URL used in the 
/etc/apt/sources.list.d/FILENAME.list file via a web browser and it 
isn't accepting the provided username/password.


Have existing credentials been expired? If so, what is the next step to 
restore access?


Thanks.


References:

[1] https://forum.open-xchange.com/forumdisplay.php?35-Dovecot-Announcements

[2] 
http://software.open-xchange.com/products/dovecot/doc/Release_Notes_for_Dovecot_Pro_2.2.26.1_2016-10-31.pdf

Re: post-delivery virus scan

[Stephan Bosch]

Op 10-11-2016 om 12:25 schreef Brad Koehn:

On Nov 10, 2016, at 3:38 AM, Stephan Bosch  wrote:

Op 11/10/2016 om 10:05 AM schreef Teemu Huovila:

On 09.11.2016 23:36, Brad Koehn wrote:

I’m wondering if there’s a better way to scan recent messages and eradicate 
them so the Dovecot isn’t upset when it happens. Maybe using doveadm search? 
Looking for suggestions.
The removal should if possible be done with the doveadm cli tool or using the 
doveadm http api.

Still, Dovecot should handle external removal of messages gracefully.
What exactly happens?

On Dovecot 2.2.6.0:
Nov 10 10:35:13 ds dovecot: imap(user): Error: Recent flags state corrupted for 
mailbox Junk
Nov 10 10:35:13 ds dovecot: imap(user): Error: 
/var/mail/user_dbox/mailboxes/Junk/dbox-Mails/dovecot.index reset, view is now 
inconsistent
Nov 10 10:35:13 ds dovecot: imap(user): IMAP session state is inconsistent, 
please relogin. in=6212 out=49396


OK, so at least it doesn't panic anymore in the last release. Also, the 
mailbox is fixed upon relogin. To prevent the remaining errors from 
occurring, i.e. to gracefully remove messages, you can use the doveadm 
expunge command (it has a man page).


Regards,

Stephan.

Re: service doveadm : ssl problems

[Tobi]

Have you specified the path to ca-certificates?
On Debian it's normally something like that

#10-ssl.conf
ssl_client_ca_dir = /etc/ssl/certs

see http://wiki.dovecot.org/Replication#SSL


Am 10.11.2016 um 16:09 schrieb nerbr...@free.fr:
> Hello,
> 
> I'm using dovecot 2.2.13 on Debian stable.
> My users are authenticated through PAM, and stored in an LDAP backend
> I'm trying to set-up replication with ssl, following (mainly) this : 
> http://wiki2.dovecot.org/Replication
> 
> 1) I only diverted from the instructed setup by not setting "doveadm_port = 
> 12345", as it would give me errors of the like:
>> Fatal: /var/run/dovecot/auth-userdb: Configured passdbs don't support 
>> crentials lookups (to see if user is proxied, because doveadm_port is set)
> but rather specifying the port in the mail_replica setting : "mail_replica = 
> tcps:my.domain.com:1465"
> (following a mail from here : 
> http://www.dovecot.org/list/dovecot/2016-September/105356.html)
> So far, this seems to be working for me.
> 
> 2) However, I'm having ssl problems. I have a let's encrypt certificate, and 
> have concatened the CA cert and my server cert in a fullchain.pem.
> Excerpt from my ssl config :
>> ssl = yes
>> ssl_cert = > ssl_key =  
> doveadm return me these errors (sudo -u dovecot doveadm -v sync -u user 
> tcps:my.domain.com:12345) :
>> doveadm(casoli): Info: Received invalid SSL certificate: unable to get local 
>> issuer certificate: /CN=my.domain.com
>> doveadm(casoli): Error: doveadm server disconnected before handshake: 
>> Received invalid SSL certificate: unable to get local issuer certificate: 
>> /CN=my.domain.com
>> doveadm(casoli): Fatal: Disconnected from remote: Received invalid SSL 
>> certificate: unable to get local issuer certificate: /CN=my.domain.com
> 
> Which I can reproduce with openssl (openssl s_client -showcerts -CApath 
> /etc/ssl/certs -connect my.domain.com:12345) :
>> (...)
>> Verify return code: 21 (unable to verify the first certificate)
> Indeed, in this case, dovecot only returns the local part of the certificate 
> (my.domain.com), and not the full chain (with the intermediate CA).
> 
> While testing regular IMAPS with openssl is ok (openssl s_client -showcerts 
> -CApath /etc/ssl/certs -connect my.domain.com:993)
>> (...)
>> Verify return code: 0 (ok)
> And I can see the full chain.
> 
> 
> So, it's seems to me that doveadm is somehow wrongly serving my certificate, 
> truncating it, but I can't see why, and if this is a misconfiguratin on my 
> part.
> I can post more config files or message outputs if needed, I kept them 
> redacted here for the sake of brevity.
> 
> Regards,
> N
> 

exim problem with Redirect the emails from domain2 to domain1

[Steffen Kaiser]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, 7 Nov 2016, Quaquaraquà wrote:



I have a VPS using these two applications. I am transitioning from a 
domain_old to a domain_new. I'd like to redirect all the emails from 
domain_old to the local mailboxes of users @ domain_new.  In exim I've 
assumed that it is enough to add domain_old to the list of local domains:


I cannot help you with exim


domainlist local_domains = @ : domain_new : domain_old
...
begin routers
...
local_users:
 debug_print = "R: local_user for $local_part@$domain"
 driver = accept
 domains = +local_domains
 transport = dovecot_lmtp
 cannot_route_message = Unknown user


However in Dovecot I'm checking both the username and the domain to perform 
the authentication:

auth_username_format = %Lu
passdb { driver = sql ; }
password_query = SELECT username, domain, password FROM users WHERE username 
= '%n' AND domain = '%d'


To have this system to work, I wish some special rule that rewrites the 
domain from domain_old to domain_new. But I'm not sure whether this needs to 
be done in exim or dovecot and how to add it?


If exim would map domain_old to domain and your users will use domain 
always, that would be the easiest way. More consitent, IMHO.


Otherwise (if you want to support users to login with old domains, for 
instance):


1) add another column with domain_old and use AND (domain = '%d' OR 
domain_old = '%d)


domain_old would contain the old domain, domain the new one.

2) if you think you get more domains per user over the time, add another 
table and use JOIN or sub-SELECT.


- -- 
Steffen Kaiser

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEVAwUBWCSTH3z1H7kL/d9rAQJNFgf/fjfpv/v9tyU8E3QXzGeEtyb1V84t3yut
2ML1oS07soZPhs+Kbh15HqDi3a+0geLElpnMbvXV7dHqYE2az11QgFKf2krV9dVO
Y/SN22Pjwn9S6T0HGpGqk+aY62FG8uN6deXZeimKnmMRXQ0b1iswtSVb1KuTml9s
jMx7OPPexxiS6keKJrFU0LsSPQjqSDc7OmtuMEbWWpJL6ANYb3pYCJl9BugUHDp/
fenmJ2Ft8e8FjSpP/kXkYlgEVhs/Xw8rSz5I2XoQ+T68IICqCe+RwrazNJ8X6N3W
1bEsX9OYG4VTNasjJCLGaJ7i1ktfI5Bu1kvv1U4oeAlrd7acNpP7oA==
=7dPg
-END PGP SIGNATURE-

service doveadm : ssl problems

[nerbrume]

Hello,

I'm using dovecot 2.2.13 on Debian stable.
My users are authenticated through PAM, and stored in an LDAP backend
I'm trying to set-up replication with ssl, following (mainly) this : 
http://wiki2.dovecot.org/Replication

1) I only diverted from the instructed setup by not setting "doveadm_port = 
12345", as it would give me errors of the like:
> Fatal: /var/run/dovecot/auth-userdb: Configured passdbs don't support 
> crentials lookups (to see if user is proxied, because doveadm_port is set)
but rather specifying the port in the mail_replica setting : "mail_replica = 
tcps:my.domain.com:1465"
(following a mail from here : 
http://www.dovecot.org/list/dovecot/2016-September/105356.html)
So far, this seems to be working for me.

2) However, I'm having ssl problems. I have a let's encrypt certificate, and 
have concatened the CA cert and my server cert in a fullchain.pem.
Excerpt from my ssl config :
> ssl = yes
> ssl_cert =  ssl_key =  doveadm(casoli): Info: Received invalid SSL certificate: unable to get local 
> issuer certificate: /CN=my.domain.com
> doveadm(casoli): Error: doveadm server disconnected before handshake: 
> Received invalid SSL certificate: unable to get local issuer certificate: 
> /CN=my.domain.com
> doveadm(casoli): Fatal: Disconnected from remote: Received invalid SSL 
> certificate: unable to get local issuer certificate: /CN=my.domain.com

Which I can reproduce with openssl (openssl s_client -showcerts -CApath 
/etc/ssl/certs -connect my.domain.com:12345) :
> (...)
> Verify return code: 21 (unable to verify the first certificate)
Indeed, in this case, dovecot only returns the local part of the certificate 
(my.domain.com), and not the full chain (with the intermediate CA).

While testing regular IMAPS with openssl is ok (openssl s_client -showcerts 
-CApath /etc/ssl/certs -connect my.domain.com:993)
> (...)
> Verify return code: 0 (ok)
And I can see the full chain.


So, it's seems to me that doveadm is somehow wrongly serving my certificate, 
truncating it, but I can't see why, and if this is a misconfiguratin on my part.
I can post more config files or message outputs if needed, I kept them redacted 
here for the sake of brevity.

Regards,
N

Dovecot & AD (was: Dovecot 2 LDAP "unknown user")

[Steffen Kaiser]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Sat, 5 Nov 2016, Peter Fraser wrote:


1. I need to make sure the user logon name in AD and the samAccountname
   are exactly the same, case and all. It seems postfix
   uses the samAccountname and Dovecot the User logon name.
2. I also noticed that if the Display name for a user in AD is blank,
   that user cannot log in using telnet  110.


OK, this is something interesting


and dovecot-ldap.conf.ext reads as follows

#Custom Settings
hosts = ip address
ldap_version = 3
scope = subtree
deref = never
base = cn=users,dc=domain,dc=com
dn = cn=administrator,cn=users,dc=domain,dc=com
dnpass = password
auth_bind = yes
auth_bind_userdn = %n
ldap_version = 3
scope = subtree
user_attrs = home=/home/vmail/%u,=uid=vmail,=gid=vmail
pass_attrs = uid=%n,userPassword=password
#pass_attrs=uid=user, userpassword=password
user_filter = (&(objectclass=person)(samaccountname=%n))
pass_filter = (&(objectclass=inetorgperson)(mail=%u))


Check out your 1. and pass_filter
"postfix uses samAccountname" <-> pass_filter uses "mail" to identify an 
user.


So I suggest you use:

pass_filter = 
(&(objectclass=inetorgperson)(|(mail=%u)(samaccountname=%n)(cn=%n)))


Note: add all LDAP attributes to the LDAP query, that identify exactly one 
user (never more than one). If cn is not unique, use something you think 
is appropriate. Some examples in the net use userPrincipalName, ...
Also, search the net for "dovecot active directory" and you'll find 
that some exclude entries with certain userAccountControl strings.
That way Dovecot finds the user regardless of what s/he enters as 
username. You could even use something like (mail=%n...@example.com)


pass_attrs=samaccountname=user, userpassword=password

This will return samaccountname as new username for userdb queries.

user_filter = (&(objectclass=person)(samaccountname=%n))

Finally, this query must find the user's data. Because pass_attrs mangle 
the "user" information of Dovecot to be samaccountname, this attribute 
must be present. If postfix delivers to this user, too, you are done. 
Otherwise use a similiar approach as with pass_filter.

Dovecot LDA and LMTP do not use pass_filter, but only user_filter.

- -- 
Steffen Kaiser

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEVAwUBWCSKXnz1H7kL/d9rAQI2Wgf+OIFn5vssn1giLEocVSpZDvirLHLe4c1m
br+PBzklJ2OtM4gYjVdcSkgOmuDGOoeIOcxZQIZwmz7413oCjmA8jloUzzYhj6Q6
6CSLHlBWMqtsnQC8+bITuEWBO+ygXT4A5HdEiJANT/oq+Jq1PXq6gN4W3CVwaq+4
f0b+H+Ejk9Xf8jjnpsvhL4SeS71fc7QwmcDZ3syxutQhWgu/urkAUqu3B0R9PD2r
FOxJS+q4lF4JTni4vlWrqtuUeK9Mv675vLq1Uw8c+jLdlBgD5QKKsFNy3LTokMEm
qU1g7uSISl16AmZ6arIk2ZEtBMpYlFYhxct/EMbjfgeKZ75zG9g6Fw==
=S+La
-END PGP SIGNATURE-

Re: search body with wildcards

[W. de Hoog]

When searching:

  doveadm search mailbox shared/* BODY calib*

solr is queried with

  ...body:calib%5c*...

so the wildcard is escaped. Is there any way to disable this escaping?
I removed '*' and '?' from solr_escape_chars in 
src/plugins/fts-solr/fts-backend-solr.c to allow to use them as 
wildcards. I also removed '\' to allow them to be escaped so now these work:


doveadm search mailbox shared/* BODY calibrat*
to find words starting with calibrat

and

doveadm search mailbox shared/* BODY calibrati\\*n
to find "calibrati*n"

In my php search form I can now use "body calibrat*" and in thunderbird 
search dialog as well. Searching for "calibrati*n" (using 
"calibrati\*n") also works for php but not in Thunderbird.


One note: searching with "\*" is rather slow.

I cannot estimate the implications it has on the other search actions 
(TO, FROM etc) so I hope someone with more knowledge can comment.



regards,

--
Willem-Jan de Hoog

Re: post-delivery virus scan

[Brad Koehn]

Turns out the technical part of your reasoning is correct: MUAs that have 
downloaded the message don’t get any updates, and hold onto the infected 
message. No legal ramifications here; it’s my personal server, and it’s in the 
US. Strange to think that deleting the content of a message would somehow be 
worse than deleting the content and the headers.  


> On Nov 10, 2016, at 6:47 AM, Frank Elsner  wrote:
> 
> On Wed, 9 Nov 2016 15:36:33 -0600 Brad Koehn wrote:
> 
>  [ ... ]
> 
>> To help detect and remove the infected messages after they’ve been delivered 
>> to users’ mailboxes, I created a small script that iterates the INBOX and 
>> Junk mailbox directories, scans recent messages for viruses, and deletes 
>> them if found. The source of my script (run via cron) is here: 
>> https://gitlab.koehn.com/snippets/9
> 
> Bad idea. The user may already taken the action needed for infection. And 
> what about legal aspects?
> In my country (Germany), information suppression would be punishable.
> 
> 
> Just my 0.02 €, Frank

Re: post-delivery virus scan

[Frank Elsner]

On Wed, 9 Nov 2016 15:36:33 -0600 Brad Koehn wrote:

  [ ... ]

> To help detect and remove the infected messages after they’ve been delivered 
> to users’ mailboxes, I created a small script that iterates the INBOX and 
> Junk mailbox directories, scans recent messages for viruses, and deletes them 
> if found. The source of my script (run via cron) is here: 
> https://gitlab.koehn.com/snippets/9

Bad idea. The user may already taken the action needed for infection. And what 
about legal aspects?
In my country (Germany), information suppression would be punishable.


Just my 0.02 €, Frank

Re: post-delivery virus scan

[Brad Koehn]

I’ve decided to try this approach. I’ve updated my script as follows:

#!/bin/bash

# Scan junk folders for messages containing  viruses we didn't have definitions 
# for when the mail was received. Truncate the body of infected messages and
# replace the body with a message.

cd /var/mail

for dir in $( find . \( -name Junk -o -name INBOX \) -type d ) ; do
  files=$( find "$dir" -type f -name u.\* -mtime -14 -print )
  for file in $files ; do
/usr/local/bin/clamdscan --quiet --fdpass "$file"
if [ $? -eq 1 ] ; then
  sed -i '/^$/,$d' "$file"
  echo "\r\n\r\n[The body of this message contained a virus and was 
deleted.]" >> "$file"
fi
  done
done

We’ll see if that does the trick.

> On Nov 9, 2016, at 6:12 PM, mick crane  wrote:
> 
> On 2016-11-09 21:36, Brad Koehn wrote:
>> I have discovered that many times the virus definitions I use for
>> scanning messages (ClamAV, with the unofficial signatures
>> http://sanesecurity.com/usage/linux-scripts/) are updated some time
>> after my server has received an infected email. It seems the virus
>> creators are trying to race the virus definition creators to see who
>> can deliver first; more than half of the infected messages are found
>> after they’ve been delivered. Great.
>> To help detect and remove the infected messages after they’ve been
>> delivered to users’ mailboxes, I created a small script that iterates
>> the INBOX and Junk mailbox directories, scans recent messages for
>> viruses, and deletes them if found. The source of my script (run via
>> cron) is here: https://gitlab.koehn.com/snippets/9
>> Unfortunately Dovecot doesn’t like it if messages are deleted (dbox)
>> out from under it. I tried a doveadm force-resync on the folder
>> containing the messages, but it seems Dovecot is still unhappy. At
>> least on the new version (2.2.26.0) it doesn’t crash; 2.2.25 would
>> panic and coredump when it discovered messages had been deleted.
>> I’m wondering if there’s a better way to scan recent messages and
>> eradicate them so the Dovecot isn’t upset when it happens. Maybe using
>> doveadm search? Looking for suggestions.
> 
> leave an empty message behind with the same name as deleted message ?
> 
> 
> 
> 
> -- 
> key ID: 0x4BFEBB31

Re: post-delivery virus scan

[Brad Koehn]

> On Nov 10, 2016, at 3:38 AM, Stephan Bosch  wrote:
> 
> Op 11/10/2016 om 10:05 AM schreef Teemu Huovila:
>> 
>> On 09.11.2016 23:36, Brad Koehn wrote:
>>> I have discovered that many times the virus definitions I use for scanning 
>>> messages (ClamAV, with the unofficial signatures 
>>> http://sanesecurity.com/usage/linux-scripts/) are updated some time after 
>>> my server has received an infected email. It seems the virus creators are 
>>> trying to race the virus definition creators to see who can deliver first; 
>>> more than half of the infected messages are found after they’ve been 
>>> delivered. Great. 
>>> 
>>> To help detect and remove the infected messages after they’ve been 
>>> delivered to users’ mailboxes, I created a small script that iterates the 
>>> INBOX and Junk mailbox directories, scans recent messages for viruses, and 
>>> deletes them if found. The source of my script (run via cron) is here: 
>>> https://gitlab.koehn.com/snippets/9
>>> 
>>> Unfortunately Dovecot doesn’t like it if messages are deleted (dbox) out 
>>> from under it. I tried a doveadm force-resync on the folder containing the 
>>> messages, but it seems Dovecot is still unhappy. At least on the new 
>>> version (2.2.26.0) it doesn’t crash; 2.2.25 would panic and coredump when 
>>> it discovered messages had been deleted. 
>>> 
>>> I’m wondering if there’s a better way to scan recent messages and eradicate 
>>> them so the Dovecot isn’t upset when it happens. Maybe using doveadm 
>>> search? Looking for suggestions. 
>> The removal should if possible be done with the doveadm cli tool or using 
>> the doveadm http api.
> 
> Still, Dovecot should handle external removal of messages gracefully.
> What exactly happens?
> 
> Regards,
> 
> Stephan.

On Dovecot 2.2.5:
Nov  9 14:32:11 ds postfix/anvil[13298]: statistics: max cache size 2 at Nov  9 
14:23:08
Nov  9 14:32:29 ds dovecot: imap(user): Error: Recent flags state corrupted for 
mailbox Junk
Nov  9 14:32:29 ds dovecot: imap(user): Error: 
/var/mail/user_dbox/mailboxes/Junk/dbox-Mails/dovecot.index reset, view is now 
inconsistent
Nov  9 14:32:29 ds dovecot: imap(user): Panic: Message count decreased
Nov  9 14:32:29 ds dovecot: imap(user): Error: Raw backtrace: 
/usr/local/lib/dovecot/libdovecot.so.0(+0x89cc0) [0x7f0b64641cc0] -> 
/usr/local/lib/dovecot/libdovecot.so.0(+0x89d9e) [0x7f0b646
41d9e] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f0b645e4165] -> 
dovecot/imap() [0x42259c] -> dovecot/imap(imap_sync_more+0x104) [0x422f14] -> 
dovecot/imap() [0x410720] -> do
vecot/imap() [0x4108d1] -> 
/usr/local/lib/dovecot/libdovecot-storage.so.0(+0x52147) [0x7f0b64917147] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xe2) 
[0x7f0b64654992]
-> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x93) 
[0x7f0b64655d83] -> 
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) 
[0x7f0b64654b45] -> /usr/l
ocal/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f0b64654cf8] -> 
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f0b645ea243] -> dovecot/imap(main+0x312) [0x40c612
] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f0b64214f45] 
-> dovecot/imap() [0x40c780]
Nov  9 14:32:30 ds dovecot: imap(bkc): Fatal: master: service(imap): child 8456 
killed with signal 6 (core dumped)


On Dovecot 2.2.6.0:
Nov 10 10:35:13 ds dovecot: imap(user): Error: Recent flags state corrupted for 
mailbox Junk
Nov 10 10:35:13 ds dovecot: imap(user): Error: 
/var/mail/user_dbox/mailboxes/Junk/dbox-Mails/dovecot.index reset, view is now 
inconsistent
Nov 10 10:35:13 ds dovecot: imap(user): IMAP session state is inconsistent, 
please relogin. in=6212 out=49396

Re: post-delivery virus scan

[Stephan Bosch]

Op 11/10/2016 om 10:05 AM schreef Teemu Huovila:
>
> On 09.11.2016 23:36, Brad Koehn wrote:
>> I have discovered that many times the virus definitions I use for scanning 
>> messages (ClamAV, with the unofficial signatures 
>> http://sanesecurity.com/usage/linux-scripts/) are updated some time after my 
>> server has received an infected email. It seems the virus creators are 
>> trying to race the virus definition creators to see who can deliver first; 
>> more than half of the infected messages are found after they’ve been 
>> delivered. Great. 
>>
>> To help detect and remove the infected messages after they’ve been delivered 
>> to users’ mailboxes, I created a small script that iterates the INBOX and 
>> Junk mailbox directories, scans recent messages for viruses, and deletes 
>> them if found. The source of my script (run via cron) is here: 
>> https://gitlab.koehn.com/snippets/9
>>
>> Unfortunately Dovecot doesn’t like it if messages are deleted (dbox) out 
>> from under it. I tried a doveadm force-resync on the folder containing the 
>> messages, but it seems Dovecot is still unhappy. At least on the new version 
>> (2.2.26.0) it doesn’t crash; 2.2.25 would panic and coredump when it 
>> discovered messages had been deleted. 
>>
>> I’m wondering if there’s a better way to scan recent messages and eradicate 
>> them so the Dovecot isn’t upset when it happens. Maybe using doveadm search? 
>> Looking for suggestions. 
> The removal should if possible be done with the doveadm cli tool or using the 
> doveadm http api.

Still, Dovecot should handle external removal of messages gracefully.
What exactly happens?

Regards,

Stephan.

Re: post-delivery virus scan

[Teemu Huovila]

On 09.11.2016 23:36, Brad Koehn wrote:
> I have discovered that many times the virus definitions I use for scanning 
> messages (ClamAV, with the unofficial signatures 
> http://sanesecurity.com/usage/linux-scripts/) are updated some time after my 
> server has received an infected email. It seems the virus creators are trying 
> to race the virus definition creators to see who can deliver first; more than 
> half of the infected messages are found after they’ve been delivered. Great. 
> 
> To help detect and remove the infected messages after they’ve been delivered 
> to users’ mailboxes, I created a small script that iterates the INBOX and 
> Junk mailbox directories, scans recent messages for viruses, and deletes them 
> if found. The source of my script (run via cron) is here: 
> https://gitlab.koehn.com/snippets/9
> 
> Unfortunately Dovecot doesn’t like it if messages are deleted (dbox) out from 
> under it. I tried a doveadm force-resync on the folder containing the 
> messages, but it seems Dovecot is still unhappy. At least on the new version 
> (2.2.26.0) it doesn’t crash; 2.2.25 would panic and coredump when it 
> discovered messages had been deleted. 
> 
> I’m wondering if there’s a better way to scan recent messages and eradicate 
> them so the Dovecot isn’t upset when it happens. Maybe using doveadm search? 
> Looking for suggestions. 
The removal should if possible be done with the doveadm cli tool or using the 
doveadm http api.

br,
Teemu Huovila
> 
> 
> 
> 
> ---
> Brad 
> 

Re: search body with wildcards

[W. de Hoog]

On 2016-11-09 09:52, W. de Hoog wrote:

Hi,

A question. We are using fts_solr. When searching for content in BODY I
noticed that dovecot only supports full words. "BODY calibration"
returns results but "BODY calibra" does not. Nor "BODY calibra*".

Solr does support searching with wildcards so why is it that dovecot
does not?


When searching:

  doveadm search mailbox shared/* BODY calib*

solr is queried with

  ...body:calib%5c*...

so the wildcard is escaped. Is there any way to disable this escaping?

regards,

--
Willem-Jan de Hoog

tons of dovecot/config processes

[Arkadiusz Miśkiewicz]

Hi.

I've noticed that dovecot (using 2.2.26.0 here) starts dovecot/config
processes that stay for long time. Example:

[root@ixion-pld ~]# service dovecot restart
Stopping Dovecot 
service...[
 DONE ]
Starting Dovecot 
service...[
 DONE ]
[root@ixion-pld ~]# ps aux|grep dovecot
root 25333  0.0  0.0  13736  2480 ?Ss   09:40   0:00 
/usr/sbin/dovecot
dovecot  25336  0.0  0.0   9480   924 ?S09:40   0:00 dovecot/anvil 
[0 connections]
root 25337  0.0  0.0   9612  2416 ?S09:40   0:00 dovecot/log
root 25339  0.0  0.0  12496  3256 ?S09:40   0:00 dovecot/config
root 25341  0.0  0.0 132168   888 pts/1S+   09:40   0:00 grep dovecot
[root@ixion-pld ~]# doveadm reload
[root@ixion-pld ~]# ps aux|grep dovecot
root 25333  0.0  0.0  13872  2720 ?Ss   09:40   0:00 
/usr/sbin/dovecot
dovecot  25336  0.0  0.0   9480   924 ?S09:40   0:00 dovecot/anvil 
[0 connections]
root 25344  0.0  0.0   9612  2428 ?S09:40   0:00 dovecot/log
root 25346  0.0  0.0  12496  3192 ?S09:40   0:00 dovecot/config
root 25348  0.0  0.0 132168   876 pts/1S+   09:40   0:00 grep dovecot

so far good - only one dovecot/config. Lets connect to pop3 and keep connection

[root@ixion-pld ~]# telnet localhost pop3
Trying 127.0.0.1.110...
Connected to localhost.
Escape character is '^]'.
+OK Mail server ready.


on the other console

[root@ixion-pld ~]# ps aux|grep dovecot
root 25333  0.0  0.0  13872  2720 ?Ss   09:40   0:00 
/usr/sbin/dovecot
dovecot  25336  0.0  0.0   9480   924 ?S09:40   0:00 dovecot/anvil 
[2 connections]
root 25344  0.0  0.0   9612  2428 ?S09:40   0:00 dovecot/log
root 25346  0.0  0.0  12496  3192 ?S09:40   0:00 dovecot/config
dovenull 25364  0.0  0.0  20908  4080 ?S09:41   0:00 
dovecot/pop3-login [127.0.0.1]
dovecot  25365  0.0  0.0 100236  7776 ?S09:41   0:00 dovecot/auth 
[0 wait, 0 passdb, 0 userdb]
root 25368  0.0  0.0 132168   856 pts/1S+   09:41   0:00 grep dovecot

so there is a client connected and one dovecot/config. Lets reload:

[root@ixion-pld ~]# doveadm reload
[root@ixion-pld ~]# ps aux|grep dovecot
root 25333  0.0  0.0  13872  2752 ?Ss   09:40   0:00 
/usr/sbin/dovecot
dovecot  25336  0.0  0.0   9480   924 ?S09:40   0:00 dovecot/anvil 
[2 connections]
root 25344  0.0  0.0   9612  2428 ?S09:40   0:00 dovecot/log
root 25346  0.0  0.0  12920  3700 ?S09:40   0:00 dovecot/config
dovenull 25364  0.0  0.0  20908  4080 ?S09:41   0:00 
dovecot/pop3-login [127.0.0.1]
dovecot  25365  0.0  0.0 100236  7776 ?S09:41   0:00 dovecot/auth 
[0 wait, 0 passdb, 0 userdb]
root 25371  0.0  0.0   9612  2196 ?S09:41   0:00 dovecot/log
root 25373  0.0  0.0  12496  3196 ?S09:41   0:00 dovecot/config
root 25375  0.0  0.0 132168   856 pts/1S+   09:41   0:00 grep dovecot

now we have two dovecot/config processes. Second dovecot/config stays there
until client disconnects (what for?).

When clients disconnects we are back to single dovecot/config:

[root@ixion-pld ~]# ps aux|grep dovecot
root 25333  0.0  0.0  13872  2752 ?Ss   09:40   0:00 
/usr/sbin/dovecot
dovecot  25336  0.0  0.0   9480   924 ?S09:40   0:00 dovecot/anvil 
[0 connections]
root 25371  0.0  0.0   9612  2196 ?S09:41   0:00 dovecot/log
root 25373  0.0  0.0  12496  3196 ?S09:41   0:00 dovecot/config
root 25418  0.0  0.0 132168   852 pts/1S+   09:43   0:00 grep dovecot


Now on production server where are tons of clients this looks more insane:

# ps aux|grep dovecot/config | wc -l
56

Note that I'm running with
shutdown_clients = no
here (+ high performance auth/login variant).

So looks like something is not right here. Obviously with shutdown_clients=yes
this doesn't occur since clients are disconnected.

doveadm reload can happen every 2 minutes (because dovecot requires
reload when SSL certificates change; new domain gets added, new cert gets
automatically created -> reload, certificate is renewed (every 2 months) -> 
reload etc)

-- 
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

Re: How does one mark all messages as read (imap4flag "seen") with sieve?

[Bill Shirley]

I don't use the Anti-Spam plugin; I just fire off a BASH script every four 
hours with
crontab which iterates thru the vmail email accounts and trains Spamassassin 
'per-user'
accounts.  If the script sounds interesting I can post it here.  It probably 
could use a little polish
though.

Bill

On 11/9/2016 6:49 PM, Ben Johnson wrote:

On 11/5/2016 1:22 PM, Larry Rosenman wrote:

What OS/MTA are you using?  Can you give me (privately if you want) a
re-hash of the LDA issues?

I'm using FreeBSD 10.3 / Exim for my set up and LMTP for ALL deliveries,
and it works great.

Thanks again for your willingness to help with this, Larry.

I'm using Postfix.

Regarding the OS, I'm using Ubuntu 16.04 here, which ships dovecot
2.2.22 at present.

 Slightly off-topic, but I'll bring it full-circle... 

Sure, a quick recap of the crashing issue I'm having with dovecot-lda:

I struggled to get this working the first time (in dovecot 2.0.19), but
prevailed with lots of help from this list. I described the roadblocks I
encountered along the way in this thread:

http://www.dovecot.org/list/dovecot/2013-June/091018.html

All was well until I upgraded from Ubuntu 12.04 LTS to 14.04 LTS and
thereby from Dovecot 2.0.19 to 2.2.9. To be clear (and it may be very
relevant), this was a "manual" server migration and not an
OS-level/package-managed upgrade. Point being, the potential to botch
some aspect of the extremely fragile configuration was absolutely present!

I wrote about the problems I encountered after the upgrade here:

http://www.dovecot.org/list/dovecot/2014-July/097234.html

The thread died-out, but I rekindled it here:

http://www.dovecot.org/list/dovecot/2014-August/097385.html

I ran out of steam after a soft dead-end.

I wrote more about it a couple months later, mostly in the context of
difficulty with dovecot-lda logging in an effort to debug the issue, but
the thread received no replies:

http://www.dovecot.org/list/dovecot/2014-October/098127.html

Nearly two years later, I tried again:

http://www.dovecot.org/list/dovecot/2016-August/105221.html

The thread received some traction, and I changed the subject line to be
more accurate partway through, which begins here:

http://www.dovecot.org/list/dovecot/2016-August/105236.html

I ended-up running with Karol's final suggestion, which was to forego
the LDA in favor of a simple filesystem move/copy operation.

But now the problem I'm having (to bring it full circle!) is that I
can't mark the Antispam plugin's incoming emails as seen/read
automatically because they aren't delivered by an LDA. And I want to get
this working with an LDA again for this reason, among others, such as
quotas not being enforced when a "trained" message is "copied" on the
filesystem instead of "delivered" via an LDA.

The more I think about this, the more I think I should go back and study
the very first thread from June, 2013... maybe the solution is more or
less the same!

-Ben