Re: [Sieve] Is the way to run external script to get attachment?

[Konstantin Khomoutov]

On Wed, 15 Feb 2017 17:13:31 +0700
Konstantin Shalygin  wrote:

> Read all about Extprograms, but have a question.
> Use case:
> Every day we have couple messages with attachment (from one sender,
> this is one-way communication). This attachment is uploads to
> ownCloud. By hand... need some automation.
> As I see, pipe cat execute scripts with text data USER/FROM/SUBJECT - 
> and I think with variables it can be any text data, how about
> attachment?
> 
> How I see this: script executes and as arg receives from sieve 
> attachment file, or message file and we can parse it for attachment. 
> When we have attachment - upload to ownCloud via RESTapi and purge
> message.
> 
> Found some realization for encrypt messages 
> https://github.com/EtiennePerot/gpgit/blob/master/encmaildir.sh
> but for me more easy just connect to IMAP and do what I want (but I
> love sieve).

Do you really need Sieve for this?

IMO such things are best handled in the SMTP server:

1) Set up an alias for the mail address receiving these mails.

2) Make that alias expand both to its real final destination
   (an address, mails to which would be delivered to the IMAP
   folder as before) and to a special "program" entry which looks
   like "| /path/to/the/external/program".

See `man aliases` [4] for more info.

The program is supposed to receive the mail message to its standard
input stream and do whatever it wishes with them.  For instance, this
could be a shell script calling something like `ripmime` [1] on the
input, saving the attachments and then calling into whatever would
upload them.

Note that if the program completed its task OK, it should return with
the zero result code.  Otherwise it's advised to use one of result
codes defined by Sendmail, which are described in [2] and whose exact
numeric values could be googled (for instance, see [3]).  If your
program return a "known" exit code on error, you'll get better
diagnostics reported for that failure by your SMTP server.

1. http://www.pldaniels.com/ripmime/
2. http://docstore.mik.ua/orelly/networking/sendmail/ch36_05.htm
3. https://gist.github.com/bojanrajkovic/831993
4. http://www.postfix.org/aliases.5.html

Re: Backing up dovecot mailboxes

[Konstantin Khomoutov]

On Wed, 9 Nov 2016 11:50:06 -0500
Jerry  wrote:

> Does anyone have a working solution as to how I can use "doveadm
> backup" to back up a virtual users mailbox to a removable USB device.
> 
> I keep receiving numerous errors and no files are backed up although
> the directories are created.

Well, a solution I demonstrated in our previous thread on this same
topic works for me (`doveadm user *` plus `dsync`.

The thread got derailed into into discussion of how to use
`doveadm backup` properly to have a one-step backup of all the virtual
users, but it was sadly inconclusive.

All-in-all, we need to see exact commands you run and exact error
messages you receive to be able to do suggestions or at least informed
guesses.

Re: Outlook 2010 woes

[Konstantin Khomoutov]

On Tue, 1 Nov 2016 18:20:14 -0500
Bryan Holloway  wrote:

> In case anyone is interested, we finally found the problem:
> 
> The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled
> along with "plain". This is what was causing the four-second delay,
> but only with Outlook clients.
> 
> Everything is working great now across the board.
> 
> Thanks again to everyone's suggestions.

Thanks for sharing.

It's pretty amazing how far removed the problem cause can be from that
problem's manifestation ;-)

Re: Backing up and Importing IMAP folders

[Konstantin Khomoutov]

On Thu, 20 Oct 2016 16:57:45 +0300 (EEST)
Aki Tuomi  wrote:

[...]
> > Alternatively you can use `dsync` to perform backup with a native
> > Dovecot tool.  It's able to sync mailboxes of any Dovecot user --
> > including synchronizing a mailbox to an empty (yet) spool.
> > You'll need to do a bit of shell scripting which would spin around
> > calling `doveadm user *` and feeding its output to something like
> > 
> >   while read user; do \
> > dest="/var/backup/dovecot/$user";
> > mkdir -p "$dest" && chown vmail:vmail "$dest" \
> >   && chmod 0755 "$dest"
> > dsync -u "$user" backup "maildir:$dest" \
> >   done
> > 
> > Note that you will only need this if you don't want to shut down
> > Dovecot to copy its mail spool out.
> 
> You can also use doveadm backup -A maildir:%u/

Could you please elaborate?

I have a typical "virtual users" setup where I do have

  mail_home = /var/local/mail/%Ln
  mail_location = maildir:~/mail

and everything is stored with uid=vmail / gid=vmail (much like
described in the wiki, that is).

I'd like to use a single call to `doveadm backup -A ...` to back up the
whole /var/local/mail/* to another location (say, /var/backups/dovecot/)
so that is has the same structure, just synchronized with the spool.
(The purpose is to then backup the replica off-site).

I tried to call

  doveadm backup -A maildir:/var/backups/dovecot/%u

and it created a directory "/var/backups/dovecot/%u" (with literal
"%u", that is), created what appeared to be a single mailbox structure
under it and after a while scared a heck out of me with a series of
error messages reading

dsync(user1): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX
can't be deleted.
dsync(user2): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX
can't be deleted.
...

for each existing user.

It appears that it luckily failed to delete anything in the source
directory (though I have no idea what it actually tried to do).

Reading the doveadm-backup(1) multiple times still failed to shed a
light for me on how to actually backup the whole maildir hierarchy for
all existing users.

So, the question: how do I really should go about backing up the whole
mailbox hierarchy in the case of virtual users?

Re: Backing up and Importing IMAP folders

[Konstantin Khomoutov]

On Thu, 20 Oct 2016 17:38:31 +0300 (EEST)
Aki Tuomi  wrote:

[...]
> > > > Alternatively you can use `dsync` to perform backup with a
> > > > native Dovecot tool.  It's able to sync mailboxes of any
> > > > Dovecot user -- including synchronizing a mailbox to an empty
> > > > (yet) spool. You'll need to do a bit of shell scripting which
> > > > would spin around calling `doveadm user *` and feeding its
> > > > output to something like
> > > > 
> > > >   while read user; do \
> > > > dest="/var/backup/dovecot/$user";
> > > > mkdir -p "$dest" && chown vmail:vmail "$dest" \
> > > >   && chmod 0755 "$dest"
> > > > dsync -u "$user" backup "maildir:$dest" \
> > > >   done
> > > > 
> > > > Note that you will only need this if you don't want to shut down
> > > > Dovecot to copy its mail spool out.
> > > 
> > > You can also use doveadm backup -A maildir:%u/
> > 
> > Looks like `doveadm` of my Dovecot 2.2 (Debian 8.0 Jessie) does not
> > support the "backup" subcommand.  Is it a past-2.2 addition?
> 
> We aren't past 2.2 yet. But it should work with dsync -A backup as
> well I guess.

Oh, that's a documentation problem: the manual page doveadm(1) does not
mention the word "backup" at all while running the command actually
tells it's supported:

  $ doveadm backup -A
  doveadm backup [-u |-A] [-S ] [-dfR] [-l ]
  [-r ] [-m ] [-n  | -N] [-x ]
  [-s ] 

Good to know, thanks!

Re: Backing up and Importing IMAP folders

[Konstantin Khomoutov]

On Thu, 20 Oct 2016 16:57:45 +0300 (EEST)
Aki Tuomi  wrote:

[...]
> > Alternatively you can use `dsync` to perform backup with a native
> > Dovecot tool.  It's able to sync mailboxes of any Dovecot user --
> > including synchronizing a mailbox to an empty (yet) spool.
> > You'll need to do a bit of shell scripting which would spin around
> > calling `doveadm user *` and feeding its output to something like
> > 
> >   while read user; do \
> > dest="/var/backup/dovecot/$user";
> > mkdir -p "$dest" && chown vmail:vmail "$dest" \
> >   && chmod 0755 "$dest"
> > dsync -u "$user" backup "maildir:$dest" \
> >   done
> > 
> > Note that you will only need this if you don't want to shut down
> > Dovecot to copy its mail spool out.
> 
> You can also use doveadm backup -A maildir:%u/

Looks like `doveadm` of my Dovecot 2.2 (Debian 8.0 Jessie) does not
support the "backup" subcommand.  Is it a past-2.2 addition?

Re: Backing up and Importing IMAP folders

[Konstantin Khomoutov]

On Thu, 20 Oct 2016 09:18:12 -0400
Jerry  wrote:

> I am running Dovecot with Postfix on a FreeBSD machine. There are
> problems with the drive and I cannot depend on it. Dovecot saves all
> mail in IMAP format. I want to back up the mail folders, install a new
> HD, install the latest FreeBSD OS and then reinstall my programs.
> Reinstalling Dovecot is simple, but how do I reinstall the IMAP
> folders? Can Dovecot backup the folders onto a CD and then import them
> when I reinstall it? My mail is kept under “/var/mail/vmail”. Should I
> just back up that entire directory structure and then restore it
> later?

That should work (just make sure Dovecot is not running to not have a
race between your backup software and the IMAP server and clients).

Alternatively you can use `dsync` to perform backup with a native
Dovecot tool.  It's able to sync mailboxes of any Dovecot user --
including synchronizing a mailbox to an empty (yet) spool.
You'll need to do a bit of shell scripting which would spin around
calling `doveadm user *` and feeding its output to something like

  while read user; do \
dest="/var/backup/dovecot/$user";
mkdir -p "$dest" && chown vmail:vmail "$dest" \
  && chmod 0755 "$dest"
dsync -u "$user" backup "maildir:$dest" \
  done

Note that you will only need this if you don't want to shut down
Dovecot to copy its mail spool out.

Re: Outlook 2010 woes

[Konstantin Khomoutov]

On Thu, 13 Oct 2016 10:35:14 -0500
Bryan Holloway  wrote:

> > [...]
> >> Is there a way to see the IMAP commands coming from the client?
> >> I've tried looking at PCAPs, but of course they're encrypted so I
> >> can't see the actual dialog going on between the server and
> >> client. I didn't see an obvious way to do this in the docs.
> >
> > If you have access to the SSL/TLS key (IOW, the private part of the
> > cert) the server uses to secure IMAP connections you can dump the
> > IMAP traffic using the `ssldump` utility (which builds on
> > `tcpdump`).
> 
> I do, but the client is using a DH key exchange so I only have the 
> server-side private key.
> 
> Tried that using Wireshark's decoder features and ran into this
> problem. I'm assuming I'd run into the same using ssldump, but I'll
> give it a shot!

I think DH is not the culprit: just to be able to actually decode SSL
traffic, you must have the server private key when you're decoding the
SSL handshake phase -- to be able to recover the session keys, which
you then use to decode the actual tunneled data.

Re: Outlook 2010 woes

[Konstantin Khomoutov]

On Thu, 13 Oct 2016 09:53:19 -0500
Bryan Holloway  wrote:

[...]
> Is there a way to see the IMAP commands coming from the client? I've 
> tried looking at PCAPs, but of course they're encrypted so I can't
> see the actual dialog going on between the server and client. I
> didn't see an obvious way to do this in the docs.

If you have access to the SSL/TLS key (IOW, the private part of the
cert) the server uses to secure IMAP connections you can dump the IMAP
traffic using the `ssldump` utility (which builds on `tcpdump`).

Re: Outlook 2010 woes

[Konstantin Khomoutov]

On Wed, 12 Oct 2016 12:06:19 -0500
Bryan Holloway  wrote:

[...]
> >> Basically what's happening is that users are seeing large delays
> >> when navigating between different IMAP folders. So, for example,
> >> user "X" is sitting idle in their INBOX. If they then click on
> >> another folder there's a good 6-7 second delay before you can view
> >> its contents. If you immediately then navigate to other folders,
> >> you get a rapid response. But if the client then goes idle again
> >> for 10+ seconds, you will get this delay again. Some are reporting
> >> the OS saying "Outlook is not responding." (Everyone is running
> >> Windows 7.)
> > [...]
> >> Any help, suggestions, or pointers would be greatly appreciated.
> >
> > Do you see
> >
> >   imap(username) Disconnected for inactivity in=X out=Y
> >
> > in the logs?
> >
> > From your description, it appears as if outlook gets disconnected at
> > some point, and that's why fast changing of the folders works OK
> > (the connection is live) and doing this after a pause forces a
> > reconnect with the following relogin.
> >
> > Just a shot in the dark but still...
> 
> I do see quite a few of those, but I tend to see those for many
> clients across the board.
> 
> That does make a lot of sense though. I'm looking at the logs right
> now, and I see what amounts to a login, followed by a disconnect
> 10-15 seconds later.
> 
> I thought that a standard IMAP connection stays open for at least 30 
> minutes based on the RFC. (?)

Unfortunately, I don't have much familiarity with this topic but please
try googling for the exact phrases dovecot + "Disconnected for
inactivity" (literally double quoted) -- and you'll discover a hefty
amount of past discussions touching this topic.  They may give you
further clues as to what things to try next.

Re: dovecot is moving messages to spam

[Konstantin Khomoutov]

On Wed, 28 Sep 2016 10:15:25 -0300
Webert de Souza Lima  wrote:

> is there any dovecot rule settings besides X-Spam-Flag header? Can it
> move messages via IMAP?
> 
> I have a message that is being moved to spam folder after delivered
> in the INBOX but it has no X-Spam-Flag and it's not beeing done by
> the user (I changed his password, suspended his account and made his
> login impossible).
> 
> This happens only when certain "FROM" address is present in the body,
> like the following message (sent via telnet):
[...]
> Sep 28 13:08:00 lmtp(my.user@my.domain): Info: OOKlA3rA61dNbwAAkzG9Ng:
> sieve: msgid=unspecified: stored mail into mailbox 'INBOX'
> 
> Sep 28 13:08:01 imap(my.user@my.domain): Info: copy from INBOX:
> box=INBOX.Spam, uid=154, msgid=, size=340, subject=Test
> 
> Sep 28 13:08:01 imap(my.user@my.domain): Info: expunge: box=INBOX,
> uid=18147, msgid=, size=340, subject=Test
[...]

Are you sure there's no Sieve script active for this user?
(Note that there also could be a global Sieve script or scripts which
are executed before/after those of a user.)

And have you really verified nothing logs into the server for sure
using that user's credentials (such as a Thunderbird instance with mail
filters enabled)?  Another thing to check is that this user's INBOX
folder is not shared with someone else (if at all possible).

Force SSL/TLS and client certificate checking only for connections from Internet

[Konstantin Khomoutov]

(Please CC me if possible.)

I have Dovecot 2.2.13 (shipped with Debian 8 "Jessie") serving IMAP for
a small organization.  It's connected both to the Internet and to that
organization LAN.

I'd like to have Dovecot has rather tight security requirements for
connections made from the Internet and reasonably laxed requirements
for the LAN clients.

Due to Dovecot authenticating users against a LDAP database which is
also used for logging into the users' machines, we have to use
somewhat weak passwords (randomly generated but short) for most users.

Because of this I want to:
* Mandate the usage of TLS for the connections made from Internet;
* Require the clients to present their SSL certificates, and
* Verify the validity of these certificates.

The certificates are issued by our own CA, so it's not a problem to
verify them.

The first iteration was to set

  ssl = required

and

  disable_plaintext_auth = yes

which I then overrode with a "remote" section for LAN connections:

  remote 192.168.0.0/16 {
disable_plaintext_auth = no
ssl = yes
  }

The next iteration was to actually enable verification of the client
certificates.

I have set the "ssl_ca" variable appropriately, then

  ssl_require_crl = yes
  ssl_verify_client_cert = yes

and

  auth_ssl_require_client_cert = yes

Then I went on and attempted to override the latter two settings
for the LAN clients by appending them to my special "remote" section
to it read:

  remote 192.168.0.0/16 {
disable_plaintext_auth = no
ssl = yes
ssl_verify_client_cert = no
auth_ssl_require_client_cert = no
  }

But after restart Dovecot complained that I cannot use the
"auth_ssl_require_client_cert" setting in a "remote" section.

I have removed it in the hope that still having

  ssl_verify_client_cert = no

there would inhibit SSL certificate checks for the LAN clients
but apparently it did not.

So, to recap:

* I need to have tight TLS settings for the clients connecting
  from the Internet:
  - TLS is required;
  - The client must present a valid certificate (issued by a CA
we control).

* LAN clients should use relaxed security settings: TLS is okay
  but that's all.

How do I go about setting this?

(On a side note, I can supposedly get away installing stunnel and
making it proxy IMAP and IMAPS connections to Dovecot -- at least from
the documentation, it appears stunnel supports IMAP+STARTTLS, -- but
I would rather have all this handled solely by Dovecot.)

Re: [Dovecot] test of mailing list

[Konstantin Khomoutov]

Pascal Volk wrote:


Im sendding messages to the list and they do not show up.

Who wrote this messages to the mailing list?
* http://dovecot.org/list/dovecot/2009-May/039893.html
* http://dovecot.org/list/dovecot/2009-May/039902.html
It's possible to switch off reception of own messages in the mailman 
settings. The original poster might have this problem.

[Dovecot] Virtual users, local delivery via LDA and LDAP troubles

[Konstantin Khomoutov]

We're investigating the possibiliy to migrate our mail system from 
Sendmail+Cyrus to Sendmail+Dovecot.


The system must use authentication against Windows AD (supposedly using 
LDAP) and must use virtual hosting.


So far we managed to work around a bug reported in [1], and IMAP/POP3 
authentication on LDAP works OK.
LDAP auth is set up using binds (Cyrus and Ejabberd authenticate against 
the same LDAP server without problems).

As we use virtual users, userdb is set to be static in a standard way:

userdb static {
  args = uid=10513 gid=10513 home=/var/local/dovecot/%u
}

After verifying IMAP/POP3 authentication works, I've set up the Dovecot 
LDA to deliver mail for domain users.
This exposed another problem which I don't understand: the delivery 
program tries to figure out whether the user exists (which is perfectly 
sensible), it talks to the master authentication process which 
seemingly uses passdb backend to search LDAP. But this fails with the 
message passdb doesn't support lookups, can't verify user's existence.


[2] suggests it's auth binds that prevent this scheme from functioning 
correctly, but we can't stop using auth binds as Windows AD doesn't 
store users' passwords in any way sensible for external consumption. 
This would also pose unnecessary security risk on the domain, as the 
account used for initial binding should have had rights to read 
passwords, and its credentials are placed in the Dovecot configuration 
file in clear text.


I read about allow_all_users in [3], but our Sendmail doesn't check 
whether the target user exists and we don't want to implement this as it 
logically pertains to the program which actually manages users' 
mailboxes -- Dovecot in our case.


Is there a way to solve the problem at hand within the specified 
constraints?


1. http://dovecot.org/pipermail/dovecot/2009-May/039540.html
2. http://www.mail-archive.com/dovecot@dovecot.org/msg09449.html
3. http://www.mail-archive.com/dovecot@dovecot.org/msg08848.html

[Dovecot] LDAP auth_bind fails to process certain LDAP server responses (Windows AD)

[Konstantin Khomoutov]

LDAP auth backend in Dovecot 1.1.5 seemingly fails to handle certain
paths of interaction with LDAP server, which is Windows AD running 
Windows 2003 server machine in my case.


The symptoms I observe look exactly as were already reported in [1]: 
authentication backend hangs after logging the fact it has sent the 
search request, then after a while the IMAP/POP3 connection is teared 
down by the server due to idle condition.


In the mentioned mail thread, it was proposed that if Dovecot doesn't 
log any LDAP server response, there is no such response, so I dumped the 
TCP traffic going between the backend and the server, and studied what 
happens with different settings.


The result is that it seems LDAP auth backend doesn't understand the 
case when the server offerrs one or more referrals which the client 
might want to check. It seems that the backend tries to do something 
else after it gets such a result from the LDAP server even if the server 
returned an entry matching the search request.


The described behaviour manifests itself with Windows AD only if I 
specify the domain name alone as the search base, e.g. 
dc=mydomain,dc=com. If I also include at least one OU, the server 
doesn't return its list of referrals and Dovecot LDAP auth backend works OK.


I attached two files produced by tcpdump -- one with hung search 
request, and another with successful one.

They are:

1) ldap-hung.pcap.gz
Captured with /etc/dovecot/dovecot-ldap.conf settings:
hosts = domain007.com
dn = saslau...@domain007.com
dnpass = secret
auth_bind = yes
base = dc=domain007, dc=com
pass_filter = ((objectClass=person)(sAMAccountName=%u))

Packet #8 is of interest in this dump.

2) ldap-ok.pcap.gz
Captured with /etc/dovecot/dovecot-ldap.conf settings:
hosts = domain007.com
dn = saslau...@domain007.com
dnpass = secret
auth_bind = yes
base = cn=Users, dc=domain007, dc=com
pass_filter = ((objectClass=person)(sAMAccountName=%u))

The only difference compared to the first case is the cn=Users 
prepended to the base DN, which prevents the server from attaching a 
list of referrals.


In both cases search result done is present in the server response.

3) ldap-ldapsearch.pcap.gz
This dump captures the conversation of the ldapsearch utility with the 
same server using the same parameters Dovecot LDAP auth backend uses in 
the first case. ldapsearch invocation was like this:
$ ldapsearch -h domain007.com -x -D saslau...@domain007.com -w secret -a 
always -b dc=domain007,dc=com 
'((objectClass=person)(sAMAccountName=kostix))'


General configuration info (package from Debian stable):

# dovecot --version
1.0.15

# dovecot -n
# 1.0.15: /etc/dovecot/dovecot.conf
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap imaps pop3 pop3s
login_dir: /var/run/dovecot/login
login_executable(default): /usr/lib/dovecot/imap-login
login_executable(imap): /usr/lib/dovecot/imap-login
login_executable(pop3): /usr/lib/dovecot/pop3-login
mail_privileged_group: mail
mail_executable(default): /usr/lib/dovecot/imap
mail_executable(imap): /usr/lib/dovecot/imap
mail_executable(pop3): /usr/lib/dovecot/pop3
mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
pop3_uidl_format(default):
pop3_uidl_format(imap):
pop3_uidl_format(pop3): %08Xu%08Xv
auth default:
  verbose: yes
  debug: yes
  passdb:
driver: ldap
args: /etc/dovecot/dovecot-ldap.conf
  userdb:
driver: passwd

1. http://www.mail-archive.com/dovecot@dovecot.org/msg09174.html


ldap-hung.pcap.gz
Description: application/gzip


ldap-ok.pcap.gz
Description: application/gzip


ldap-ldapsearch.pcap.gz
Description: application/gzip