Returning error response to the client from passdb (with dict protocol)

[Nagy, Attila]

Hi,

When using a passdb with (UNIX socket) dict protocol, I can return a 
JSON with fields. When I want to deny the user, I can send a JSON like this:


{"fail":true}

which yields (on POP) a friendly -ERR [AUTH] Authentication failed.

Is it possible to return the response text itself? I want to tell the 
user why the authentication has failed.

Re: Is it possible to disable pipelining in imapc?

[Nagy, Attila]

On 09/11/2017 03:01 PM, Aki Tuomi wrote:

On 11.09.2017 15:56, Nagy, Attila wrote:

On 09/11/2017 12:12 PM, Aki Tuomi wrote:

Is there some reason you can't use normal proxy instead of imap backend?
That is,return proxy, host=imap_backend, port=1430? There seems to be no
pipeline setting currently for imapc in v2.2.


Yes, because it's a dumb IMAP server, which doesn't implement a lot of
things, like SEARCH, FETCH BODYSTRUCTURE and similar.
Dovecot is used as a smart proxy, which makes it possible to use it as
a fully featured IMAP server.

Does the dumb server advertize pipelining?

Exactly how does an IMAP server advertise pipelining other than it says 
it's an IMAP server? :)
There is no such capability, extension etc. It's in RFC3501, which 
defines IMAP v4r1.


Dovecot has a proxy_nopipelining setting for its IMAP proxy, I just want 
to use that (ie: no pipelining) with imapc too. Whether it's dynamically 
configurable, even with the existing proxy_nopipelining, or with an 
imapc_features flag, it doesn't matter for me.


Of course, setting this per user would be nicer.

Re: Is it possible to disable pipelining in imapc?

[Nagy, Attila]

On 09/11/2017 12:12 PM, Aki Tuomi wrote:

Is there some reason you can't use normal proxy instead of imap backend?
That is,return proxy, host=imap_backend, port=1430? There seems to be no
pipeline setting currently for imapc in v2.2.

Yes, because it's a dumb IMAP server, which doesn't implement a lot of 
things, like SEARCH, FETCH BODYSTRUCTURE and similar.
Dovecot is used as a smart proxy, which makes it possible to use it as a 
fully featured IMAP server.

Re: Is it possible to disable pipelining in imapc?

[Nagy, Attila]

On 09/11/2017 11:14 AM, Aki Tuomi wrote:


On 11.09.2017 11:59, Nagy, Attila wrote:

On 09/11/2017 10:42 AM, Sami Ketola wrote:

On 11 Sep 2017, at 11.24, Nagy, Attila <b...@fsn.hu> wrote:
I use dovecot with a broken IMAP server (which doesn't properly
implement command pipelining amongst others) as an imapc backend.
Dovecot issues the above command sequence (SELECT and UID FETCH
pipelined), which doesn't work with this server.

Therefore I'm requesting an imapc_features setting,
IMAPC_FEATURE_NO_PIPELINING, which disables PIPELINING in dovecot
imapc. Similarly to other workarounds, like
IMAPC_FEATURE_FETCH_MSN_WORKAROUNDS,
IMAPC_FEATURE_FETCH_FIX_BROKEN_MAILS et al.

Dovecot IMAPC should not pipeline unless remote advertises pipelining
support in CAPABILITY response.
Are you saying that you have imap server which advertses support for
it but then does not support it
after all?


It's a capability in POP3 and an extension in (E)SMTP.
I don't know about pipelining capability in IMAP (BTW, the server
doesn't advertise such capability, in fact, it doesn't support any
capabilities), it's standard, plain old IMAP RFC:
https://tools.ietf.org/html/rfc3501#section-5.5

The whole topic is about a bad IMAP server, which doesn't properly
implement pipelining (hence IMAP RFC), like others, for which dovecot
implements a workaround in imapc.

But are you using it like with doveadm? or proxying connection to? How
are you invoking imapc?

Dovecot acts as a frontend to this broken server and I redirect users 
with passdb/userdb via dict protocol, returning entries like:

js = {
  'password': '',
  'nopassword': '',
  'host':'imap_backend',
  'port':1430,
  'driver':'imap',
  'proxy_nopipelining':'y',
  'userdb_user':qdata['Auth-User'],
  'userdb_imapc_user':qdata['Auth-User'],
  'userdb_imapc_password':qdata['Auth-Pass'],
  'destuser':qdata['Auth-User'],
  }

(and this is where I tried proxy_nopipelining as you can see, but of 
course it's irrelevant)

Re: Is it possible to disable pipelining in imapc?

[Nagy, Attila]

On 09/11/2017 10:42 AM, Sami Ketola wrote:

On 11 Sep 2017, at 11.24, Nagy, Attila <b...@fsn.hu> wrote:
I use dovecot with a broken IMAP server (which doesn't properly implement 
command pipelining amongst others) as an imapc backend.
Dovecot issues the above command sequence (SELECT and UID FETCH pipelined), 
which doesn't work with this server.

Therefore I'm requesting an imapc_features setting, 
IMAPC_FEATURE_NO_PIPELINING, which disables PIPELINING in dovecot imapc. 
Similarly to other workarounds, like IMAPC_FEATURE_FETCH_MSN_WORKAROUNDS, 
IMAPC_FEATURE_FETCH_FIX_BROKEN_MAILS et al.


Dovecot IMAPC should not pipeline unless remote advertises pipelining support 
in CAPABILITY response.
Are you saying that you have imap server which advertses support for it but 
then does not support it
after all?


It's a capability in POP3 and an extension in (E)SMTP.
I don't know about pipelining capability in IMAP (BTW, the server 
doesn't advertise such capability, in fact, it doesn't support any 
capabilities), it's standard, plain old IMAP RFC:

https://tools.ietf.org/html/rfc3501#section-5.5

The whole topic is about a bad IMAP server, which doesn't properly 
implement pipelining (hence IMAP RFC), like others, for which dovecot 
implements a workaround in imapc.

Re: Is it possible to disable pipelining in imapc?

[Nagy, Attila]

On 09/11/2017 09:17 AM, Aki Tuomi wrote:


On 08.09.2017 15:29, Nagy, Attila wrote:

On 09/08/2017 01:53 PM, Aki Tuomi wrote:

On 08.09.2017 14:50, Nagy, Attila wrote:

Hi,

I've a broken IMAP server, which doesn't support pipelining and fails
on dovecot's attempt to do this ([C] is dovecot's imapc, [S] is the
IMAP server):

[C] 24 LOGIN "user" "pass"
[S] 23 OK
[C] 25 SELECT "INBOX"
[C] 23 UID FETCH 2 (INTERNALDATE)
[S] 25 OK
[S] 23 BAD No mailbox selected

Sadly, fixing the server would be a hard task, turning off pipelining
in dovecot's imapc (which already supports many workarounds for
different servers) could be easy...

... except I can see no place to do this in imapc (only in pop3c and
imap proxy, which I guess doesn't apply here).

If I'm right, may I ask for a knob for such dumb servers?

Thanks,

For proxy, use proxy_nopipelining


Are you sure it works with imapc? I've tried it, and dovecot still
does the above.

So what are you exactly doing?


I use dovecot with a broken IMAP server (which doesn't properly 
implement command pipelining amongst others) as an imapc backend.
Dovecot issues the above command sequence (SELECT and UID FETCH 
pipelined), which doesn't work with this server.


Therefore I'm requesting an imapc_features setting, 
IMAPC_FEATURE_NO_PIPELINING, which disables PIPELINING in dovecot imapc. 
Similarly to other workarounds, like 
IMAPC_FEATURE_FETCH_MSN_WORKAROUNDS, 
IMAPC_FEATURE_FETCH_FIX_BROKEN_MAILS et al.

Re: Is it possible to disable pipelining in imapc?

[Nagy, Attila]

On 09/08/2017 01:53 PM, Aki Tuomi wrote:


On 08.09.2017 14:50, Nagy, Attila wrote:

Hi,

I've a broken IMAP server, which doesn't support pipelining and fails
on dovecot's attempt to do this ([C] is dovecot's imapc, [S] is the
IMAP server):

[C] 24 LOGIN "user" "pass"
[S] 23 OK
[C] 25 SELECT "INBOX"
[C] 23 UID FETCH 2 (INTERNALDATE)
[S] 25 OK
[S] 23 BAD No mailbox selected

Sadly, fixing the server would be a hard task, turning off pipelining
in dovecot's imapc (which already supports many workarounds for
different servers) could be easy...

... except I can see no place to do this in imapc (only in pop3c and
imap proxy, which I guess doesn't apply here).

If I'm right, may I ask for a knob for such dumb servers?

Thanks,

For proxy, use proxy_nopipelining

Are you sure it works with imapc? I've tried it, and dovecot still does 
the above.

Is it possible to disable pipelining in imapc?

[Nagy, Attila]

Hi,

I've a broken IMAP server, which doesn't support pipelining and fails on 
dovecot's attempt to do this ([C] is dovecot's imapc, [S] is the IMAP 
server):


[C] 24 LOGIN "user" "pass"
[S] 23 OK
[C] 25 SELECT "INBOX"
[C] 23 UID FETCH 2 (INTERNALDATE)
[S] 25 OK
[S] 23 BAD No mailbox selected

Sadly, fixing the server would be a hard task, turning off pipelining in 
dovecot's imapc (which already supports many workarounds for different 
servers) could be easy...


... except I can see no place to do this in imapc (only in pop3c and 
imap proxy, which I guess doesn't apply here).


If I'm right, may I ask for a knob for such dumb servers?

Thanks,

Re: Dict protocol changes string

[Nagy, Attila]

Hi,

On 03/13/2017 09:25 AM, Aki Tuomi wrote:


On 28.02.2017 17:59, Nagy, Attila wrote:

On 09/23/2016 08:05 AM, Aki Tuomi wrote:

On 29.07.2016 15:35, Nagy, Attila wrote:

I use pass and userdb with dict protocol in a similar way:

key passdb {
key = passdb^MAuth-User: %u^MAuth-Pass: %w^MAuth-Protocol:
%s^MClient-IP: %r
format = json
}

(^M is an \r character, inserted with vi CTRL-v + enter)

Until 2.2.24 this has worked, but 2.2.25 seems to convert that ASCII
13 into an ASCII 1 and an "r".

Python printout from what I get with 2.2.25:

'Lshared/passdb\x01rAuth-User: user\x01rAuth-Pass:
pass\x01rAuth-Protocol: pop3\x01rClient-IP: 1.2.3.4'

Is this change intentional? Why?

Hi!

Dict protocol escapes you newlines. You are expected to de-escape them
yourself.

Following escapes are done, you can de-escape them with your client.

\x00 => \x10
\x01 => \x11
\t => \x1t
\r => \x1r
\n => \x1n



Following up on this: dovecot 2.2.27 and 2.2.28 goes even further
(2.2.25 was OK).
If a user specifies a password with a % in it, dovecot silently
truncates it.
So for example if I specify (just to check this simple example is also
bad):
key passdb {
   key = %w
   format = json
}

and a user tries to log in with the password 'Lofasznehogyma%',
dovecot sends the following into the dict socket:
'Lshared/Lofasznehogyma'

According to user reports, other characters may also be affected.

Could you please fix this?

Hi!

Can you try this?

https://github.com/dovecot/core/commit/30feb7a30f193197f1aab8a7b04a26b42735.patch

Aki

I use 2.2.28, so I've actually tried this:
--- work/dovecot-2.2.28/src/auth/db-dict.c2017-03-13 
13:47:09.406931000 +0100
+++ work/dovecot-2.2.28/src/auth/db-dict.c.orig2017-03-13 
13:45:47.903461000 +0100

@@ -408,7 +408,7 @@
 continue;

 str_truncate(path, strlen(DICT_PATH_SHARED));
-str_append(path, key->key->key);
+var_expand(path, key->key->key, iter->var_expand_table);
 ret = dict_lookup(iter->conn->dict, iter->pool,
   str_c(path), >value);
 if (ret > 0) {

So far it looks ok.

Thanks,

Re: v2.2.28 released

[Nagy, Attila]

On 03/06/2017 11:30 PM, Timo Sirainen wrote:

On 6 Mar 2017, at 9.17, Tom Sommer  wrote:


On 2017-02-24 14:34, Timo Sirainen wrote:

http://dovecot.org/releases/2.2/dovecot-2.2.28.tar.gz
http://dovecot.org/releases/2.2/dovecot-2.2.28.tar.gz.sig

Are there any plans to do a bugfix-release, that includes the few issues seen 
in the mailing-list, or do you consider 2.2.28 safe to upgrade to?

I don't see anything critical. A couple of bugs that might or might not affect 
you. We'll have 2.2.29 soon enough, so no plans for other releases before that.

Truncating passwords with dict protocol* seems quite critical to me. :-O
Or is it just me, who's affected by that?

*: http://dovecot.org/list/dovecot/2017-February/107265.html

Re: Dict protocol changes string

[Nagy, Attila]

On 09/23/2016 08:05 AM, Aki Tuomi wrote:

On 29.07.2016 15:35, Nagy, Attila wrote:

I use pass and userdb with dict protocol in a similar way:

key passdb {
   key = passdb^MAuth-User: %u^MAuth-Pass: %w^MAuth-Protocol:
%s^MClient-IP: %r
   format = json
}

(^M is an \r character, inserted with vi CTRL-v + enter)

Until 2.2.24 this has worked, but 2.2.25 seems to convert that ASCII
13 into an ASCII 1 and an "r".

Python printout from what I get with 2.2.25:

'Lshared/passdb\x01rAuth-User: user\x01rAuth-Pass:
pass\x01rAuth-Protocol: pop3\x01rClient-IP: 1.2.3.4'

Is this change intentional? Why?

Hi!

Dict protocol escapes you newlines. You are expected to de-escape them
yourself.

Following escapes are done, you can de-escape them with your client.

\x00 => \x10
\x01 => \x11
\t => \x1t
\r => \x1r
\n => \x1n


Following up on this: dovecot 2.2.27 and 2.2.28 goes even further 
(2.2.25 was OK).
If a user specifies a password with a % in it, dovecot silently 
truncates it.

So for example if I specify (just to check this simple example is also bad):
key passdb {
  key = %w
  format = json
}

and a user tries to log in with the password 'Lofasznehogyma%', dovecot 
sends the following into the dict socket:

'Lshared/Lofasznehogyma'

According to user reports, other characters may also be affected.

Could you please fix this?

Re: imapc fetch optimization

[Nagy, Attila]

On 07/02/2015 04:22 PM, Nagy, Attila wrote:

On 05/04/15 17:45, Timo Sirainen wrote:

On 04 May 2015, at 16:59, Nagy, Attila <b...@fsn.hu> wrote:

Hi,

On 05/04/15 10:59, Timo Sirainen wrote:

On 28 Apr 2015, at 23:49, Nagy, Attila <b...@fsn.hu> wrote:

Hi,

imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, 
because it works even with the dumbest IMAP server, altough it 
really kills performance, especially on high latency lines.


I wonder: if IMAP servers can effectively handle boundless fetches 
(like a list with all wanted UIDs, or simply 1:* if all are 
needed), do you see this as a good addition to develop?
Set mail_prefetch_count = 10 or 100 or something and it'll do 
larger FETCHes. The higher the value, the more memory/disk space is 
used for storing the received mails.



I'm aware of that, but it doesn't, or at least not always.
For example Thunderbird issues this:
13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID 
Priority X-Priority References Newsgroups In-Reply-To Content-Type 
Reply-To)])

Dovecot does this to the IMAP backend
18 UID FETCH 333574 (BODY.PEEK[])

Oh, there were several bugs related to that. Fixed:

http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b
I've just tried 2.2.18, which has the above modifications, but the 
effect is the same.

I have these in the config:
# doveconf -n | egrep 'prefe|imapc'
imapc_features = rfc822.size fetch-headers
imapc_host = 10.3.34.12
mail_location = imapc:~/imapc
mail_prefetch_count = 100

With an empty imapc directory a POP3 login yields these on the backend 
IMAP server:

2 LOGIN "user" "pass"
1 LIST "" ""
3 SELECT "INBOX"
4 NOOP
5 UID FETCH 1:* (FLAGS)
6 UID FETCH 2 (RFC822.SIZE)
7 UID FETCH 3 (RFC822.SIZE)
8 UID FETCH 4 (RFC822.SIZE)
9 UID FETCH 5 (RFC822.SIZE)
10 UID FETCH 6 (RFC822.SIZE)
11 UID FETCH 7 (RFC822.SIZE)
12 UID FETCH 8 (RFC822.SIZE)
[...]
so on, for every message...

The response times for each of the FETCHes are around 13-16 ms, so 
logging into dovecot for the first time takes more than a minute with 
5000 mails in the INBOX and a half day with around a million...


It would be good to have a:
6 UID FETCH 1:* (RFC822.SIZE)
or even a:
5 UID FETCH 1:* (FLAGS RFC822.SIZE)
if needed (dovecot first fills up its cache, or other cases when the 
full list is needed)


If dovecot would issue the latter for this 5000-mail inbox, the first 
fetch (and POP login) would take only 500 ms instead of the 75 seconds 
that it takes now...

Could you please tweak this some more? :)

Running 2.2.25 I can observe the same behaviour:
Dovecot fetches each message's RFC822.SIZE independently, instead of 
doing a much more efficient fetch 1:* for all messages.
This delays the first login way too long, most clients just time out 
which Dovecot is busy fetching the sizes and try again and again and 
again, eating up all resources for some time.

Any chance that this will be fixed?

Thanks,

Dict protocol changes string

[Nagy, Attila]

Hi,

I use pass and userdb with dict protocol in a similar way:

key passdb {
  key = passdb^MAuth-User: %u^MAuth-Pass: %w^MAuth-Protocol: 
%s^MClient-IP: %r

  format = json
}

(^M is an \r character, inserted with vi CTRL-v + enter)

Until 2.2.24 this has worked, but 2.2.25 seems to convert that ASCII 13 
into an ASCII 1 and an "r".


Python printout from what I get with 2.2.25:

'Lshared/passdb\x01rAuth-User: user\x01rAuth-Pass: 
pass\x01rAuth-Protocol: pop3\x01rClient-IP: 1.2.3.4'


Is this change intentional? Why?

Re: WP8/Windows 10 mail app can't login, tries double auth

[Nagy, Attila]

On 04/28/16 21:27, Joseph Tam wrote:

Attila Nagy writes:


Are there anybody who could make WP8 (and W10 desktop clients it seems)
devices working with Dovecot's POP3 server?

The conversation looks like this:
S +OK ready
C CAPA
S +OK
S CAPA
S TOP
S UIDL
S RESP-CODES
S PIPELINING
S AUTH-RESP-CODE
S USER
S SASL PLAIN
S .
C AUTH
S +OK
S PLAIN
S .
C AUTH PLAIN
S +
C successful login they want to log in again with USER, which gets 
rejected.


I dimly remember some Windows clients preferring either "LOGIN" or 
"PLAIN"

style mechanism.  Have you tried allowing "LOGIN" style authentication?

auth_mechanisms = plain login

Yes, the effect was the same I remember.

Re: WP8/Windows 10 mail app can't login, tries double auth

[Nagy, Attila]

On 04/27/16 14:54, Nagy, Attila wrote:


The only solution here seems to be giving back -ERR to the client's 
AUTH command, which makes it try USER instead.

I'm not aware of any configuration options which makes this possible.
Or a client workaround option to allow any USER/PASS combination after a 
successful AUTH (however I haven't tried whether the client accepts it, 
or in this phase it's internal state is completely flawed).

WP8/Windows 10 mail app can't login, tries double auth

[Nagy, Attila]

Hi,

Are there anybody who could make WP8 (and W10 desktop clients it seems) 
devices working with Dovecot's POP3 server?


The conversation looks like this:
S +OK ready
C CAPA
S +OK
S CAPA
S TOP
S UIDL
S RESP-CODES
S PIPELINING
S AUTH-RESP-CODE
S USER
S SASL PLAIN
S .
C AUTH
S +OK
S PLAIN
S .
C AUTH PLAIN
S +
C First I thought the extra space after that OK causes this, but removing 
it changed nothing.


So these devices try to log in with AUTH PLAIN, then even after a 
successful login they want to log in again with USER, which gets rejected.


The only solution here seems to be giving back -ERR to the client's AUTH 
command, which makes it try USER instead.

I'm not aware of any configuration options which makes this possible.

Any other ideas?

Thanks,

Re: imapc tries to fetch uid 0

[Nagy, Attila]

On 03/29/16 19:41, Timo Sirainen wrote:

On 19 Feb 2016, at 17:12, Nagy, Attila <b...@fsn.hu> wrote:

Hi,

This is a capture from an imapc client session (which is triggered by a 
message, received via LMTP):
* OK [CAPABILITY IMAP4rev1 NAMESPACE] Ready
2 LOGIN "uid" "pass"
2 OK LOGIN succeeded
1 LIST "" ""
* LIST (\Noselect) "/" ""
1 OK LIST completed
3 APPEND "INBOX" {568}
+ Ready for 568 octets of data
[...]
3 OK APPEND complete
4 EXAMINE "INBOX"
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags 
permitted.
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1455885804]
* OK [UIDNEXT 5]
4 OK [READ-ONLY] EXAMINE successful
5 UID FETCH 0 (RFC822.SIZE BODY.PEEK[HEADER.FIELDS (Message-ID)])
5 BAD Illegal syntax: Bad message number 0

Dovecot 2.2.21

I don't see any problems with the conversation otherwise. Is it safe to assume 
this is a bug in imapc?

doveconf -n would be helpful. I guess you have mail_log plugin enabled? And 
imapc isn't used with any index directory? So Dovecot attempts to log 
Message-ID and size after saving a mail, but those aren't in cache (because 
indexes are disabled) and the transaction isn't committed yet either so the 
mail has no UID.

So the only fix for now that I can do is to just not send the FETCH command and 
instead fail it immediately:
https://github.com/dovecot/core/commit/969f57f722570982a5febbaab2462b692aa68733

Any further fixing would become difficult.

I don't have mail_log enabled (currently, but will be) and indexes 
aren't disabled.


Here's the actual config:
# 2.2.22 (fe789d2): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.13 (7b14904)
# OS: FreeBSD 10.3-PRERELEASE amd64
auth_mechanisms = plain login
disable_plaintext_auth = no
imapc_features = rfc822.size fetch-headers
imapc_host = 127.0.0.1
imapc_port = 1430
lmtp_proxy = yes
mail_gid = imapproxy
mail_home = /usr/home/imapproxy/%2.256Nu/%u
mail_location = imapc:~/imapc
mail_prefetch_count = 100
mail_uid = imapproxy
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart extracttext

namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix =
}
passdb {
  args = /usr/local/etc/dovecot/dovecot-dict-auth.conf.ext
  driver = dict
}
postmaster_address = postmaster
service lmtp {
  inet_listener lmtp {
address = 127.0.0.1
port = 24
  }
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
  }
}
ssl = no
userdb {
  driver = prefetch
}
userdb {
  args = /usr/local/etc/dovecot/dovecot-dict-auth.conf.ext
  driver = dict
}
protocol lmtp {
  mail_plugins = " sieve"
}

The user home contains a dovecot.index.cache and a .log file.
During the LMTP delivery (which turns into an IMAP APPEND, maybe this is 
where the bug lies? It's not too typical to do this I guess), they 
aren't changed.

Re: v2.2.22 released

[Nagy, Attila]

On 03/16/16 09:27, Timo Sirainen wrote:

http://dovecot.org/releases/2.2/dovecot-2.2.22.tar.gz
http://dovecot.org/releases/2.2/dovecot-2.2.22.tar.gz.sig

  + Added doveadm HTTP API: See
http://wiki2.dovecot.org/Design/DoveadmProtocol/HTTP
  + virtual plugin: Mailbox filtering can now be done based on the
mailbox metadata. See http://wiki2.dovecot.org/Plugins/Virtual
  + stats: Added doveadm stats reset to reset global stats.
  + stats: Added authentication statistics if auth_stats=yes.
  + dsync, imapc, pop3c & pop3-migration: Many optimizations,
improvements and error handling fixes.

Any chances of this fixed in this release?
http://dovecot.org/list/dovecot/2016-February/103251.html

imapc tries to fetch uid 0

[Nagy, Attila]

Hi,

This is a capture from an imapc client session (which is triggered by a 
message, received via LMTP):

* OK [CAPABILITY IMAP4rev1 NAMESPACE] Ready
2 LOGIN "uid" "pass"
2 OK LOGIN succeeded
1 LIST "" ""
* LIST (\Noselect) "/" ""
1 OK LIST completed
3 APPEND "INBOX" {568}
+ Ready for 568 octets of data
[...]
3 OK APPEND complete
4 EXAMINE "INBOX"
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] 
Flags permitted.

* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1455885804]
* OK [UIDNEXT 5]
4 OK [READ-ONLY] EXAMINE successful
5 UID FETCH 0 (RFC822.SIZE BODY.PEEK[HEADER.FIELDS (Message-ID)])
5 BAD Illegal syntax: Bad message number 0

Dovecot 2.2.21

I don't see any problems with the conversation otherwise. Is it safe to 
assume this is a bug in imapc?

Re: RFC: HTTP based storage API

[Nagy, Attila]

On 02/11/16 03:50, David Niklas wrote:

On 08 Feb 2016, at 11:01, Nagy, Attila <b...@fsn.hu> wrote:

Nearly every popular programming language has an LMTP/POP/IMAP
implementation, most of them suck in many different ways.
I don't know any server or library which provides a well-established,
compatible protocol frontend with an open backend API, which could be
used to easily make a custom storage backend for the LMTP/POP/IMAP
frontend in any language, in any programming paradigm, irregardless
of the frontend implementation.

Sorry to come in late, but as a curl lover I'd like to know what is wrong
with their implementation of pop3 and imap (project homepage
curl.haxx.se).

Well, most of the time it quickly turns out these are just partial 
addons, with so basic bugs, like #1456 was.
There is a reason why postfix, dovecot and others have so many man-hours 
in them.

BTW, this is about a server implementation...

RFC: HTTP based storage API

[Nagy, Attila]

Hi,

Nearly every popular programming language has an LMTP/POP/IMAP 
implementation, most of them suck in many different ways.
I don't know any server or library which provides a well-established, 
compatible protocol frontend with an open backend API, which could be 
used to easily make a custom storage backend for the LMTP/POP/IMAP 
frontend in any language, in any programming paradigm, irregardless of 
the frontend implementation.


Dovecot is very close to this: it speaks IMAP to the backend, but you 
still have to make a nearly complete IMAP implementation, which is a PITA.


Therefore I am curious about your opinion: what about an HTTP-based open 
backend (lib-storage) API to Dovecot?


Some thoughts on this:
1. it should be a simple REST API
2. it should support efficient operations on both sides (client, 
server), for example listing a multi-million folder shouldn't be a 
single JSON response, but a JSON stream, which could be produced and 
parsed individually or in smaller batches efficiently
3. it should support capabilities, somewhat like the current imapc 
backend, so if the backend service suppports search, it should announce 
this capability and dovecot could offload these requests to it, but if 
it doesn't, dovecot could build its indices.

4. only the needed amount of POP/IMAP legacy should appear in the protocol
5. the operations should be asynchronous on Dovecot's side (to be scalable)
6. maybe even Dovecot's index-queries could be offloaded to the backend 
service (this way the local file system wouldn't be used at all and all 
caches could be stored in a distributed dictionary), this could also be 
a capability

7. it could support some kind of push (for IDLE-ing and the like)
8. it should be stateless

If Dovecot could make this right, it could be the FUSE of the mail 
protocols, acting as a glue for experimental or production ready mail 
storage backends, without the need to write these backends in "Dovecot 
C" (I mean it's process and API structure).


Of course HTTP here is not a strict requirement. It could be anything, 
which can be easy to implement and understand and is able to fulfill the 
needs.

An example could be Dovecot's own dictionary protocol.

What do you think about this?

Re: RFC: HTTP based storage API

[Nagy, Attila]

On 02/08/16 11:16, Timo Sirainen wrote:

On 08 Feb 2016, at 11:59, Timo Sirainen <t...@iki.fi> wrote:

On 08 Feb 2016, at 11:01, Nagy, Attila <b...@fsn.hu> wrote:

Hi,

Nearly every popular programming language has an LMTP/POP/IMAP implementation, 
most of them suck in many different ways.
I don't know any server or library which provides a well-established, 
compatible protocol frontend with an open backend API, which could be used to 
easily make a custom storage backend for the LMTP/POP/IMAP frontend in any 
language, in any programming paradigm, irregardless of the frontend 
implementation.

Dovecot is very close to this: it speaks IMAP to the backend, but you still 
have to make a nearly complete IMAP implementation, which is a PITA.

Therefore I am curious about your opinion: what about an HTTP-based open 
backend (lib-storage) API to Dovecot?

We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close 
enough to how Dovecot works internally (although we'd still need the threading 
support) and also at least attempting to become a standard. We haven't really 
started it yet though, so now would be a good time to give any alternative 
suggestions or complaints about jmap :)

Oh, you were thinking about ability to provide IMAP/etc support for other 
random servers, and have Dovecot act as kind of a middleware and translate the 
requests. Maybe the answer is still jmap though? It would require jmap 
lib-storage backend similar to imapc, which would be doable, although not 
really something we're right now planning to implement.


Yeah, the opposite, in this case a jmap backend to Dovecot.
BTW, I think jmap is too high level and implementing a jmap server is 
very much like implementing an IMAP one.


I much more think of a pluggable, easy (remote) storage API, which has 
much less to do with IMAP, but can offer capabilities, which can help 
Dovecot (like the search and indexes).

Re: RFC: HTTP based storage API

[Nagy, Attila]

On 02/08/16 12:07, Timo Sirainen wrote:

On 08 Feb 2016, at 12:56, Nagy, Attila <b...@fsn.hu> wrote:

On 02/08/16 11:16, Timo Sirainen wrote:

Oh, you were thinking about ability to provide IMAP/etc support for other 
random servers, and have Dovecot act as kind of a middleware and translate the 
requests. Maybe the answer is still jmap though? It would require jmap 
lib-storage backend similar to imapc, which would be doable, although not 
really something we're right now planning to implement.


Yeah, the opposite, in this case a jmap backend to Dovecot.
BTW, I think jmap is too high level and implementing a jmap server is very much 
like implementing an IMAP one.

I much more think of a pluggable, easy (remote) storage API, which has much 
less to do with IMAP, but can offer capabilities, which can help Dovecot (like 
the search and indexes).

What kind of use cases are you thinking for this remote storage API? What kind 
of remote storages would implement it, and what kind of installations would use 
them?

Interfacing non-email systems to e-mail protocols and implementing other 
storage options (I can't say a better analogue than what FUSE is).
Using Dovecot as a protocol implementation, but not for the underlying 
storage method, which may be custom to the given solution.

A flexible way of implementing and adapting anything to IMAP.

For example if I want to store really old and archived e-mails in a 
costly compression format somewhere in the cloud, but hot mails locally, 
based on my custom policies.
In an international company there may be some regulations about what 
should go where, so for example if I can handle object placement myself, 
I can use the same installation to store non-EU mails in non-EU 
countries and others in US or EU clouds if law permits.


Intermixing several systems into one (like merging two IMAP accounts 
into one) etc.


Some of these are very costly to develop in Dovecot, while just a few 
lines in -say- Python or go.

Re: RFC: HTTP based storage API

[Nagy, Attila]

On 02/08/16 11:44, Marcus Rueckert wrote:

that sounds like s3 API based backend.

also note that dovecot comes with some "non local mail storage plugins"
in the commercial part:

```
In this release, we support Windows Azure, Amazon S3, Scality and
Dropbox systems.
```

so that should come close to what you want. for a local version you
could use e.g. radosdb/ceph or riak. both can provide the s3 API for
you.

Close, but this may be too low level, for example it may hide whose 
e-mail do you store (I don't know the exact implementation, it's not 
open source AFAIK).

userdb prefetch doesn't work with imapc (2.2.18)

[Nagy, Attila]

Hi,

I have the following config in dovecot 2.2.18:
# doveconf -n userdb passdb
userdb {
  driver = prefetch
}
passdb {
  args = host=127.0.0.1 port=1430
  default_fields = userdb_imapc_user=%u userdb_imapc_password=%w
  driver = imap
}

I guess it should cause a single login to the backend IMAP server when 
issuing a LOGIN to dovecot.
Yet, dovecot opens two TCP sessions. In the first, it issues a LOGIN and 
in the second it logs in again (LOGIN) and issues LIST "" "".


At the end (after LOGIN succeeds to dovecot), only one of the two 
connections remain open to the backend server.


Any ideas about this?

Thanks,

Re: imapc fetch optimization

[Nagy, Attila]

On 05/04/15 17:45, Timo Sirainen wrote:

On 04 May 2015, at 16:59, Nagy, Attila b...@fsn.hu wrote:

Hi,

On 05/04/15 10:59, Timo Sirainen wrote:

On 28 Apr 2015, at 23:49, Nagy, Attila b...@fsn.hu wrote:

Hi,

imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it 
works even with the dumbest IMAP server, altough it really kills performance, 
especially on high latency lines.

I wonder: if IMAP servers can effectively handle boundless fetches (like a list 
with all wanted UIDs, or simply 1:* if all are needed), do you see this as a 
good addition to develop?

Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. 
The higher the value, the more memory/disk space is used for storing the 
received mails.


I'm aware of that, but it doesn't, or at least not always.
For example Thunderbird issues this:
13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority 
X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])
Dovecot does this to the IMAP backend
18 UID FETCH 333574 (BODY.PEEK[])

Oh, there were several bugs related to that. Fixed:

http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b
I've just tried 2.2.18, which has the above modifications, but the 
effect is the same.

I have these in the config:
# doveconf -n | egrep 'prefe|imapc'
imapc_features = rfc822.size fetch-headers
imapc_host = 10.3.34.12
mail_location = imapc:~/imapc
mail_prefetch_count = 100

With an empty imapc directory a POP3 login yields these on the backend 
IMAP server:

2 LOGIN user pass
1 LIST  
3 SELECT INBOX
4 NOOP
5 UID FETCH 1:* (FLAGS)
6 UID FETCH 2 (RFC822.SIZE)
7 UID FETCH 3 (RFC822.SIZE)
8 UID FETCH 4 (RFC822.SIZE)
9 UID FETCH 5 (RFC822.SIZE)
10 UID FETCH 6 (RFC822.SIZE)
11 UID FETCH 7 (RFC822.SIZE)
12 UID FETCH 8 (RFC822.SIZE)
[...]
so on, for every message...

The response times for each of the FETCHes are around 13-16 ms, so 
logging into dovecot for the first time takes more than a minute with 
5000 mails in the INBOX and a half day with around a million...


It would be good to have a:
6 UID FETCH 1:* (RFC822.SIZE)
or even a:
5 UID FETCH 1:* (FLAGS RFC822.SIZE)
if needed (dovecot first fills up its cache, or other cases when the 
full list is needed)


If dovecot would issue the latter for this 5000-mail inbox, the first 
fetch (and POP login) would take only 500 ms instead of the 75 seconds 
that it takes now...

Could you please tweak this some more? :)

Thanks,

Re: imapc fetch optimization

[Nagy, Attila]

Hi,

On 05/04/15 10:59, Timo Sirainen wrote:

On 28 Apr 2015, at 23:49, Nagy, Attila b...@fsn.hu wrote:

Hi,

imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it 
works even with the dumbest IMAP server, altough it really kills performance, 
especially on high latency lines.

I wonder: if IMAP servers can effectively handle boundless fetches (like a list 
with all wanted UIDs, or simply 1:* if all are needed), do you see this as a 
good addition to develop?

Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. 
The higher the value, the more memory/disk space is used for storing the 
received mails.


I'm aware of that, but it doesn't, or at least not always.
For example Thunderbird issues this:
13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority 
X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])

Dovecot does this to the IMAP backend
18 UID FETCH 333574 (BODY.PEEK[])
19 UID FETCH 333575 (BODY.PEEK[])
20 UID FETCH 333576 (BODY.PEEK[])
21 UID FETCH 333577 (BODY.PEEK[])
And so on.

I have mail_prefetch_count set to 100:
# doveconf -n | grep prefet
mail_prefetch_count = 100

Re: imapc fetch optimization

[Nagy, Attila]

On 05/04/15 17:45, Timo Sirainen wrote

I'm aware of that, but it doesn't, or at least not always.
For example Thunderbird issues this:
13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority 
X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])
Dovecot does this to the IMAP backend
18 UID FETCH 333574 (BODY.PEEK[])

Oh, there were several bugs related to that. Fixed:

http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b

Thank you very much for the quick fix(es), I will try them.

Re: imapc fetch optimization

[Nagy, Attila]

On 05/04/15 17:47, Timo Sirainen wrote:
Also if you want imapc to use the full set of features with latest 
Dovecot, use: imapc_features=rfc822.size fetch-headers search After 
that it shouldn't be fetching BODY.PEEK[] anymore unless the body was 
actually wanted to be fetched. 
Very nice, and also the path you follow is great: allow the backend IMAP 
server to be dumb and yet -with these feature flags- we can exploit it 
more if needed.


And having the ability to deliver mails through lmtpd to the IMAP 
backend is priceless. :)

imapc fetch optimization

[Nagy, Attila]

Hi,

imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because 
it works even with the dumbest IMAP server, altough it really kills 
performance, especially on high latency lines.


I wonder: if IMAP servers can effectively handle boundless fetches (like 
a list with all wanted UIDs, or simply 1:* if all are needed), do you 
see this as a good addition to develop?


This could be a new imapc_features setting afterall...

Thanks,

[PATCH] Make Delivered-To optional

[Nagy, Attila]

Hi,

I've read at least one e-mail on this list about making delivered-to in 
lmtpd optional, but now I need this too, so I made a patch. The default 
remains as is now (enabled).


Rationale: I would like to dsync users and I have catchall POP mailboxes 
(meaning: a single mailbox gets the mails for a lot of e-mail addresses).
If I deliver the e-mails to Dovecot's lmtpd as the original RCPT TO, the 
Delivered-To header can be used for the POP client to sort the messages 
on their side to the correct mailbox, but because Dovecot sees this as a 
new user, it adds it to the replicator.db.
Now imagine a single mailbox with thousands of e-mail addresses, this 
will make thousands of entries in replicator.db, all of which Dovecot 
would like to replicate independently...


If I deliver to the mailbox uid, Dovecot dsync will work right (only one 
user will be added, no matter how many different e-mail addresses end up 
in this mailbox), but the Delivered-To will contain the uid, so the user 
can't use that to sort the messages into their final destination on 
their side.


So the solution here is that I disable the addition of Delivered-To 
header in Dovecot (because it doesn't know what is the original address) 
and add it in an upper layer, so during the lmtp transaction, the 
message will already contain the right value.
Currently Dovecot adds another Delivered-To line, which confuses the 
clients.


Please add this feature to Dovecot.

Thanks,
diff -r e3640ccaa76d doc/example-config/conf.d/20-lmtp.conf
--- a/doc/example-config/conf.d/20-lmtp.confSat Jan 10 04:32:42 2015 +0200
+++ b/doc/example-config/conf.d/20-lmtp.confTue Jan 13 11:29:20 2015 +0100
@@ -13,8 +13,11 @@
 # Verify quota before replying to RCPT TO. This adds a small overhead.
 #lmtp_rcpt_check_quota = no
 
+# Insert Delivered-To header to the messages, delivered through LMTP.
+#lmtp_add_delivered_to = yes
+
 protocol lmtp {
   # Space separated list of plugins to load (default is global mail_plugins).
   #mail_plugins = $mail_plugins
 }
- 
\ No newline at end of file
+ 
diff -r e3640ccaa76d src/lmtp/commands.c
--- a/src/lmtp/commands.c   Sat Jan 10 04:32:42 2015 +0200
+++ b/src/lmtp/commands.c   Tue Jan 13 11:29:20 2015 +0100
@@ -996,7 +996,7 @@
if (array_count(client-state.rcpt_to)  0) {
str_printfa(str, Return-Path: %s\r\n,
client-state.mail_from);
-   if (rcpt_to != NULL)
+   if (rcpt_to != NULL  client-lmtp_set-lmtp_add_delivered_to)
str_printfa(str, Delivered-To: %s\r\n, rcpt_to);
}
 
diff -r e3640ccaa76d src/lmtp/lmtp-settings.c
--- a/src/lmtp/lmtp-settings.c  Sat Jan 10 04:32:42 2015 +0200
+++ b/src/lmtp/lmtp-settings.c  Tue Jan 13 11:29:20 2015 +0100
@@ -60,6 +60,7 @@
DEF(SET_BOOL, lmtp_proxy),
DEF(SET_BOOL, lmtp_save_to_detail_mailbox),
DEF(SET_BOOL, lmtp_rcpt_check_quota),
+   DEF(SET_BOOL, lmtp_add_delivered_to),
DEF(SET_STR, lmtp_address_translate),
DEF(SET_STR_VARS, login_greeting),
DEF(SET_STR, login_trusted_networks),
@@ -71,6 +72,7 @@
.lmtp_proxy = FALSE,
.lmtp_save_to_detail_mailbox = FALSE,
.lmtp_rcpt_check_quota = FALSE,
+   .lmtp_add_delivered_to = TRUE,
.lmtp_address_translate = ,
.login_greeting = PACKAGE_NAME ready.,
.login_trusted_networks = 
diff -r e3640ccaa76d src/lmtp/lmtp-settings.h
--- a/src/lmtp/lmtp-settings.h  Sat Jan 10 04:32:42 2015 +0200
+++ b/src/lmtp/lmtp-settings.h  Tue Jan 13 11:29:20 2015 +0100
@@ -8,6 +8,7 @@
bool lmtp_proxy;
bool lmtp_save_to_detail_mailbox;
bool lmtp_rcpt_check_quota;
+   bool lmtp_add_delivered_to;
const char *lmtp_address_translate;
const char *login_greeting;
const char *login_trusted_networks;

mdbox GUID

[Nagy, Attila]

Hi,

The dbox page says:
Each message has a 128 bit globally unique identifier (GUID)
I guess this is what guid_128_generate() generates.

Is this believed to be really globally unique?

As far as I can tell, in the current architecture they are -given there 
are enough time precision-, because all dovecot daemons are single 
threaded, so they have concurrency but doesn't have parallelism (they 
are not multi threaded).


I would like to list all of the messages, with this GUID as the unique 
identifier, so a collision would be fatal here...