Re: imapc fetch optimization

2016-09-27 Thread Nagy, Attila 

On 07/02/2015 04:22 PM, Nagy, Attila wrote:

On 05/04/15 17:45, Timo Sirainen wrote:

On 04 May 2015, at 16:59, Nagy, Attila  wrote:

Hi,

On 05/04/15 10:59, Timo Sirainen wrote:

On 28 Apr 2015, at 23:49, Nagy, Attila  wrote:

Hi,

imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, 
because it works even with the dumbest IMAP server, altough it 
really kills performance, especially on high latency lines.


I wonder: if IMAP servers can effectively handle boundless fetches 
(like a list with all wanted UIDs, or simply 1:* if all are 
needed), do you see this as a good addition to develop?
Set mail_prefetch_count = 10 or 100 or something and it'll do 
larger FETCHes. The higher the value, the more memory/disk space is 
used for storing the received mails.



I'm aware of that, but it doesn't, or at least not always.
For example Thunderbird issues this:
13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID 
Priority X-Priority References Newsgroups In-Reply-To Content-Type 
Reply-To)])

Dovecot does this to the IMAP backend
18 UID FETCH 333574 (BODY.PEEK[])

Oh, there were several bugs related to that. Fixed:

http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b
I've just tried 2.2.18, which has the above modifications, but the 
effect is the same.

I have these in the config:
# doveconf -n | egrep 'prefe|imapc'
imapc_features = rfc822.size fetch-headers
imapc_host = 10.3.34.12
mail_location = imapc:~/imapc
mail_prefetch_count = 100

With an empty imapc directory a POP3 login yields these on the backend 
IMAP server:

2 LOGIN "user" "pass"
1 LIST "" ""
3 SELECT "INBOX"
4 NOOP
5 UID FETCH 1:* (FLAGS)
6 UID FETCH 2 (RFC822.SIZE)
7 UID FETCH 3 (RFC822.SIZE)
8 UID FETCH 4 (RFC822.SIZE)
9 UID FETCH 5 (RFC822.SIZE)
10 UID FETCH 6 (RFC822.SIZE)
11 UID FETCH 7 (RFC822.SIZE)
12 UID FETCH 8 (RFC822.SIZE)
[...]
so on, for every message...

The response times for each of the FETCHes are around 13-16 ms, so 
logging into dovecot for the first time takes more than a minute with 
5000 mails in the INBOX and a half day with around a million...


It would be good to have a:
6 UID FETCH 1:* (RFC822.SIZE)
or even a:
5 UID FETCH 1:* (FLAGS RFC822.SIZE)
if needed (dovecot first fills up its cache, or other cases when the 
full list is needed)


If dovecot would issue the latter for this 5000-mail inbox, the first 
fetch (and POP login) would take only 500 ms instead of the 75 seconds 
that it takes now...

Could you please tweak this some more? :)

Running 2.2.25 I can observe the same behaviour:
Dovecot fetches each message's RFC822.SIZE independently, instead of 
doing a much more efficient fetch 1:* for all messages.
This delays the first login way too long, most clients just time out 
which Dovecot is busy fetching the sizes and try again and again and 
again, eating up all resources for some time.

Any chance that this will be fixed?

Thanks,

Re: imapc fetch optimization

2015-07-02 Thread Nagy, Attila 

On 05/04/15 17:45, Timo Sirainen wrote:

On 04 May 2015, at 16:59, Nagy, Attila b...@fsn.hu wrote:

Hi,

On 05/04/15 10:59, Timo Sirainen wrote:

On 28 Apr 2015, at 23:49, Nagy, Attila b...@fsn.hu wrote:

Hi,

imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it 
works even with the dumbest IMAP server, altough it really kills performance, 
especially on high latency lines.

I wonder: if IMAP servers can effectively handle boundless fetches (like a list 
with all wanted UIDs, or simply 1:* if all are needed), do you see this as a 
good addition to develop?

Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. 
The higher the value, the more memory/disk space is used for storing the 
received mails.


I'm aware of that, but it doesn't, or at least not always.
For example Thunderbird issues this:
13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority 
X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])
Dovecot does this to the IMAP backend
18 UID FETCH 333574 (BODY.PEEK[])

Oh, there were several bugs related to that. Fixed:

http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b
I've just tried 2.2.18, which has the above modifications, but the 
effect is the same.

I have these in the config:
# doveconf -n | egrep 'prefe|imapc'
imapc_features = rfc822.size fetch-headers
imapc_host = 10.3.34.12
mail_location = imapc:~/imapc
mail_prefetch_count = 100

With an empty imapc directory a POP3 login yields these on the backend 
IMAP server:

2 LOGIN user pass
1 LIST  
3 SELECT INBOX
4 NOOP
5 UID FETCH 1:* (FLAGS)
6 UID FETCH 2 (RFC822.SIZE)
7 UID FETCH 3 (RFC822.SIZE)
8 UID FETCH 4 (RFC822.SIZE)
9 UID FETCH 5 (RFC822.SIZE)
10 UID FETCH 6 (RFC822.SIZE)
11 UID FETCH 7 (RFC822.SIZE)
12 UID FETCH 8 (RFC822.SIZE)
[...]
so on, for every message...

The response times for each of the FETCHes are around 13-16 ms, so 
logging into dovecot for the first time takes more than a minute with 
5000 mails in the INBOX and a half day with around a million...


It would be good to have a:
6 UID FETCH 1:* (RFC822.SIZE)
or even a:
5 UID FETCH 1:* (FLAGS RFC822.SIZE)
if needed (dovecot first fills up its cache, or other cases when the 
full list is needed)


If dovecot would issue the latter for this 5000-mail inbox, the first 
fetch (and POP login) would take only 500 ms instead of the 75 seconds 
that it takes now...

Could you please tweak this some more? :)

Thanks,

Re: imapc fetch optimization

2015-05-04 Thread Timo Sirainen 
On 28 Apr 2015, at 23:49, Nagy, Attila b...@fsn.hu wrote:
 
 Hi,
 
 imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it 
 works even with the dumbest IMAP server, altough it really kills performance, 
 especially on high latency lines.
 
 I wonder: if IMAP servers can effectively handle boundless fetches (like a 
 list with all wanted UIDs, or simply 1:* if all are needed), do you see this 
 as a good addition to develop?

Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. 
The higher the value, the more memory/disk space is used for storing the 
received mails.

Re: imapc fetch optimization

2015-05-04 Thread Nagy, Attila 

Hi,

On 05/04/15 10:59, Timo Sirainen wrote:

On 28 Apr 2015, at 23:49, Nagy, Attila b...@fsn.hu wrote:

Hi,

imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it 
works even with the dumbest IMAP server, altough it really kills performance, 
especially on high latency lines.

I wonder: if IMAP servers can effectively handle boundless fetches (like a list 
with all wanted UIDs, or simply 1:* if all are needed), do you see this as a 
good addition to develop?

Set mail_prefetch_count = 10 or 100 or something and it'll do larger FETCHes. 
The higher the value, the more memory/disk space is used for storing the 
received mails.


I'm aware of that, but it doesn't, or at least not always.
For example Thunderbird issues this:
13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority 
X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])

Dovecot does this to the IMAP backend
18 UID FETCH 333574 (BODY.PEEK[])
19 UID FETCH 333575 (BODY.PEEK[])
20 UID FETCH 333576 (BODY.PEEK[])
21 UID FETCH 333577 (BODY.PEEK[])
And so on.

I have mail_prefetch_count set to 100:
# doveconf -n | grep prefet
mail_prefetch_count = 100

Re: imapc fetch optimization

2015-05-04 Thread Timo Sirainen 
On 04 May 2015, at 16:59, Nagy, Attila b...@fsn.hu wrote:
 
 Hi,
 
 On 05/04/15 10:59, Timo Sirainen wrote:
 On 28 Apr 2015, at 23:49, Nagy, Attila b...@fsn.hu wrote:
 Hi,
 
 imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because it 
 works even with the dumbest IMAP server, altough it really kills 
 performance, especially on high latency lines.
 
 I wonder: if IMAP servers can effectively handle boundless fetches (like a 
 list with all wanted UIDs, or simply 1:* if all are needed), do you see 
 this as a good addition to develop?
 Set mail_prefetch_count = 10 or 100 or something and it'll do larger 
 FETCHes. The higher the value, the more memory/disk space is used for 
 storing the received mails.
 
 I'm aware of that, but it doesn't, or at least not always.
 For example Thunderbird issues this:
 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
 BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority 
 X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])
 Dovecot does this to the IMAP backend
 18 UID FETCH 333574 (BODY.PEEK[])

Oh, there were several bugs related to that. Fixed:

http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b

Re: imapc fetch optimization

2015-05-04 Thread Timo Sirainen 
On 04 May 2015, at 18:45, Timo Sirainen t...@iki.fi wrote:
 
 On 04 May 2015, at 16:59, Nagy, Attila b...@fsn.hu wrote:
 
 Hi,
 
 On 05/04/15 10:59, Timo Sirainen wrote:
 On 28 Apr 2015, at 23:49, Nagy, Attila b...@fsn.hu wrote:
 Hi,
 
 imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because 
 it works even with the dumbest IMAP server, altough it really kills 
 performance, especially on high latency lines.
 
 I wonder: if IMAP servers can effectively handle boundless fetches (like a 
 list with all wanted UIDs, or simply 1:* if all are needed), do you see 
 this as a good addition to develop?
 Set mail_prefetch_count = 10 or 100 or something and it'll do larger 
 FETCHes. The higher the value, the more memory/disk space is used for 
 storing the received mails.
 
 I'm aware of that, but it doesn't, or at least not always.
 For example Thunderbird issues this:
 13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
 BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority 
 X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])
 Dovecot does this to the IMAP backend
 18 UID FETCH 333574 (BODY.PEEK[])
 
 Oh, there were several bugs related to that. Fixed:
 
 http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
 http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
 http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b

Also if you want imapc to use the full set of features with latest Dovecot, use:

imapc_features=rfc822.size fetch-headers search

After that it shouldn't be fetching BODY.PEEK[] anymore unless the body was 
actually wanted to be fetched.

Re: imapc fetch optimization

2015-05-04 Thread Nagy, Attila 

On 05/04/15 17:45, Timo Sirainen wrote

I'm aware of that, but it doesn't, or at least not always.
For example Thunderbird issues this:
13 UID fetch 333574:333601,333630:333801 (UID RFC822.SIZE FLAGS 
BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority 
X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])
Dovecot does this to the IMAP backend
18 UID FETCH 333574 (BODY.PEEK[])

Oh, there were several bugs related to that. Fixed:

http://hg.dovecot.org/dovecot-2.2/rev/8f20aa806bcc
http://hg.dovecot.org/dovecot-2.2/rev/d350a23207c2
http://hg.dovecot.org/dovecot-2.2/rev/8c49fb6d789b

Thank you very much for the quick fix(es), I will try them.

Re: imapc fetch optimization

2015-05-04 Thread Nagy, Attila 

On 05/04/15 17:47, Timo Sirainen wrote:
Also if you want imapc to use the full set of features with latest 
Dovecot, use: imapc_features=rfc822.size fetch-headers search After 
that it shouldn't be fetching BODY.PEEK[] anymore unless the body was 
actually wanted to be fetched. 
Very nice, and also the path you follow is great: allow the backend IMAP 
server to be dumb and yet -with these feature flags- we can exploit it 
more if needed.


And having the ability to deliver mails through lmtpd to the IMAP 
backend is priceless. :)

imapc fetch optimization

2015-04-28 Thread Nagy, Attila 

Hi,

imapc does a lot of UID FETCH $UID (BODY.PEEK[]), which is nice, because 
it works even with the dumbest IMAP server, altough it really kills 
performance, especially on high latency lines.


I wonder: if IMAP servers can effectively handle boundless fetches (like 
a list with all wanted UIDs, or simply 1:* if all are needed), do you 
see this as a good addition to develop?


This could be a new imapc_features setting afterall...

Thanks,