Re: Bug reports - auth is broken in Dovecot 2.3.3

2018-10-02 Thread Aki Tuomi 


On 03.10.2018 08:41, Berindeie Avram-Teodor wrote:
> Hello,
> I have compiled Dovecot  2.3.3 from sources but I have problems with
> authentication.
> I tried to rebuild Dovecot 2.3.2.1 but the error persists.
> I attach a .txt file with the information requested to report a bug.
> My main mail server is fallen, (the same for the test server) I will
> send the report from a Gmail address.
> If you need additional information I will be at your disposal.
Hi!

This is a problem coming from new glibc, the fix unfortunately did not
make into 2.3.3. Please try applying the following patch

https://github.com/dovecot/core/commit/63a74b9e8e0604486a15a879e7f1a27257322400.patch

Aki

Bug reports - auth is broken in Dovecot 2.3.3

2018-10-02 Thread Berindeie Avram-Teodor 
Hello,
I have compiled Dovecot  2.3.3 from sources but I have problems with
authentication.
I tried to rebuild Dovecot 2.3.2.1 but the error persists.
I attach a .txt file with the information requested to report a bug.
My main mail server is fallen, (the same for the test server) I will send
the report from a Gmail address.
If you need additional information I will be at your disposal.
dovecot -n
# 2.3.3 (dcead646b): /etc/dovecot/dovecot.conf
# OS: Linux 4.14.73 x86_64 Slackware 14.2+
glibc-2.28-x86_64
gcc-8.2.0-x86_64
Filesystem: ext4

telnet localhost imap
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* BYE Disconnected: Auth process broken
Connection closed by foreign host.

cat /var/log/maillog
Oct  3 08:15:44 mail dovecot: auth: Fatal: master: service(auth): child 9586 
killed with signal 11 (core dumped)
Oct  3 08:15:44 mail dovecot: master: Error: service(auth): command startup 
failed, throttling for 60 secs
Oct  3 08:15:44 mail dovecot: imap-login: Disconnected: Auth process broken 
(disconnected before auth was ready, waited 0 secs): user=<>, rip=127.0.0.1, 
lip=127.0.0.1, secured, session=
Oct  3 08:15:44 mail dovecot: imap-login: Warning: Timeout leak: 0x7fba6c273870 
(auth-server-connection.c:397)

gdb /usr/local/libexec/dovecot/auth /tmp/cores/core.auth.9586.mail.1538543744
GNU gdb (GDB) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-slackware-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/libexec/dovecot/auth...done.
[New LWP 9586]
Core was generated by `dovecot/auth'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7f93c1ff4646 in __strcmp_ssse3 () from /lib64/libc.so.6
(gdb) bt full
#0  0x7f93c1ff4646 in __strcmp_ssse3 () from /lib64/libc.so.6
No symbol table info available.
#1  0x55574dbb58e8 in password_scheme_register_crypt () at 
password-scheme-crypt.c:190
i = 0
crypted = 
__func__ = 
#2  0x55574dbb543c in password_schemes_init () at password-scheme.c:874
i = 
#3  0x55574db91936 in main_preinit () at main.c:185
mod_set = {abi_version = 0x0, binary_name = 0x0, setting_name = 
0x7ffe57a5d998 "\253ݥW\376\177", filter_callback = 0x7ffe57a5d9a8,
  filter_context = 0x7ffe57a5d8d0, require_init_funcs = false, debug = 
false, ignore_dlopen_errors = true, ignore_missing = true}
services = 
mod_set = 
services = 
#4  main (argc=, argv=) at main.c:392
c =

Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Sami Ketola 


> On 2 Oct 2018, at 15.28, Jerry  wrote:
>> 
> 
> Nobody is going to stop using Outlook because some *.nix user has gotten his
> shorts up in a knot. Now, I did ask this question on a MS Outlook Tech Forum.
> One of the first responses I got was to ask exactly what Dovecot's response
> was to the attempt to create this folder; i.e., what error message or code
> was returned by Dovecot. I don't have the answer to that. If someone can
> supply me with the complete and accurate return code I will post that and see
> what transpires.
> 

A NO [CANNOT] Character not allowed in mailbox name: '/' (0.001 + 0.000 secs).

Sami

Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Helmut K. C. Tessarek 
On 2018-10-02 16:02, Timo Sirainen wrote:
> No, that's not how the listescape plugin works. You can't change the 
> filesystem separator with or without listescape plugin. It's always "." with 
> Maildir++.

This exactly my point. This fact is nowhere stated in the documentation.

I'm actually not talking about the plugin at all. (I'm very sorry that
I've replied to your listescape reply, my bad.)

The documentation only mentions that the separator can be changed by
setting the `separator` option. But it doesn't explain what it actually
does and why one wanted to do so in the first place.

e.g. Someone uses the default. Then changes it to something else. What
happens then? (you answered it already partly - clients might be confused).

But if the filesystem always used the . for Maildir, why would somebody
want to change the separator (without the plugin)?

Cheers,
  K. C.

-- 
regards Helmut K. C. Tessarek  KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/



signature.asc
Description: OpenPGP digital signature

Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Timo Sirainen 
On 2 Oct 2018, at 22.52, Helmut K. C. Tessarek  wrote:
> 
> On 2018-10-01 04:07, Timo Sirainen wrote:
>> 
>> https://wiki2.dovecot.org/Plugins/Listescape maybe?
> 
> It should be mentioned somehow that one can't just change the hierarchy
> separator on the fly (without manual changes to the fs).
> 
> If you used . as the separator, it would look this in the filesystem:
> .testfolder.sub1
> Now you change the separator to $.
> Your mail client will see the existing folders as .testfolder.sub1
> instead of:
> 
> testfolder
>   sub1

No, that's not how the listescape plugin works. You can't change the filesystem 
separator with or without listescape plugin. It's always "." with Maildir++. 
You can change the namespace's visible separator to e.g. "$" which allows you 
to start using "." in the folder names with listescape. But using "." will be 
encoded as \2e (or something) in the filesystem.

Changing the namespace separator can have other problems though. Clients 
generally don't like it much, and may redownload all mails in subfolders if 
it's changed. Some clients may become even more confused.



signature.asc
Description: Message signed with OpenPGP

Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Helmut K. C. Tessarek 
On 2018-10-01 04:07, Timo Sirainen wrote:
> 
> https://wiki2.dovecot.org/Plugins/Listescape maybe?

It should be mentioned somehow that one can't just change the hierarchy
separator on the fly (without manual changes to the fs).

If you used . as the separator, it would look this in the filesystem:
.testfolder.sub1
Now you change the separator to $.
Your mail client will see the existing folders as .testfolder.sub1
instead of:

testfolder
   sub1

Cheers,
  K. C.

-- 
regards Helmut K. C. Tessarek  KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/



signature.asc
Description: OpenPGP digital signature

Re: Need to convert mbox to Maildir

2018-10-02 Thread Mark Foley 
On Tue, 2 Oct 2018 21:17:20 +0300 Sami Ketola  wrote:
>
> > On 2 Oct 2018, at 21.05, Mark Foley  wrote:
> > 
> > I have a mbox file of emails.  I want to convert this to Maildir giving me 
> > individual message
> > files per email.  I've looked at dsync, but as far as I can tell this wants 
> > a specific target
> > user and it appears that it will "distribute" the converted messages into 
> > that user's INBOX. 
> > 
> > I don't want to put these mbox messages into any particular user's Maildir 
> > hierarchy, just
> > export to file-per-message format to a destination directory of my 
> > choosing. 
> > 
> > Is this possible?
>
>
> Yes. 
>
> [root@ketola /]# mkdir /test
> [root@ketola /]# chown vmail /test
> [root@ketola /]# doveadm backup -u sami Maildir:/test/
>
> done.
>
> Sami

Excellent! Thank you. I'll give that a try. I also found: mb2md.pl downloadable 
from
https://wiki.dovecot.org/Migration/MailFormat. 

--Mark

Re: Need to convert mbox to Maildir

2018-10-02 Thread Sami Ketola 


> On 2 Oct 2018, at 21.05, Mark Foley  wrote:
> 
> I have a mbox file of emails.  I want to convert this to Maildir giving me 
> individual message
> files per email.  I've looked at dsync, but as far as I can tell this wants a 
> specific target
> user and it appears that it will "distribute" the converted messages into 
> that user's INBOX. 
> 
> I don't want to put these mbox messages into any particular user's Maildir 
> hierarchy, just
> export to file-per-message format to a destination directory of my choosing. 
> 
> Is this possible?


Yes. 

[root@ketola /]# mkdir /test
[root@ketola /]# chown vmail /test
[root@ketola /]# doveadm backup -u sami Maildir:/test/

done.

Sami

Need to convert mbox to Maildir

2018-10-02 Thread Mark Foley 
I have a mbox file of emails.  I want to convert this to Maildir giving me 
individual message
files per email.  I've looked at dsync, but as far as I can tell this wants a 
specific target
user and it appears that it will "distribute" the converted messages into that 
user's INBOX. 

I don't want to put these mbox messages into any particular user's Maildir 
hierarchy, just
export to file-per-message format to a destination directory of my choosing. 

Is this possible?

THX --Mark

Delete vs. Expunge in Public Namespace

2018-10-02 Thread Chris 
All,

my IMAP client [1] marks messages as deleted, but doesn't expunge.
Dovecot (2.2.27-3+deb9u2) expunges them. Is this because the mailbox is
part of public namespace?


::1.38247-::1.00143: MALP6 STORE
48,49,50,51,52,53,54,55,56,57,58,59,60,61 +FLAGS (\Deleted)

::1.00143-::1.38247: * 48 FETCH (FLAGS (\Deleted \Seen))
* 49 FETCH (FLAGS (\Deleted \Seen))
* 50 FETCH (FLAGS (\Deleted \Seen))
* 51 FETCH (FLAGS (\Deleted \Seen))
* 52 FETCH (FLAGS (\Deleted \Seen))
* 53 FETCH (FLAGS (\Deleted \Seen))
* 54 FETCH (FLAGS (\Deleted \Seen))
* 55 FETCH (FLAGS (\Deleted \Seen))
* 56 FETCH (FLAGS (\Deleted \Seen))
* 57 FETCH (FLAGS (\Deleted \Seen))
* 58 FETCH (FLAGS (\Deleted \Seen))
* 59 FETCH (FLAGS (\Deleted \Seen))
* 60 FETCH (FLAGS (\Deleted \Seen))
* 61 FETCH (FLAGS (\Deleted \Seen))
MALP6 OK Store completed.

::1.38247-::1.00143: MALP7 SEARCH DELETED



Oct  2 10:18:03 hyptest dovecot: imap(chris): delete:
box=MailingListen/test125, uid=180,
msgid=<740a9ba3-d637-3fa0-1b87-ff8d79491bf0@ hyp.postbox.xyz>,
size=1509 Oct  2 10:18:03 hyptest dovecot: imap(chris): expunge:
box=MailingListen/test125, uid=167,
msgid=<740a9ba3-d637-3fa0-1b87-ff8d79491bf0 @hyp.postbox.xyz>, size=1509


namespace {
  list = yes
  location =
maildir:/var/vmail/public/:LAYOUT=fs:INDEXPVT=%h/.public-indexes prefix
= MailingListen/ separator = /
  subscriptions = no
  type = public
}


- Chris






[1]
https://raw.githubusercontent.com/quentinsf/IMAPdedup/master/imapdedup.py

Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Jerry 
>On Tue, Oct 2, 2018 at 09:59, Wojciech Puchar  wrote:
>>>
>>> As I have no control over their minds, hands, and client software, I wish
>>> I could enforce the policy from the server, returning an error message to
>>> the client.
>>>
>>> On its turn, this requires the client to listen to such server messages,
>>> operated by a smarter user.
>>>
>>> At the end of the day, it feels like we are re-discovering the wheel, as
>>> such problems should have been addressed and solved long ago by an RFC.
>>>
>>> Listescape is a welcome patch. Let see if it works. I just have to select
>>> a character that no user could type and still practical for the
>>> filesystem to use...
>>>
>>>
>>> On Mon, Oct 1, 2018 at 10:07, Timo Sirainen  wrote:
>>> On 28 Sep 2018, at 16.44, Wojciech Puchar  wrote:
>>>
>>> user attempts to create folders with / dovecot naturally cannot create it
>>> so it returns error but outlook of course "create" it and keep data in
>>> local store only. data is lost when you remove local store .pst file.
>>>
>>> The question is - can dovecot be configured so it will automatically
>>> replace slash in name with something else?
>>>
>>>
>>> https://wiki2.dovecot.org/Plugins/Listescape maybe?

>>
>> the problem with pseudomail pseudoprogram outlook is that it simply
>> ignores error and shows folder created, then even allow to store messages
>> in it (store it locally in temporary file).
>>
>> When file is deleted messages are lost.
>>

>On Tue, 02 Oct 2018 11:29:40 +, Rupert Gallagher stated:

>I think we need a public compliance test, similar to html and ssl, then
>people would start questioning the quality of their own client, and migrate
>to better ones. When Micro$oft will eventually feel the pinch, then they
>will start fixing their $hit. People have the power! (I like that song.)

Nobody is going to stop using Outlook because some *.nix user has gotten his
shorts up in a knot. Now, I did ask this question on a MS Outlook Tech Forum.
One of the first responses I got was to ask exactly what Dovecot's response
was to the attempt to create this folder; i.e., what error message or code
was returned by Dovecot. I don't have the answer to that. If someone can
supply me with the complete and accurate return code I will post that and see
what transpires.

By the way, could we please do away with the "TOP POSTING" on this thread. It
seems rather counter productive to be bitching about the behavior of MS
Outlook while a poster posts in an unsatisfactory manner.

-- 
Jerry

Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Rupert Gallagher 
I think we need a public compliance test, similar to html and ssl, then people 
would start questioning the quality of their own client, and migrate to better 
ones. When Micro$oft will eventually feel the pinch, then they will start 
fixing their $hit. People have the power! (I like that song.)

On Tue, Oct 2, 2018 at 09:59, Wojciech Puchar  wrote:

>>
>> As I have no control over their minds, hands, and client software, I wish I 
>> could enforce the policy from the server, returning an
>> error message to the client.
>>
>> On its turn, this requires the client to listen to such server messages, 
>> operated by a smarter user.
>
> the problem with pseudomail pseudoprogram outlook is that it simply
> ignores error and shows folder created, then even allow to store messages
> in it (store it locally in temporary file).
>
> When file is deleted messages are lost.
>
>>
>> At the end of the day, it feels like we are re-discovering the wheel, as 
>> such problems should have been addressed and solved long
>> ago by an RFC.
>>
>> Listescape is a welcome patch. Let see if it works. I just have to select a 
>> character that no user could type and still practical
>> for the filesystem to use...
>>
>>
>> On Mon, Oct 1, 2018 at 10:07, Timo Sirainen  wrote:
>> On 28 Sep 2018, at 16.44, Wojciech Puchar  wrote:
>>
>> user attempts to create folders with / dovecot naturally cannot create it so 
>> it returns error but outlook of
>> course "create" it and keep data in local store only. data is lost when you 
>> remove local store .pst file.
>>
>> The question is - can dovecot be configured so it will automatically replace 
>> slash in name with something
>> else?
>>
>>
>> https://wiki2.dovecot.org/Plugins/Listescape maybe?
>>
>>
>>

Re: Help for UID THREAD and SORT optimization

2018-10-02 Thread Alessio Cecchi 

Il 02/10/2018 12:08, Timo Sirainen ha scritto:

On 2 Oct 2018, at 12.22, Alessio Cecchi > wrote:


Hello,

we are developing a library to show last arrived messages of all 
threads in a folder with more than 300k messages.


As per the RFC 5256, the IMAP thread command has only the option to 
specify the grouping criteria (REFERENCES vs ORDEREDSUBJECT). So we 
implemented an algorithm which gets the full UID SORT ordered by date 
in reverse order then gets all thread relations and then post process 
all outputs to obtain the final result.


It sounds like what you want is REFS threading, which Dovecot 
supports, but it didn't make it into official RFC: 
https://tools.ietf.org/html/draft-gulbrandsen-imap-inthread-05



Thanks Timo,

you are the best!

p.s. was a pleasure to meet you and the OX team in Rome last week

--
Alessio Cecchi
Postmaster @ http://www.qboxmail.it
https://www.linkedin.com/in/alessice

Re: Help for UID THREAD and SORT optimization

2018-10-02 Thread Timo Sirainen 
On 2 Oct 2018, at 12.22, Alessio Cecchi  wrote:
> 
> Hello,
> 
> we are developing a library to show last arrived messages of all threads in a 
> folder with more than 300k messages.
> 
> As per the RFC 5256, the IMAP thread command has only the option to specify 
> the grouping criteria (REFERENCES vs ORDEREDSUBJECT). So we implemented an 
> algorithm which gets the full UID SORT ordered by date in reverse order then 
> gets all thread relations and then post process all outputs to obtain the 
> final result.
> 
It sounds like what you want is REFS threading, which Dovecot supports, but it 
didn't make it into official RFC: 
https://tools.ietf.org/html/draft-gulbrandsen-imap-inthread-05

Re: 2.3.3: Panic: file ostream-zlib.c: line 37 (o_stream_zlib_close): assertion failed

2018-10-02 Thread Timo Sirainen 
On 2 Oct 2018, at 12.26, Tom Sommer  wrote:
> 
> I see this in my logs after 2.3.3:
> 
> using zlib plugin, ofc.
> 
> Oct 02 10:01:39 imap(u...@example.com)<50643>: Panic: file 
> ostream-zlib.c: line 37 (o_stream_zlib_close): assertion failed: 
> (zstream->ostream.finished || zstream->ostream.ostream.stream_errno != 0 || 
> zstream->ostream.error_handling_disabled)
> Oct 02 10:01:39 imap(u...@example.com)<50643>: Error: Raw 
> backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xce56a) [0x7f442487556a] -> 
> /usr/lib64/dovecot/libdovecot.so.0(+0xce5b1) [0x7f44248755b1] -> 
> /usr/lib64/dovecot/libdovecot.so.0(+0x3d941) [0x7f44247e4941] -> 
> /usr/lib64/dovecot/lib20_zlib_plugin.so(+0x5cdf) [0x7f44233c4cdf] -> 
> /usr/lib64/dovecot/libdovecot.so.0(+0xf4b46) [0x7f442489bb46] -> 
> /usr/lib64/dovecot/libdovecot.so.0(o_stream_destroy+0x13) [0x7f442489be83] -> 
> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_save_finish+0x173) 
> [0x7f4424b93973] -> 
> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_save_cancel+0x36) 
> [0x7f4424b68696] -> dovecot/imap [u...@example.com X.X.X.X APPEND](+0xede9) 
> [0x55a9d7e86de9] -> dovecot/imap [u...@example.com X.X.X.X 
> APPEND](command_exec+0x65) [0x55a9d7e956e5] -> dovecot/imap [u...@example.com 
> X.X.X.X APPEND](+0xf9e6) [0x55a9d7e879e6] -> 
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x55) [0x7f442488c275] -> 
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbf) 
> [0x7f442488e13f] -> 
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x55) [0x7f442488c365] 
> -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f442488c588] -> 
> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4424808053] 
> -> dovecot/imap [u...@example.com X.X.X.X APPEND](main+0x32d) 
> [0x55a9d7ea20dd] -> /lib64/libc.so.6(__libc_start_main+0x100) 
> [0x7f4424431d20] -> dovecot/imap [u...@example.com X.X.X.X APPEND](+0xe419) 
> [0x55a9d7e86419]
> 
> No clue how to reproduce

It crashes if client disconnects during APPEND, but only with Maildir and zlib 
plugin. The bug already existed in earlier v2.3.x releases though.

2.3.3: Panic: file ostream-zlib.c: line 37 (o_stream_zlib_close): assertion failed

2018-10-02 Thread Tom Sommer 

I see this in my logs after 2.3.3:

using zlib plugin, ofc.

Oct 02 10:01:39 imap(u...@example.com)<50643>: Panic: 
file ostream-zlib.c: line 37 (o_stream_zlib_close): assertion failed: 
(zstream->ostream.finished || zstream->ostream.ostream.stream_errno != 0 
|| zstream->ostream.error_handling_disabled)
Oct 02 10:01:39 imap(u...@example.com)<50643>: Error: 
Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xce56a) 
[0x7f442487556a] -> /usr/lib64/dovecot/libdovecot.so.0(+0xce5b1) 
[0x7f44248755b1] -> /usr/lib64/dovecot/libdovecot.so.0(+0x3d941) 
[0x7f44247e4941] -> /usr/lib64/dovecot/lib20_zlib_plugin.so(+0x5cdf) 
[0x7f44233c4cdf] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf4b46) 
[0x7f442489bb46] -> 
/usr/lib64/dovecot/libdovecot.so.0(o_stream_destroy+0x13) 
[0x7f442489be83] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(maildir_save_finish+0x173) 
[0x7f4424b93973] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_save_cancel+0x36) 
[0x7f4424b68696] -> dovecot/imap [u...@example.com X.X.X.X 
APPEND](+0xede9) [0x55a9d7e86de9] -> dovecot/imap [u...@example.com 
X.X.X.X APPEND](command_exec+0x65) [0x55a9d7e956e5] -> dovecot/imap 
[u...@example.com X.X.X.X APPEND](+0xf9e6) [0x55a9d7e879e6] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x55) 
[0x7f442488c275] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbf) 
[0x7f442488e13f] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x55) 
[0x7f442488c365] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7f442488c588] -> 
/usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f4424808053] -> dovecot/imap [u...@example.com X.X.X.X 
APPEND](main+0x32d) [0x55a9d7ea20dd] -> 
/lib64/libc.so.6(__libc_start_main+0x100) [0x7f4424431d20] -> 
dovecot/imap [u...@example.com X.X.X.X APPEND](+0xe419) [0x55a9d7e86419]


No clue how to reproduce


--
Tom

Help for UID THREAD and SORT optimization

2018-10-02 Thread Alessio Cecchi 

Hello,

we are developing a library to show last arrived messages of all threads 
in a folder with more than 300k messages.


As per the RFC 5256, the IMAP thread command has only the option to 
specify the grouping criteria (REFERENCES vs ORDEREDSUBJECT). So we 
implemented an algorithm which gets the full UID SORT ordered by date in 
reverse order then gets all thread relations and then post process all 
outputs to obtain the final result.
Once we have both output we loop through the first uids set (which is 
ordered) and for every item of it we loop through the other uids set 
with all the thread relations. If the uid of the first set is contained 
in one of the elements of the thread messages, we pick it up and push it 
in the resulting array.

This is a pseudo-code that shows what we are doing:

array_of_latest_messages_in_thread = []
array_of_sorted_uids = [n,n,n,n,...] // UID SORT (REVERSE DATE) UTF-8 ALL
array_of_thread_uids = [n,[n,n],n,[n,n,n],[n,n],n,n,...] // UID THREAD 
(REFERENCES) UTF-8 ALL

foreach(array_of_sorted_uids as s_uid){
    foreach(array_of_thread_uids as t_uids){
    if(t_uids contains s_uid){ // or a function to loop recursively 
t_uids to search if a leaf is equal to s_uid

    array_of_latest_messages_in_thread.push(s_uid)
    break
    }
    }
}

We have also made some little optimizations in the above code like for 
example skipping from the outer loop the uids already processed in the 
inner loop that have not being selected. Anyway, the loop is very 
expensive in term of computation and we are wondering if there is a 
better approach to this issue directly using IMAP instead of post 
processing the output of the two commands at application level.


Thanks.

--
Alessio Cecchi
Postmaster @ http://www.qboxmail.it
https://www.linkedin.com/in/alessice

Re: Bug in conditionals to assign values to variables?

2018-10-02 Thread Angel L. Mateo 

Sorry for my last mail. I didn't save the file before reloading dovecot.

passdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}

userdb {
  driver = prefetch
}
userdb {
  args = /etc/dovecot/dovecot-ldap-userdb.conf.ext
  driver = ldap
}

auth_debug =yes
auth_verbose =yes


In dovecot-ldap-userdb.conf.ext I have:

user_attrs = 
=user=%{ldap:mail},=quota_rule=%{ldap:quota:*:storage=10G},=home_prefix=/mail/mailboxes/%L{domain},=index_prefix=/mail/indexes/%L{domain},=relpath=%{if;%{username};~;^.+;%2L{username}/%L{username};other_value}


In dovecot-ldap.conf.ext I have:
pass_attrs = 
=userdb_user=%{ldap:mail},=userdb_quota_rule=%{ldap:quota:*:storage=10G},=userdb_home_prefix=/mail/mailboxes/%L{domain},=userdb_index_prefix=/mail/indexes/%L{domain},=userdb_relpath=%{if;%{username};~;^.+;%2L{username}/%L{username};other_value}


And this the log I have for a "doveadm user angel.l...@um.es" command:

Oct  2 09:59:49 myotis60 dovecot: auth: Debug: Loading modules from 
directory: /usr/lib/dovecot/modules/auth
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: Module loaded: 
/usr/lib/dovecot/modules/auth/lib20_auth_var_expand_crypt.so
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: Loading modules from 
directory: /usr/lib/dovecot/modules/auth
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: Module loaded: 
/usr/lib/dovecot/modules/auth/libauthdb_ldap.so
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: Read auth token secret 
from /var/run/dovecot/auth-token-secret.dat
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: master in: 
USER#0111#011angel.l...@um.es#011service=doveadm#011debug
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: 
prefetch(angel.l...@um.es): passdb didn't return userdb entries, trying 
the next userdb
Oct  2 09:59:49 myotis60 dovecot: auth: Error: 
var_expand_long(if;%{username}) failed: if: requires four or five 
parameters, got 1
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: ldap(angel.l...@um.es): 
userdb cache miss
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: ldap(angel.l...@um.es): 
user search: base=ou=Usuarios,dc=Telematica scope=subtree 
filter=(&(objectClass=CourierMailAccount)(irisUserStatus=urn:mace:rediris.es:um.es:userstatus:correo:estado:activo)(irisMailhost=popimapclu11.um.es)(|(uid=angel.l...@um.es)(mail=angel.l...@um.es))) 
fields=mail,quota
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: ldap(angel.l...@um.es): 
result: mail=angel.l...@um.es quota=*:storage=10G; mail,quota unused
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: ldap(angel.l...@um.es): 
result: mail=angel.l...@um.es quota=*:storage=10G
Oct  2 09:59:49 myotis60 dovecot: auth: Error: 
var_expand_long(if;%{username}) failed: if: requires four or five 
parameters, got 1
Oct  2 09:59:49 myotis60 dovecot: auth: Debug: userdb out: 
USER#0111#011angel.l...@um.es#011quota_rule=*:storage=10G#011home_prefix=/mail/mailboxes/um.es#011index_prefix=/mail/indexes/um.es#011relpath=an/angel.luis


--
Angel L. Mateo Martínez
Sección de Telemática
Área de Tecnologías de la Información
y las Comunicaciones Aplicadas (ATICA)
http://www.um.es/atica
Tfo: 868889150
Fax: 86337

Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Wojciech Puchar 


As I have no control over their minds, hands, and client software, I wish I 
could enforce the policy from the server, returning an
error message to the client. 

On its turn, this requires the client to listen to such server messages, 
operated by a smarter user. 


the problem with pseudomail pseudoprogram outlook is that it simply 
ignores error and shows folder created, then even allow to store messages 
in it (store it locally in temporary file).


When file is deleted messages are lost.




At the end of the day, it feels like we are re-discovering the wheel, as such 
problems should have been addressed and solved long
ago by an RFC. 

Listescape is a welcome patch. Let see if it works. I just have to select a 
character that no user could type and still practical
for the filesystem to use...


On Mon, Oct 1, 2018 at 10:07, Timo Sirainen  wrote:
  On 28 Sep 2018, at 16.44, Wojciech Puchar  wrote:

user attempts to create folders with / dovecot naturally cannot 
create it so it returns error but outlook of
course "create" it and keep data in local store only. data is lost 
when you remove local store .pst file.

The question is - can dovecot be configured so it will 
automatically replace slash in name with something
else?


https://wiki2.dovecot.org/Plugins/Listescape maybe?

Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Wojciech Puchar 

this seems to form properly.

thank you very much

On Mon, 1 Oct 2018, Timo Sirainen wrote:


On 28 Sep 2018, at 16.44, Wojciech Puchar  wrote:

  user attempts to create folders with / dovecot naturally cannot create it 
so it returns error but outlook of course
  "create" it and keep data in local store only. data is lost when you 
remove local store .pst file.

  The question is - can dovecot be configured so it will automatically 
replace slash in name with something else?


https://wiki2.dovecot.org/Plugins/Listescape maybe?

Re: Bug in conditionals to assign values to variables?

2018-10-02 Thread Angel L. Mateo 


I can't seem to be able to reproduce this, not even with v2.2.33. It simply 
works without logging any errors.


I've made a lot of changes now. I can't neither reproduce it anymore 
:-(.

If I could, I'll send the complete configuration.

--
Angel L. Mateo Martínez
Sección de Telemática
Área de Tecnologías de la Información
y las Comunicaciones Aplicadas (ATICA)
http://www.um.es/atica
Tfo: 868889150
Fax: 86337

Re: outlook idiocy - IMAP folders with /

2018-10-02 Thread Wojciech Puchar 


The question is - can dovecot be configured so it will automatically
replace slash in name with something else?


And how would this help? Outlook would still go ahead and create the folder 
with / in it, and dovecot would create a folder with, say - in it and now you'd 
have two folders.

Aki



right