Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot
for instance, if I do a search from roundcube, the inbo name is NOT passed to the backend (which is normal) the same search from the command line add the mailbox name ADDITIONALLY to the mailbox * pointer However, passing a search from roudcube ask TWICE the backend (first with AND flag,

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot
Timo, A little of logic here : 1 - the mailbox is passed by dovecot to the backend as a mailbox * pointer , NOT as a search parameter. -> It works properly when entering a search from roundcube or evolution for instance. -> therefore this is a clear bug of the command line 2 - the

Re: FTS delays

2019-04-21 Thread Timo Sirainen via dovecot
It's because you're misunderstanding how the lookup() function works. It gets ALL the search parameters, including the "mailbox inbox". This is intentional, and not a bug. Two reasons being: 1) The FTS plugin in theory could support indexing/searching any kinds of searches, not just regular

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot
No, the parsing is made by dovecot core, that is nothing the backend can do about it. The backend shall *never* reveive this. (would it be buggy or no) PLease, have a look deeper And the loop is a very big problem as it times out all the time (and once again, this is not in any of the

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot
Antoher example so you understand how may understand the bug in dovecote core : # doveadm search -u j...@grosjo.net mailbox SENT text milan doveadm(j...@grosjo.net): Info: Get last UID of Sent = 61707 -> CORRECTLY ASSIGNED THE PROPER MAILBOX TO THE BACK END doveadm(j...@grosjo.net): Info:

Re: FTS delays

2019-04-21 Thread Timo Sirainen via dovecot
Inbox appears in the list of arguments, because fts_backend_xapian_lookup() is parsing the search args wrong. Not sure about the other issue. > On 21 Apr 2019, at 19.31, Joan Moreau wrote: > > For this first point, the problem is that dovecot core sends TWICE the > request and "Inbox" appears

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot
For this first point, the problem is that dovecot core sends TWICE the request and "Inbox" appears in the list of arguments ! (inbox shall serve to select teh right mailbox, never sent to the backend) And even if this would be solved, the dovecot core loops *after* the backend hs returneds the

Re: FTS delays

2019-04-21 Thread Timo Sirainen via dovecot
On 3 Apr 2019, at 20.30, Joan Moreau via dovecot wrote: > doveadm search -u j...@grosjo.net mailbox inbox text milan > output > > doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox > OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) > AND

Re: FTS delays

2019-04-20 Thread Joan Moreau via dovecot
I have no idea how to use git-bitsec On 2019-04-15 15:31, Josef 'Jeff' Sipek wrote: On Sun, Apr 14, 2019 at 21:09:54 +0800, Joan Moreau wrote: ... THe "loop" part seems the most urgent : It breaks everything (search timeout 100% of the time) Any luck with git-bisect? Jeff. On

Re: FTS delays

2019-04-15 Thread Josef 'Jeff' Sipek via dovecot
On Sun, Apr 14, 2019 at 21:09:54 +0800, Joan Moreau wrote: ... > THe "loop" part seems the most urgent : It breaks everything (search > timeout 100% of the time) Any luck with git-bisect? Jeff. > > On 2019-04-06 09:56, Joan Moreau via dovecot wrote: > > > For the point 1, this is not

Re: FTS delays

2019-04-14 Thread Joan Moreau via dovecot
I have tried to spend some time of understanding the logic (if any !) of the fts part Honestly, the one who created this mess shall be the one to fix it, or one shall refactor it totally. Basically, the fts "core" should be able to do - select the backend according to conf file - send

Re: FTS delays

2019-04-05 Thread Joan Moreau via dovecot
For the point 1, this is not "suboptimal", it is plain wrong (results are damn wrong ! and this is not related to the backend, but the FTS logic in Dovecot core) For the point 2 , this has been discussed already numerous times but without action. The dovecot core shall be the one re-submitting

Re: FTS delays

2019-04-05 Thread Josef 'Jeff' Sipek via dovecot
On Fri, Apr 05, 2019 at 19:33:57 +0800, Joan Moreau via dovecot wrote: > Hi > > If you plan to fix the FTS part of Dovecot, I will be very gratefull. I'm trying to figure out what is causing the 3rd issue you listed, so we can decide how severe it is and therefore how quickly it needs to be

Re: FTS delays

2019-04-05 Thread Joan Moreau via dovecot
Hi If you plan to fix the FTS part of Dovecot, I will be very gratefull. Not sure this is related to any specific commit but rahter the overall design The list of bugs so far 1 - Double call to fts plugins with inconsistent parameter (first call diferent from second call for the same

Re: FTS delays

2019-04-05 Thread Josef 'Jeff' Sipek via dovecot
On Fri, Apr 05, 2019 at 17:45:36 +0800, Joan Moreau wrote: > I am on master (very latest) > > No clue exactly when this problem appears, but > > 1 - the "request twice the fts plugin instead of once" issue has always > been there (since my first RC release of fts-xapian) Ok, good to know. >

Re: FTS delays

2019-04-05 Thread Joan Moreau via dovecot
I am on master (very latest) No clue exactly when this problem appears, but 1 - the "request twice the fts plugin instead of once" issue has always been there (since my first RC release of fts-xapian) 2 - the body/text loop has appeared recently (maybe during the month of March) On

Re: FTS delays

2019-04-05 Thread Josef 'Jeff' Sipek via dovecot
On Wed, Apr 03, 2019 at 19:02:52 +0800, Joan Moreau via dovecot wrote: > issue seems in the Git version : Which git revision? Before you updated to the broken revision, which revision/version were you running? Can you try it with 5f6e39c50ec79ba8847b2fdb571a9152c71cd1b6 (the commit just before

Re: FTS delays

2019-04-03 Thread Joan Moreau via dovecot
issue seems in the Git version : FTS search in teh body ends up with looping Other search call twice the FTS plugin (for no reason) On 2019-04-03 18:58, @lbutlr via dovecot wrote: On 3 Apr 2019, at 04:30, Joan Moreau via dovecot wrote: doveadm search -u j...@grosjo.net mailbox inbox

Re: FTS delays

2019-04-03 Thread @lbutlr via dovecot
On 3 Apr 2019, at 04:30, Joan Moreau via dovecot wrote: > doveadm search -u j...@grosjo.net mailbox inbox text milan Did that search over my list mail and got 83 results, not able to duplicate your issue. What version of dovecot and have you tried to reindex? dovecot-2.3.5.1 here. -- There

Re: FTS delays

2019-04-03 Thread Joan Moreau via dovecot
Example from real life From Roubdcube, I serach "milan" in full message (body & headers) Logs : Apr 3 10:24:01 gjserver dovecot[29778]: imap(j...@grosjo.net)<30311><4pACp52FfCF/AAAB>: Query : ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR subject:milan OR

Re: FTS delays

2019-04-02 Thread Timo Sirainen via dovecot
On 2 Apr 2019, at 6.38, Joan Moreau via dovecot wrote: > > Further on this topic: > > > > When choosing any headers in the search box, dovecot core calls the plugin > TWICE (and returns the results quickly, but not immediatly after getting the > IDs from the plugins) > > When choosing the

Re: FTS delays

2019-04-01 Thread Joan Moreau via dovecot
Further on this topic: When choosing any headers in the search box, dovecot core calls the plugin TWICE (and returns the results quickly, but not immediatly after getting the IDs from the plugins) When choosing the BODY search, dovecot core calls the plugin ONCE (and never returns) (whereas

Re: FTS delays

2019-03-30 Thread Joan Moreau via dovecot
it is already on On March 31, 2019 03:47:52 Aki Tuomi via dovecot wrote: On 30 March 2019 21:37 Joan Moreau via dovecot wrote: Hi When I do a FTS search (using Xapian plugin) in the BODY part, the plugins returns the matching IDs within few milliseconds (as seen in the log).

Re: FTS delays

2019-03-30 Thread Aki Tuomi via dovecot
On 30 March 2019 21:37 Joan Moreau via dovecot wrote: Hi When I do a FTS search (using Xapian plugin) in the BODY part, the plugins returns the matching IDs within few milliseconds (as seen in the log). However, roundcube

FTS delays

2019-03-30 Thread Joan Moreau via dovecot
Hi When I do a FTS search (using Xapian plugin) in the BODY part, the plugins returns the matching IDs within few milliseconds (as seen in the log). However, roundcube (connected on dovecot) takes ages to show (headers only vie IMAP) the few results (I tested with a matching requests of 9