Re: Indexing Mail faster

2015-02-12 Thread Kevin Laurie
Hi Micheal, Here is how it should look:- ~# telnet localhost imap Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. . login u...@domain.net password . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS

Re: Indexing Mail faster

2015-01-29 Thread Kevin Laurie
Dear Marc, Noted. This is starting to make a lot of sense. Of course, more computing power, hence more performance. Also filesystems and such all have and effect on the outcome. I might experiment with adding more computing power and seeing the result. My mailbox does not have many users so I

Re: Indexing Mail faster

2015-01-29 Thread Kevin Laurie
Dear Joseph, Just to clarify, the pid you are referring to is of dovecot? Correct? Thanks Kevin On Thu, Jan 29, 2015 at 3:34 PM, Joseph Tam jtam.h...@gmail.com wrote: Kevin writes: Appreciate if you could help with this. I have been trying to address this slow search issue for a while with

Re: Indexing Mail faster

2015-01-29 Thread Kevin Laurie
Yes. I was missing due to repositories. Added more sources and now its fine. Thanks! Kevin On Thu, Jan 29, 2015 at 4:09 PM, Kevin Laurie superinterstel...@gmail.com wrote: Hi Thom, No results. I think I am missing some repositories. I might add some from ubuntu. On Thu, Jan 29, 2015 at 4:02

Re: Indexing Mail faster

2015-01-29 Thread Marc Stürmer
Am 27.01.2015 um 03:24 schrieb Kevin Laurie: Hi Thomas, That's very interesting. Below is my search result. Any idea why is my result so slow:- Appreciate if you could advise. It depends on a couple of facts. Full text searches must go through a number of bottle necks, depending on your

Re: Indexing Mail faster

2015-01-29 Thread Kevin Laurie
Dear Thomas, You do seem to have some computing power. I think that does help. Thanks Kevin A. On Thu, Jan 29, 2015 at 4:20 PM, Thomas Leuxner t...@leuxner.net wrote: * Peter Hodur peteho...@gmail.com 2015.01.29 05:25: With a current version of Dovecot a search is pretty fast _without_

Re: Indexing Mail faster

2015-01-29 Thread Thomas Leuxner
* Peter Hodur peteho...@gmail.com 2015.01.29 05:25: With a current version of Dovecot a search is pretty fast _without_ using external indexes. I have a view defined (virtual plugin) with around 22.000 messages in it, and searching the full view only takes 2.5 seconds: hmmm, could you

Re: Indexing Mail faster

2015-01-29 Thread Kevin Laurie
Hi Thom, No results. I think I am missing some repositories. I might add some from ubuntu. On Thu, Jan 29, 2015 at 4:02 PM, Thom Miller t...@cagroups.com wrote: On Thu, 2015-01-29 at 15:56 +0800, Kevin Laurie wrote: Hey Joseph Thanks. Strace seems like a very useful tool Only problem is

Re: Indexing Mail faster

2015-01-29 Thread Thom Miller
On Thu, 2015-01-29 at 15:56 +0800, Kevin Laurie wrote: Hey Joseph Thanks. Strace seems like a very useful tool Only problem is that I dont think it is maintained on ubuntu. Tried to run:- apt-get install strace but could not download it. Might need to download and build it. Do you know any

Re: Indexing Mail faster

2015-01-29 Thread Joseph Tam
On Thu, 29 Jan 2015, Kevin Laurie wrote: Just to clarify, the pid you are referring to is of dovecot? Yes, the dovecot/imap worker process. If there is no system call that is using up a lot of time, but the cumulative open/read time for going through many files in your Maildir, then you have

Re: Indexing Mail faster

2015-01-28 Thread superinterstellar
‎Dear Mr.Thomas, Thanks for your feedback. Will try dovecot-ee.  Also could you advise which dovecot are you using? Thanks Kevin   Original Message   From: Thomas Leuxner Sent: Wednesday, 28 January 2015 08:45 To: dovecot@dovecot.org Reply To: dovecot@dovecot.org Subject: Re: Indexing Mail faster

Re: Indexing Mail faster

2015-01-28 Thread Kevin Laurie
Dear Thomas, Could you advise(or provide a link to) for a safe way to upgrade the Dovecot version. I am a little worried about testing and making trial-and-error approach as it could be risky. Appreciate if you could help with this. I have been trying to address this slow search issue for a while

Re: Indexing Mail faster

2015-01-28 Thread Thomas Leuxner
* superinterstel...@gmail.com superinterstel...@gmail.com 2015.01.28 12:07: Do you think I should try 2.2.15? Or stick to EE? This is in the repo. My version just has *all* the latest changes applied which sometime can be untested. signature.asc Description: Digital signature

Re: Indexing Mail faster

2015-01-28 Thread Thomas Leuxner
* Kevin Laurie superinterstel...@gmail.com 2015.01.28 03:31: . OK Search completed (21.478 secs). Really hard to say what commit probably fixed that already. Dovecot uses a very dynamic development model. There are old versions like 2.0.9 which people still try to get help with since they are

Re: Indexing Mail faster

2015-01-28 Thread superinterstellar
Dear Thomas, Do you think I should try 2.2.15? Or stick to EE? Tempted to try the latest dovecot. Thanks Kevin   Original Message   From: Thomas Leuxner Sent: Wednesday, 28 January 2015 11:00 To: dovecot@dovecot.org Reply To: dovecot@dovecot.org Subject: Re: Indexing Mail faster * superinterstel

Re: Indexing Mail faster

2015-01-28 Thread Thomas Leuxner
* superinterstel...@gmail.com superinterstel...@gmail.com 2015.01.28 11:53: Also could you advise which dovecot are you using? Bleeding edge (latest and hopefully greatest compiled from HG): # 2.2.15 (7557234ac0f4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 signature.asc

Re: Indexing Mail faster

2015-01-28 Thread Peter Hodur
* Kevin Laurie superinterstel...@gmail.com 2015.01.24 19:41: Currently the time it takes to search 25,000mails is 4mins. If indexed how much faster are we looking at? With a current version of Dovecot a search is pretty fast _without_ using external indexes. I have a view defined

Re: Indexing Mail faster

2015-01-28 Thread Kevin Laurie
Dear Peter, My inbox is MDA_external Storage: 17GB of 24GB Subject / From / To is fast but FTS(Full Text Search) for body is horrible. I suppose this is where we need Apache Solr. Do you think my mail storage format is bad? Do I need to change for better performance? Please advise Kevin On

Re: Indexing Mail faster

2015-01-28 Thread Kevin Laurie
Hi Peter, Sorry, I think its maildir Output of my dovecot -n is listed below:- # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS auth_master_user_separator = * auth_mechanisms = PLAIN LOGIN dict { acl = mysql:/etc/dovecot/dovecot-

Re: Indexing Mail faster

2015-01-28 Thread Kevin Laurie
Dear Peter, Noted. Thanks for your input. Appreciate it. At this point my urgent most priority is to get FTS. Waiting 3 mins for a body search email is bad(but of course my email box is large) I need to have this sorted out by today as I have been putting this off for too long mostly because of

Re: Indexing Mail faster

2015-01-28 Thread Kevin Laurie
Hey Joseph Thanks. Strace seems like a very useful tool Only problem is that I dont think it is maintained on ubuntu. Tried to run:- apt-get install strace but could not download it. Might need to download and build it. Do you know any other way of getting it? Thanks Kevin A. On Thu, Jan 29,

Re: Indexing Mail faster

2015-01-28 Thread Joseph Tam
Kevin writes: Appreciate if you could help with this. I have been trying to address this slow search issue for a while with very limited success(I was trying to implement FTS also), so I will appreciate if you could support. When I'm stumped, one of the diagnostic tools I use is process

Re: Indexing Mail faster

2015-01-28 Thread Kevin Laurie
Hi guys, Please correct me if I am adding repos to dovecot correctly. I hope I am doing it correctly Step 1. Add the following repos to the following file /etc/apt/sources.list. deb http://xi.rename-it.nl/debian/pool/stable-auto/dovecot-2.2/ trusty universe deb-src

Re: Indexing Mail faster

2015-01-28 Thread Kevin Laurie
Dear Thomas, From what I have read is that I cannot automatically install dovecot, I would need to add the repos to my repo list and then use pining to install the latest version. Kindly advise if I am on correct track? Thanks Kevin On Wed, Jan 28, 2015 at 10:08 PM, Kevin Laurie

Re: Indexing Mail faster

2015-01-27 Thread Thomas Leuxner
* Kevin Laurie superinterstel...@gmail.com 2015.01.27 03:24: . search text dear * OK Searched 9% of the mailbox, ETA 1:39 * OK Searched 20% of the mailbox, ETA 1:18 * OK Searched 29% of the mailbox, ETA 1:12 * OK Searched 38% of the mailbox, ETA 1:03 * OK Searched 50% of the mailbox, ETA

Re: Indexing Mail faster

2015-01-27 Thread Kevin Laurie
Dear Thomas, Below is the output of my 'dovecot -n' . The slow speed is really bad, imagine waiting 4 mins for every search. . Something is terribly wrong in my config. Please have a look and advise. Thanks Kevin # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64

Re: Indexing Mail faster

2015-01-27 Thread Kevin Laurie
Dear Thomas, I have removed the fts plugin. See below:- # dovecot -n # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS auth_master_user_separator = * auth_mechanisms = PLAIN LOGIN dict { acl = mysql:/etc/dovecot/dovecot-share-folder.conf

Re: Indexing Mail faster

2015-01-27 Thread Thomas Leuxner
* Kevin Laurie superinterstel...@gmail.com 2015.01.27 11:15: Hi Kevin, # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS Although this was released back in 2013, let's assume the core search is not substantially broken for this release.

Re: Indexing Mail faster

2015-01-27 Thread Kevin Laurie
Also tried to reload the config but still getting slow search. Just tried to reindex the box by running dovecot index -u u...@email.net inbox I am still getting searches close to 2:45 minutes. It seems strange. A smaller inbox took less time:- . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR

Re: Indexing Mail faster

2015-01-26 Thread Kevin Laurie
Hi Thomas, That's very interesting. Below is my search result. Any idea why is my result so slow:- Appreciate if you could advise. Thanks Kevin . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT

Re: Indexing Mail faster

2015-01-25 Thread Andreas Schulze
Thomas Leuxner: I have a view defined (virtual plugin) with around 22.000 messages in it interesting. I assume you drop multiple mailing lists together in a mailbox and separate them using the virtual plugin, right? could publish how you configured dovecot virtual plugin? I never had success on

Re: Indexing Mail faster

2015-01-25 Thread Thomas Leuxner
* A. Schulze s...@andreasschulze.de 2015.01.25 16:58: once setup correctly it works like expected :-) Good! how do you separate older postings into the Archive? something like doveadm search + move ? Yes basically with move. I'd like to have them archived by year e.g.

Re: Indexing Mail faster

2015-01-25 Thread A. Schulze
Thomas Leuxner: namespace { location = virtual:~/mdbox/virtual prefix = Virtual/ separator = / } $ cat virtual/Flagged/dovecot-virtual * Public/* flagged once setup correctly it works like expected :-) Another example, the one I used in the original reply, is 'gluing' together

Re: Indexing Mail faster

2015-01-25 Thread LuKreme
On Jan 25, 2015, at 5:05 AM, Thomas Leuxner t...@leuxner.net wrote: The namespace Virtual will look for its configuration in the user's home directory: namespace { location = virtual:~/mdbox/virtual prefix = Virtual/ separator = / } Inside the ~/mdbox/virtual directory the actual

Dovecot Virtual Plugin Use Cases (was Re: Indexing Mail faster)

2015-01-25 Thread Thomas Leuxner
* LuKreme krem...@kreme.com 2015.01.25 19:32: Another example, the one I used in the original reply, is 'gluing' together archives as a single view: $ cat virtual/Postfix-Users/dovecot-virtual Public/Archive/Mailing-Lists/Postfix-Users/* Public/Mailing-Lists/Postfix-Users all

Re: Indexing Mail faster

2015-01-25 Thread Thomas Leuxner
* Andreas Schulze s...@andreasschulze.de 2015.01.25 12:47: I assume you drop multiple mailing lists together in a mailbox and separate them using the virtual plugin, right? could publish how you configured dovecot virtual plugin? The namespace Virtual will look for its configuration in the

Indexing Mail faster

2015-01-24 Thread Kevin Laurie
Hi, I am trying to get faster search results on our webmail client(Roundcube). Besides using Lucene for FTS are there other options? Would having all mails indexed give fast results? Currently the time it takes to search 25,000mails is 4mins. If indexed how much faster are we looking at? Really

Re: Indexing Mail faster

2015-01-24 Thread Thomas Leuxner
* Kevin Laurie superinterstel...@gmail.com 2015.01.24 19:41: Currently the time it takes to search 25,000mails is 4mins. If indexed how much faster are we looking at? With a current version of Dovecot a search is pretty fast _without_ using external indexes. I have a view defined (virtual