Re: Indexing Mail faster
Hi Micheal, Here is how it should look:- ~# telnet localhost imap Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. . login u...@domain.net password . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE SEARCH=FUZZY QUOTA] Logged in . select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded \*)] Flags permitted. * 49319 EXISTS * 12 RECENT * OK [UNSEEN 46791] First unseen. * OK [UIDVALIDITY 1414214135] UIDs valid * OK [UIDNEXT 106554] Predicted next UID * OK [NOMODSEQ] No permanent modsequences . OK [READ-WRITE] Select completed (0.128 secs). . search text hello * OK Searched 6% of the mailbox, ETA 2:34 Finally at the end you logout by . logout On Thu, Jan 29, 2015 at 4:39 PM, Kevin Laurie superinterstel...@gmail.com wrote: Dear Michael, You will need to telnet locally to imap. So run the following commands in your server 1. telnet localhost imap 2. . login em...@domain.com password 3. . select inbox 4. . search text passport (This command here will search your inbox and give you the time needed) 5. . logout Notice the period(.) You need to use it . Hope this is clear. Regards Kevin On Thu, Jan 29, 2015 at 4:32 PM, M. Koehler michael.koeh...@makomi.de wrote: Hi Kevin, I want to test it with my dovecot but I don´t know how to measure it. Could you please provide my the configuration/terminal commands to get an reliable output? I just have mailboxes up to 4GB in VM on a proxmox server (Debian Wheezy at the moment). regards, Michael Am 29.01.2015 um 06:29 schrieb Kevin Laurie superinterstel...@gmail.com: Dear Peter, Noted. Thanks for your input. Appreciate it. At this point my urgent most priority is to get FTS. Waiting 3 mins for a body search email is bad(but of course my email box is large) I need to have this sorted out by today as I have been putting this off for too long mostly because of lack of troubleshooting documentation online(if i do figure this out will create some for guidance) Regards Kevin On Thu, Jan 29, 2015 at 1:18 PM, Peter Hodur peteho...@gmail.com wrote: On Thursday, January 29, 2015, Kevin Laurie superinterstel...@gmail.com wrote: Dear Peter, Oh. Sorry(didnt know you were addressing someone else) my apologies. But as you can see , I am desperately trying to address this issue. No problem ;))) i wrote because my search result is good but not so good as someone wrote here couple of days ago. I have 1 disk with 200GB running on a VPS. The file system is ext4. Im not dovecot expert, but i think this is your problem. You need more iops. If you need performance, VPS may be (if connected to fast SAN) but basicly is not a good choice. I think I will need to implement FTS to fix this as I will need body searches. FTS could be a solution. You are right. Have you tried FTS before? Im sorry, never :( But may be someone more skilled than me could answer the main question, how much dovecot index helps in case imap search agains body. Im not sure, but my opinion is that it helps little (may be not at all) Ps: may be you use it, but do not forget to setup delivering from mail server via LDA or LMTP instead of direct to maildirs. This is better because dovecot updates your indexes at the time of message arrival. Later accesses should be faster. Peter Regards Kevin On Thu, Jan 29, 2015 at 12:55 PM, Peter Hodur peteho...@gmail.com wrote: Kevin, My message was not addressed to you ;) because my results are NOT so good as someone wrote it here. I can fulltext cca 8k messages in 7-8 seconds. Someone wrote that he has results cca 22k messages in 4 seconds :( Im not sure, but the answer is may be in the index and disk subsystem. If and only if dovecot index does not have saved in the index keywords from BODY of messages the problem is your/my disk subsystem. Generally, messages from mailing list like this are pretty small. The problem is that dovecot must read all messages. If you have maildir its mean open and read many files. But my test search was agains archive of my personal inbox - so not onoy small messages like this but also messages with big attachments etc. If dovecot index only headers, than it in case of full text search read whole messages - mime parts allows that plain text body can be after attachment etc. And here is it all about iops and throughput. How many disks you have? And what setup? Generally, the only good setup is many smaller disks
Re: Indexing Mail faster
Dear Marc, Noted. This is starting to make a lot of sense. Of course, more computing power, hence more performance. Also filesystems and such all have and effect on the outcome. I might experiment with adding more computing power and seeing the result. My mailbox does not have many users so I might try that. Currently running a 2gb RAM with 2core processors. On Thu, Jan 29, 2015 at 4:23 PM, Marc Stürmer m...@marc-stuermer.de wrote: Am 27.01.2015 um 03:24 schrieb Kevin Laurie: Hi Thomas, That's very interesting. Below is my search result. Any idea why is my result so slow:- Appreciate if you could advise. It depends on a couple of facts. Full text searches must go through a number of bottle necks, depending on your installation, namely: a) type of disk drive(s), b) way of installation of those drives, c) the underlying file system itself, d) the storage format being chosen to save the mails, e) how many users you've got on your system, f) memory of your system, g) CPU power of your system and h) how busy your system is. If you got for example a lonely box with let's sax four gigs of RAM and you are using it by yourself only, chances are high enough that even with a big enough up time most of your mails are in the file system cache of your OS. Meaning a full text search would happen mostly in RAM and therefor of course would be blazingly fast. If you take the same machine and let it serve let's say about 2000 mailboxes, this would be a very different kind of matter. Your file system cache would be flushed frequently and full text searches without index would be way slower, because now those search actually mean I/O operations for Dovecot on your type of storage. If you really want consistent, fast full text search speeds on a busy box, incremental full text indexing is the only way to go. If you are the only user of a big box chances are high you get similar results without.
Re: Indexing Mail faster
Dear Joseph,
Just to clarify, the pid you are referring to is of dovecot?
Correct?
Thanks
Kevin
On Thu, Jan 29, 2015 at 3:34 PM, Joseph Tam jtam.h...@gmail.com wrote:
Kevin writes:
Appreciate if you could help with this. I have been trying to address this
slow search issue for a while with very limited success(I was trying to
implement FTS also), so I will appreciate if you could support.
When I'm stumped, one of the diagnostic tools I use is process tracing.
Connect via IMAP, in another window/session process trace the IMAP worker
process (Linux: strace -r -otrace.dump {pid})), initiate the search and
quit. Then you can look through the trace and see whether it gets hung on
a particular operation.
Joseph Tam jtam.h...@gmail.com
Re: Indexing Mail faster
Yes. I was missing due to repositories. Added more sources and now its fine. Thanks! Kevin On Thu, Jan 29, 2015 at 4:09 PM, Kevin Laurie superinterstel...@gmail.com wrote: Hi Thom, No results. I think I am missing some repositories. I might add some from ubuntu. On Thu, Jan 29, 2015 at 4:02 PM, Thom Miller t...@cagroups.com wrote: On Thu, 2015-01-29 at 15:56 +0800, Kevin Laurie wrote: Hey Joseph Thanks. Strace seems like a very useful tool Only problem is that I dont think it is maintained on ubuntu. Tried to run:- apt-get install strace but could not download it. Might need to download and build it. Do you know any other way of getting it? Thanks Kevin A. try apt-cache search strace On Jessie (I'm not running Ubuntu) I get: devscripts - scripts to make the life of a Debian Package maintainer easier dnstracer - trace DNS queries to the source ioapps - IO profiler and IO traces replayer netsniff-ng - Linux network packet sniffer toolkit python-ptrace - Python bindings for ptrace strace - System call tracer subversion-tools - Assorted tools related to Apache Subversion xtrace - trace communication between X client and server You should be able to find the appropriate package to install to get strace. -Thom
Re: Indexing Mail faster
Am 27.01.2015 um 03:24 schrieb Kevin Laurie: Hi Thomas, That's very interesting. Below is my search result. Any idea why is my result so slow:- Appreciate if you could advise. It depends on a couple of facts. Full text searches must go through a number of bottle necks, depending on your installation, namely: a) type of disk drive(s), b) way of installation of those drives, c) the underlying file system itself, d) the storage format being chosen to save the mails, e) how many users you've got on your system, f) memory of your system, g) CPU power of your system and h) how busy your system is. If you got for example a lonely box with let's sax four gigs of RAM and you are using it by yourself only, chances are high enough that even with a big enough up time most of your mails are in the file system cache of your OS. Meaning a full text search would happen mostly in RAM and therefor of course would be blazingly fast. If you take the same machine and let it serve let's say about 2000 mailboxes, this would be a very different kind of matter. Your file system cache would be flushed frequently and full text searches without index would be way slower, because now those search actually mean I/O operations for Dovecot on your type of storage. If you really want consistent, fast full text search speeds on a busy box, incremental full text indexing is the only way to go. If you are the only user of a big box chances are high you get similar results without.
Re: Indexing Mail faster
Dear Thomas, You do seem to have some computing power. I think that does help. Thanks Kevin A. On Thu, Jan 29, 2015 at 4:20 PM, Thomas Leuxner t...@leuxner.net wrote: * Peter Hodur peteho...@gmail.com 2015.01.29 05:25: With a current version of Dovecot a search is pretty fast _without_ using external indexes. I have a view defined (virtual plugin) with around 22.000 messages in it, and searching the full view only takes 2.5 seconds: hmmm, could you please tell me more about your setup? What storage format do you use? Maildir or? What is the cumulative size of your messages? Although the question was not formatted/quoted correctly I guess it is more towards myself as it explicitely speaks of a non-FTS setup: $ dmesg | grep smpboot [0.00] smpboot: Allowing 64 CPUs, 56 hotplug CPUs [0.148600] smpboot: CPU0: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz (fam: 06, model: 3c, stepping: 03) [0.300030] smpboot: Total of 8 processors activated (52677.72 BogoMIPS) The volume providing the mail root is using ext4 on an MD mirror (with CRYPT and LVM overhead): $ lsblk [...] └─sda38:30 ... 0 part └─md2 9:20 ... 0 raid1 └─crypt1 (dm-0) 253:00 ... 0 crypt └─vg0-vmail (dm-11) 253:11 0 ... 0 lvm /var/vmail $ doveconf -n | head -3 # 2.2.15 (7557234ac0f4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 # OS: Linux 3.16.0-0.bpo.4-amd64 x86_64 Debian 7.8 I'm using unspectacular 7,200rpm SATA drives and mdbox.
Re: Indexing Mail faster
* Peter Hodur peteho...@gmail.com 2015.01.29 05:25: With a current version of Dovecot a search is pretty fast _without_ using external indexes. I have a view defined (virtual plugin) with around 22.000 messages in it, and searching the full view only takes 2.5 seconds: hmmm, could you please tell me more about your setup? What storage format do you use? Maildir or? What is the cumulative size of your messages? Although the question was not formatted/quoted correctly I guess it is more towards myself as it explicitely speaks of a non-FTS setup: $ dmesg | grep smpboot [0.00] smpboot: Allowing 64 CPUs, 56 hotplug CPUs [0.148600] smpboot: CPU0: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz (fam: 06, model: 3c, stepping: 03) [0.300030] smpboot: Total of 8 processors activated (52677.72 BogoMIPS) The volume providing the mail root is using ext4 on an MD mirror (with CRYPT and LVM overhead): $ lsblk [...] └─sda38:30 ... 0 part └─md2 9:20 ... 0 raid1 └─crypt1 (dm-0) 253:00 ... 0 crypt └─vg0-vmail (dm-11) 253:11 0 ... 0 lvm /var/vmail $ doveconf -n | head -3 # 2.2.15 (7557234ac0f4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 # OS: Linux 3.16.0-0.bpo.4-amd64 x86_64 Debian 7.8 I'm using unspectacular 7,200rpm SATA drives and mdbox. signature.asc Description: Digital signature
Re: Indexing Mail faster
Hi Thom, No results. I think I am missing some repositories. I might add some from ubuntu. On Thu, Jan 29, 2015 at 4:02 PM, Thom Miller t...@cagroups.com wrote: On Thu, 2015-01-29 at 15:56 +0800, Kevin Laurie wrote: Hey Joseph Thanks. Strace seems like a very useful tool Only problem is that I dont think it is maintained on ubuntu. Tried to run:- apt-get install strace but could not download it. Might need to download and build it. Do you know any other way of getting it? Thanks Kevin A. try apt-cache search strace On Jessie (I'm not running Ubuntu) I get: devscripts - scripts to make the life of a Debian Package maintainer easier dnstracer - trace DNS queries to the source ioapps - IO profiler and IO traces replayer netsniff-ng - Linux network packet sniffer toolkit python-ptrace - Python bindings for ptrace strace - System call tracer subversion-tools - Assorted tools related to Apache Subversion xtrace - trace communication between X client and server You should be able to find the appropriate package to install to get strace. -Thom
Re: Indexing Mail faster
On Thu, 2015-01-29 at 15:56 +0800, Kevin Laurie wrote: Hey Joseph Thanks. Strace seems like a very useful tool Only problem is that I dont think it is maintained on ubuntu. Tried to run:- apt-get install strace but could not download it. Might need to download and build it. Do you know any other way of getting it? Thanks Kevin A. try apt-cache search strace On Jessie (I'm not running Ubuntu) I get: devscripts - scripts to make the life of a Debian Package maintainer easier dnstracer - trace DNS queries to the source ioapps - IO profiler and IO traces replayer netsniff-ng - Linux network packet sniffer toolkit python-ptrace - Python bindings for ptrace strace - System call tracer subversion-tools - Assorted tools related to Apache Subversion xtrace - trace communication between X client and server You should be able to find the appropriate package to install to get strace. -Thom
Re: Indexing Mail faster
On Thu, 29 Jan 2015, Kevin Laurie wrote: Just to clarify, the pid you are referring to is of dovecot? Yes, the dovecot/imap worker process. If there is no system call that is using up a lot of time, but the cumulative open/read time for going through many files in your Maildir, then you have some sort of filesystem metadata performance problem. Joseph Tam jtam.h...@gmail.com
Re: Indexing Mail faster
Dear Mr.Thomas, Thanks for your feedback. Will try dovecot-ee. Also could you advise which dovecot are you using? Thanks Kevin Original Message From: Thomas Leuxner Sent: Wednesday, 28 January 2015 08:45 To: dovecot@dovecot.org Reply To: dovecot@dovecot.org Subject: Re: Indexing Mail faster * Kevin Laurie superinterstel...@gmail.com 2015.01.28 03:31: . OK Search completed (21.478 secs). Really hard to say what commit probably fixed that already. Dovecot uses a very dynamic development model. There are old versions like 2.0.9 which people still try to get help with since they are in some dusted distro repository, but known to have partially broken features... Looking at the CVS of Dovecot there are so many changes it's really hard to say there was an issue or not. http://hg.dovecot.org/dovecot-2.2/shortlog One changelog (2.2.11) however indicates the search was not working as it should: - IMAP: SEARCH/SORT PARTIAL was handled completely wrong in v2.2.11+ You may want to look at the official Enterprise repository which does support Ubuntu. This one does have the current stable releases in it and is not undergoing so many changes compared to the CVS auto-compiles also offered: http://shop.dovecot.fi/home/8-dovecot-ee-repository-access.html Give that one a try with vanilla search...
Re: Indexing Mail faster
Dear Thomas, Could you advise(or provide a link to) for a safe way to upgrade the Dovecot version. I am a little worried about testing and making trial-and-error approach as it could be risky. Appreciate if you could help with this. I have been trying to address this slow search issue for a while with very limited success(I was trying to implement FTS also), so I will appreciate if you could support. Apologies for my noobness. Thank You Kevin Do you have much ideas on how to upgrade dovecot. I am little worried about upgrading dovecot as there could be risks to data. I am trying to upgrade from 2.2.9 to 2.2.15 or Dovecot-EE. On Wed, Jan 28, 2015 at 9:16 PM, Thomas Leuxner t...@leuxner.net wrote: * superinterstel...@gmail.com superinterstel...@gmail.com 2015.01.28 12 :07: Do you think I should try 2.2.15? Or stick to EE? This is in the repo. My version just has *all* the latest changes applied which sometime can be untested.
Re: Indexing Mail faster
* superinterstel...@gmail.com superinterstel...@gmail.com 2015.01.28 12:07: Do you think I should try 2.2.15? Or stick to EE? This is in the repo. My version just has *all* the latest changes applied which sometime can be untested. signature.asc Description: Digital signature
Re: Indexing Mail faster
* Kevin Laurie superinterstel...@gmail.com 2015.01.28 03:31: . OK Search completed (21.478 secs). Really hard to say what commit probably fixed that already. Dovecot uses a very dynamic development model. There are old versions like 2.0.9 which people still try to get help with since they are in some dusted distro repository, but known to have partially broken features... Looking at the CVS of Dovecot there are so many changes it's really hard to say there was an issue or not. http://hg.dovecot.org/dovecot-2.2/shortlog One changelog (2.2.11) however indicates the search was not working as it should: - IMAP: SEARCH/SORT PARTIAL was handled completely wrong in v2.2.11+ You may want to look at the official Enterprise repository which does support Ubuntu. This one does have the current stable releases in it and is not undergoing so many changes compared to the CVS auto-compiles also offered: http://shop.dovecot.fi/home/8-dovecot-ee-repository-access.html Give that one a try with vanilla search... signature.asc Description: Digital signature
Re: Indexing Mail faster
Dear Thomas, Do you think I should try 2.2.15? Or stick to EE? Tempted to try the latest dovecot. Thanks Kevin Original Message From: Thomas Leuxner Sent: Wednesday, 28 January 2015 11:00 To: dovecot@dovecot.org Reply To: dovecot@dovecot.org Subject: Re: Indexing Mail faster * superinterstel...@gmail.com superinterstel...@gmail.com 2015.01.28 11:53: Also could you advise which dovecot are you using? Bleeding edge (latest and hopefully greatest compiled from HG): # 2.2.15 (7557234ac0f4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6
Re: Indexing Mail faster
* superinterstel...@gmail.com superinterstel...@gmail.com 2015.01.28 11:53: Also could you advise which dovecot are you using? Bleeding edge (latest and hopefully greatest compiled from HG): # 2.2.15 (7557234ac0f4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 signature.asc Description: Digital signature
Re: Indexing Mail faster
* Kevin Laurie superinterstel...@gmail.com 2015.01.24 19:41: Currently the time it takes to search 25,000mails is 4mins. If indexed how much faster are we looking at? With a current version of Dovecot a search is pretty fast _without_ using external indexes. I have a view defined (virtual plugin) with around 22.000 messages in it, and searching the full view only takes 2.5 seconds: hmmm, could you please tell me more about your setup? What storage format do you use? Maildir or? What is the cumulative size of your messages? My results without FTS on ZFS FS with SSD L2ARC are not so good: a4 select INBOX._OLD-OUTLOOK * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk JunkRecorded $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. ** 7748 EXISTS* * 0 RECENT * OK [UIDVALIDITY 1421198037] UIDs valid * OK [UIDNEXT 11509] Predicted next UID * OK [HIGHESTMODSEQ 12204] Highest a4 OK [READ-WRITE] Select completed (0.001 secs). a5 search charset utf-8 body mall * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 240 249 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 385 389 390 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 531 532 533 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 672 677 692 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 821 833 855 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 1010 1011 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 1137 1140 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 1282 1286 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 1471 1494 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 1606 1611 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 1735 1781 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 2002 2006 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 2208 2210 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 2377 2378 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 2539 2551 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 2754 2780 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 2936 2938 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 3161 3178 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 3386 3428 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 3801 3808 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 3953 3954 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 4059 4063 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 4145 4146 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 4258 4259 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 4502 4538 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 4594 4604 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 4787 4788 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 4901 4905 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 5129 5131 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 5276 5277 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 5391 5392 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 5642 5647 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 5866 5867 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 5958 5990 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 6233 6271 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 6482 6492 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 6852 6859 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 7042 7108 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 7317 7322 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 7417 7418 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 7502 7503 7558 7588 7589 7628 7685 7695 7699 7703 7723 *a5 OK Search completed (7.846 secs).* Searching against subject is pretty fast, few miliseconds ...
Re: Indexing Mail faster
Dear Peter, My inbox is MDA_external Storage: 17GB of 24GB Subject / From / To is fast but FTS(Full Text Search) for body is horrible. I suppose this is where we need Apache Solr. Do you think my mail storage format is bad? Do I need to change for better performance? Please advise Kevin On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur peteho...@gmail.com wrote: * Kevin Laurie superinterstel...@gmail.com 2015.01.24 19:41: Currently the time it takes to search 25,000mails is 4mins. If indexed how much faster are we looking at? With a current version of Dovecot a search is pretty fast _without_ using external indexes. I have a view defined (virtual plugin) with around 22.000 messages in it, and searching the full view only takes 2.5 seconds: hmmm, could you please tell me more about your setup? What storage format do you use? Maildir or? What is the cumulative size of your messages? My results without FTS on ZFS FS with SSD L2ARC are not so good: a4 select INBOX._OLD-OUTLOOK * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk JunkRecorded $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. ** 7748 EXISTS* * 0 RECENT * OK [UIDVALIDITY 1421198037] UIDs valid * OK [UIDNEXT 11509] Predicted next UID * OK [HIGHESTMODSEQ 12204] Highest a4 OK [READ-WRITE] Select completed (0.001 secs). a5 search charset utf-8 body mall * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 240 249 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 385 389 390 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 531 532 533 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 672 677 692 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 821 833 855 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 1010 1011 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 1137 1140 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 1282 1286 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 1471 1494 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 1606 1611 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 1735 1781 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 2002 2006 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 2208 2210 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 2377 2378 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 2539 2551 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 2754 2780 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 2936 2938 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 3161 3178 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 3386 3428 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 3801 3808 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 3953 3954 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 4059 4063 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 4145 4146 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 4258 4259 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 4502 4538 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 4594 4604 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 4787 4788 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 4901 4905 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 5129 5131 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 5276 5277 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 5391 5392 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 5642 5647 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 5866 5867 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 5958 5990 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 6233 6271 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 6482 6492 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 6852 6859 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 7042 7108 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 7317 7322 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 7417 7418 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 7502 7503 7558 7588 7589 7628 7685 7695 7699 7703 7723 *a5 OK Search completed (7.846 secs).* Searching against subject is pretty fast, few miliseconds ...
Re: Indexing Mail faster
Hi Peter,
Sorry,
I think its maildir
Output of my dovecot -n is listed below:-
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS
auth_master_user_separator = *
auth_mechanisms = PLAIN LOGIN
dict {
acl = mysql:/etc/dovecot/dovecot-
share-folder.conf
quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf
}
first_valid_uid = 2000
last_valid_uid = 2000
listen = *
log_path = /var/log/dovecot.log
mail_debug = yes
mail_gid = 2000
mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
mail_plugins = quota fts
mail_uid = 2000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
namespace {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Sent Messages {
auto = no
special_use = \Sent
}
mailbox Spam {
auto = no
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = /
type = private
}
namespace {
list = children
location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u
prefix = Shared/%%u/
separator = /
subscriptions = yes
type = shared
}
passdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
passdb {
args = /etc/dovecot/dovecot-master-users-password
driver = passwd-file
master = yes
}
plugin {
acl = vfile
acl_shared_dict = proxy::acl
auth_socket_path = /var/run/dovecot/auth-master
quota = dict:user::proxy::quotadict
quota_rule = *:storage=1G
quota_warning = storage=85%% quota-warning 85 %u
quota_warning2 = storage=90%% quota-warning 90 %u
quota_warning3 = storage=95%% quota-warning 95 %u
sieve = /%Lh/sieve/dovecot.sieve
sieve_default = /var/vmail/sieve/dovecot.sieve
sieve_dir = /%Lh/sieve
sieve_global_dir = /var/vmail/sieve
}
protocols = pop3 imap sieve lmtp
service auth {
unix_listener /var/spool/postfix/private/dovecot-auth {
group = postfix
mode = 0666
user = postfix
}
unix_listener auth-master {
group = vmail
mode = 0666
user = vmail
}
unix_listener auth-userdb {
group = vmail
mode = 0660
user = vmail
}
}
service dict {
unix_listener dict {
group = vmail
mode = 0660
user = vmail
}
}
service imap-login {
process_limit = 500
service_count = 1
}
service lmtp {
executable = lmtp -L
inet_listener lmtp {
port = 24
}
process_min_avail = 5
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
user = vmail
}
service pop3-login {
service_count = 1
}
service quota-warning {
executable = script /usr/local/bin/dovecot-quota-warning.sh
unix_listener quota-warning {
group = vmail
mode = 0660
user = vmail
}
}
ssl = required
ssl_cert = /etc/ssl/certs/sicl.net.crt
ssl_key = /etc/ssl/private/sicl.net.key
userdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
lda_mailbox_autocreate = yes
log_path = /var/log/dovecot-sieve.log
mail_plugins = quota fts sieve
postmaster_address = root
}
protocol lmtp {
info_log_path = /var/log/dovecot-lmtp.log
lmtp_save_to_detail_mailbox = yes
mail_plugins = quota sieve
postmaster_address = postmaster
recipient_delimiter = +
}
protocol imap {
imap_client_workarounds = tb-extra-mailbox-sep
mail_plugins = quota fts imap_quota
}
protocol pop3 {
mail_plugins = quota fts
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %08Xu%08Xv
}
On Thu, Jan 29, 2015 at 12:37 PM, Kevin Laurie superinterstel...@gmail.com
wrote:
Dear Peter,
My inbox is MDA_external
Storage: 17GB of 24GB
Subject / From / To is fast but FTS(Full Text Search) for body is
horrible. I suppose this is where we need Apache Solr.
Do you think my mail storage format is bad? Do I need to change for better
performance?
Please advise
Kevin
On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur peteho...@gmail.com wrote:
* Kevin Laurie superinterstel...@gmail.com 2015.01.24 19:41:
Currently the time it takes to search 25,000mails is 4mins. If indexed
how
much faster are we looking at?
With a current version of Dovecot a search is pretty fast _without_
using
external indexes. I have a view defined (virtual plugin) with around
22.000
messages in it, and searching the full view only takes 2.5 seconds:
hmmm, could you please tell me more about your setup? What storage format
do you use? Maildir or? What is the cumulative size of your messages?
My results without FTS on ZFS FS with SSD L2ARC are not so good:
a4 select
Re: Indexing Mail faster
Dear Peter,
Noted. Thanks for your input. Appreciate it.
At this point my urgent most priority is to get FTS. Waiting 3 mins for a
body search email is bad(but of course my email box is large)
I need to have this sorted out by today as I have been putting this off for
too long mostly because of lack of troubleshooting documentation online(if
i do figure this out will create some for guidance)
Regards
Kevin
On Thu, Jan 29, 2015 at 1:18 PM, Peter Hodur peteho...@gmail.com wrote:
On Thursday, January 29, 2015, Kevin Laurie superinterstel...@gmail.com
wrote:
Dear Peter,
Oh. Sorry(didnt know you were addressing someone else) my apologies. But
as you can see , I am desperately trying to address this issue.
No problem ;))) i wrote because my search result is good but not so
good as someone wrote here couple of days ago.
I have 1 disk with 200GB running on a VPS.
The file system is ext4.
Im not dovecot expert, but i think this is your problem. You need more
iops. If you need performance, VPS may be (if connected to fast SAN) but
basicly is not a good choice.
I think I will need to implement FTS to fix this as I will need body
searches.
FTS could be a solution. You are right.
Have you tried FTS before?
Im sorry, never :(
But may be someone more skilled than me could answer the main question,
how much dovecot index helps in case imap search agains body.
Im not sure, but my opinion is that it helps little (may be not at all)
Ps: may be you use it, but do not forget to setup delivering from mail
server via LDA or LMTP instead of direct to maildirs. This is better
because dovecot updates your indexes at the time of message arrival. Later
accesses should be faster.
Peter
Regards
Kevin
On Thu, Jan 29, 2015 at 12:55 PM, Peter Hodur peteho...@gmail.com
wrote:
Kevin,
My message was not addressed to you ;) because my results are NOT so
good as someone wrote it here. I can fulltext cca 8k messages in 7-8
seconds.
Someone wrote that he has results cca 22k messages in 4 seconds :(
Im not sure, but the answer is may be in the index and disk subsystem.
If and only if dovecot index does not have saved in the index keywords
from BODY of messages the problem is your/my disk subsystem.
Generally, messages from mailing list like this are pretty small. The
problem is that dovecot must read all messages. If you have maildir its
mean open and read many files.
But my test search was agains archive of my personal inbox - so not onoy
small messages like this but also messages with big attachments etc.
If dovecot index only headers, than it in case of full text search read
whole messages - mime parts allows that plain text body can be after
attachment etc.
And here is it all about iops and throughput.
How many disks you have? And what setup?
Generally, the only good setup is many smaller disks in RAID 10
(stripped mirrors) and if your fs allows it, with added read cache - in
case of zfs: l2arc on ssd.
Pete
On Thursday, January 29, 2015, Kevin Laurie superinterstel...@gmail.com
wrote:
Hi Peter,
Sorry,
I think its maildir
Output of my dovecot -n is listed below:-
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS
auth_master_user_separator = *
auth_mechanisms = PLAIN LOGIN
dict {
acl = mysql:/etc/dovecot/dovecot-
share-folder.conf
quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf
}
first_valid_uid = 2000
last_valid_uid = 2000
listen = *
log_path = /var/log/dovecot.log
mail_debug = yes
mail_gid = 2000
mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
mail_plugins = quota fts
mail_uid = 2000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric relational
regex imap4flags copy include variables body enotify environment mailbox
date ihave
namespace {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Sent Messages {
auto = no
special_use = \Sent
}
mailbox Spam {
auto = no
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = /
type = private
}
namespace {
list = children
location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u
prefix = Shared/%%u/
separator = /
subscriptions = yes
type = shared
}
passdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
passdb {
args = /etc/dovecot/dovecot-master-users-password
driver = passwd-file
master = yes
}
plugin {
acl = vfile
acl_shared_dict = proxy::acl
auth_socket_path = /var/run/dovecot/auth-master
quota =
Re: Indexing Mail faster
Hey Joseph
Thanks. Strace seems like a very useful tool
Only problem is that I dont think it is maintained on ubuntu.
Tried to run:-
apt-get install strace but could not download it.
Might need to download and build it. Do you know any other way of getting
it?
Thanks
Kevin A.
On Thu, Jan 29, 2015 at 3:34 PM, Joseph Tam jtam.h...@gmail.com wrote:
Kevin writes:
Appreciate if you could help with this. I have been trying to address this
slow search issue for a while with very limited success(I was trying to
implement FTS also), so I will appreciate if you could support.
When I'm stumped, one of the diagnostic tools I use is process tracing.
Connect via IMAP, in another window/session process trace the IMAP worker
process (Linux: strace -r -otrace.dump {pid})), initiate the search and
quit. Then you can look through the trace and see whether it gets hung on
a particular operation.
Joseph Tam jtam.h...@gmail.com
Re: Indexing Mail faster
Kevin writes:
Appreciate if you could help with this. I have been trying to address this
slow search issue for a while with very limited success(I was trying to
implement FTS also), so I will appreciate if you could support.
When I'm stumped, one of the diagnostic tools I use is process tracing.
Connect via IMAP, in another window/session process trace the IMAP worker
process (Linux: strace -r -otrace.dump {pid})), initiate the search and
quit. Then you can look through the trace and see whether it gets hung on
a particular operation.
Joseph Tam jtam.h...@gmail.com
Re: Indexing Mail faster
Hi guys, Please correct me if I am adding repos to dovecot correctly. I hope I am doing it correctly Step 1. Add the following repos to the following file /etc/apt/sources.list. deb http://xi.rename-it.nl/debian/pool/stable-auto/dovecot-2.2/ trusty universe deb-src http://xi.rename-it.nl/debian/pool/stable-auto/dovecot-2.2/ trusty universe Step 2. apt-get update Step 3. apt-get build-dep dovecot-2.2 Step 4. apt-get -b source -t trusty dovecot-2.2 I hope the process listed above is correct and that if I execute it I wont end up in a mess. Please comment. Thanks Kevin On Wed, Jan 28, 2015 at 11:02 PM, Kevin Laurie superinterstel...@gmail.com wrote: Dear Thomas, From what I have read is that I cannot automatically install dovecot, I would need to add the repos to my repo list and then use pining to install the latest version. Kindly advise if I am on correct track? Thanks Kevin On Wed, Jan 28, 2015 at 10:08 PM, Kevin Laurie superinterstel...@gmail.com wrote: Dear Thomas, Could you advise(or provide a link to) for a safe way to upgrade the Dovecot version. I am a little worried about testing and making trial-and-error approach as it could be risky. Appreciate if you could help with this. I have been trying to address this slow search issue for a while with very limited success(I was trying to implement FTS also), so I will appreciate if you could support. Apologies for my noobness. Thank You Kevin Do you have much ideas on how to upgrade dovecot. I am little worried about upgrading dovecot as there could be risks to data. I am trying to upgrade from 2.2.9 to 2.2.15 or Dovecot-EE. On Wed, Jan 28, 2015 at 9:16 PM, Thomas Leuxner t...@leuxner.net wrote: * superinterstel...@gmail.com superinterstel...@gmail.com 2015.01.28 12:07: Do you think I should try 2.2.15? Or stick to EE? This is in the repo. My version just has *all* the latest changes applied which sometime can be untested.
Re: Indexing Mail faster
Dear Thomas, From what I have read is that I cannot automatically install dovecot, I would need to add the repos to my repo list and then use pining to install the latest version. Kindly advise if I am on correct track? Thanks Kevin On Wed, Jan 28, 2015 at 10:08 PM, Kevin Laurie superinterstel...@gmail.com wrote: Dear Thomas, Could you advise(or provide a link to) for a safe way to upgrade the Dovecot version. I am a little worried about testing and making trial-and-error approach as it could be risky. Appreciate if you could help with this. I have been trying to address this slow search issue for a while with very limited success(I was trying to implement FTS also), so I will appreciate if you could support. Apologies for my noobness. Thank You Kevin Do you have much ideas on how to upgrade dovecot. I am little worried about upgrading dovecot as there could be risks to data. I am trying to upgrade from 2.2.9 to 2.2.15 or Dovecot-EE. On Wed, Jan 28, 2015 at 9:16 PM, Thomas Leuxner t...@leuxner.net wrote: * superinterstel...@gmail.com superinterstel...@gmail.com 2015.01.28 12 :07: Do you think I should try 2.2.15? Or stick to EE? This is in the repo. My version just has *all* the latest changes applied which sometime can be untested.
Re: Indexing Mail faster
* Kevin Laurie superinterstel...@gmail.com 2015.01.27 03:24: . search text dear * OK Searched 9% of the mailbox, ETA 1:39 * OK Searched 20% of the mailbox, ETA 1:18 * OK Searched 29% of the mailbox, ETA 1:12 * OK Searched 38% of the mailbox, ETA 1:03 * OK Searched 50% of the mailbox, ETA 0:49 * OK Searched 60% of the mailbox, ETA 0:39 * OK Searched 69% of the mailbox, ETA 0:30 * OK Searched 79% of the mailbox, ETA 0:20 I guess we need to start looking at your version and the format/mail_location configuration, best to post your 'doveconf -n'. Emulating the same search on my box looks a lot better: $ /usr/lib/dovecot/imap -u t...@leuxner.net * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NOTIFY QUOTA ACL RIGHTS=texk] Logged in as t...@leuxner.net a select Virtual/Postfix-Users * FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old \*)] Flags permitted. * 22120 EXISTS * 0 RECENT * OK [UNSEEN 22109] First unseen. * OK [UIDVALIDITY 1420123174] UIDs valid * OK [UIDNEXT 44062] Predicted next UID * OK [HIGHESTMODSEQ 702] Highest a OK [READ-WRITE] Select completed (0.031 secs). b search text dear * SEARCH 111 187 189 237 266 293 294 339 842 853 882 883 929 946 1346 1348 1349 1433 1434 1435 1474 1475 1483 1500 1504 1509 1512 1514 1639 1642 1766 2364 2366 2367 2635 2638 2639 2640 2642 2644 2664 2738 2750 2830 2849 2857 3264 3265 3266 3267 3268 3271 3274 3301 3302 3303 3305 3486 3489 3494 3499 3500 3752 3899 4013 4135 4136 4209 4421 4460 4464 4465 4469 4477 4478 4479 4480 4482 4483 4522 4535 4536 4538 4751 4769 5168 5169 5178 5179 5219 5254 5257 5300 5301 5304 5333 5336 5363 5364 5384 5406 5476 5479 5482 5493 5494 5499 5507 5516 5521 5768 6241 6243 7470 7471 7861 7862 7864 7874 7875 7876 7877 7879 8389 8399 8428 8429 8907 8937 8938 8940 9469 9495 9610 9688 9735 9881 9882 9883 9885 9913 9914 9915 9996 10009 10091 10094 10218 10229 10230 10332 10335 10340 10341 10517 10610 10612 10826 10829 10830 11081 11082 11087 11089 11097 11144 11145 11212 11213 11359 11365 11366 11367 11369 11375 11600 11673 11710 11941 11942 11946 11948 12046 12205 12208 12209 12407 12409 12411 13104 13551 13589 13806 13815 13818 13881 13883 13885 13980 14453 14578 14617 14618 14620 14666 14667 14668 14677 14685 14783 14790 14964 14966 14967 14972 15050 15436 15476 15949 15959 15964 15965 15971 16106 16110 16283 16284 16293 16310 16318 16331 16372 16373 16374 16785 16786 16787 16985 16987 17050 17855 17857 17871 17875 17876 17877 17959 17978 17979 18044 18807 18814 18821 18822 18823 18917 18918 18936 19005 19121 19291 19292 20058 20372 20467 20468 20469 20475 20489 20510 20516 20521 20522 20523 20524 20541 20560 20562 20582 20584 20653 20654 20904 21532 21595 21596 21716 21717 21719 21724 21725 * 7 RECENT b OK Search completed (4.468 secs). signature.asc Description: Digital signature
Re: Indexing Mail faster
Dear Thomas,
Below is the output of my 'dovecot -n' . The slow speed is really bad,
imagine waiting 4 mins for every search. . Something is terribly wrong in
my config.
Please have a look and advise.
Thanks
Kevin
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS
auth_master_user_separator = *
auth_mechanisms = PLAIN LOGIN
dict {
acl = mysql:/etc/dovecot/dovecot-share-folder.conf
quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf
}
first_valid_uid = 2000
last_valid_uid = 2000
listen = *
log_path = /var/log/dovecot.log
mail_debug = yes
mail_gid = 2000
mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
mail_plugins = quota fts
mail_uid = 2000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
namespace {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Sent Messages {
auto = no
special_use = \Sent
}
mailbox Spam {
auto = no
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = /
type = private
}
namespace {
list = children
location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u
prefix = Shared/%%u/
separator = /
subscriptions = yes
type = shared
}
passdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
passdb {
args = /etc/dovecot/dovecot-master-users-password
driver = passwd-file
master = yes
}
plugin {
acl = vfile
acl_shared_dict = proxy::acl
auth_socket_path = /var/run/dovecot/auth-master
quota = dict:user::proxy::quotadict
quota_rule = *:storage=1G
quota_warning = storage=85%% quota-warning 85 %u
quota_warning2 = storage=90%% quota-warning 90 %u
quota_warning3 = storage=95%% quota-warning 95 %u
sieve = /%Lh/sieve/dovecot.sieve
sieve_default = /var/vmail/sieve/dovecot.sieve
sieve_dir = /%Lh/sieve
sieve_global_dir = /var/vmail/sieve
}
protocols = pop3 imap sieve lmtp
service auth {
unix_listener /var/spool/postfix/private/dovecot-auth {
group = postfix
mode = 0666
user = postfix
}
unix_listener auth-master {
group = vmail
mode = 0666
user = vmail
}
unix_listener auth-userdb {
group = vmail
mode = 0660
user = vmail
}
}
service dict {
unix_listener dict {
group = vmail
mode = 0660
user = vmail
}
}
service imap-login {
process_limit = 500
service_count = 1
}
service lmtp {
executable = lmtp -L
inet_listener lmtp {
port = 24
}
process_min_avail = 5
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
user = vmail
}
service pop3-login {
service_count = 1
}
service quota-warning {
executable = script /usr/local/bin/dovecot-quota-warning.sh
unix_listener quota-warning {
group = vmail
mode = 0660
user = vmail
}
}
ssl = required
ssl_cert = /etc/ssl/certs/sicl.net.crt
ssl_key = /etc/ssl/private/sicl.net.key
userdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
lda_mailbox_autocreate = yes
log_path = /var/log/dovecot-sieve.log
mail_plugins = quota fts sieve
postmaster_address = root
}
protocol lmtp {
info_log_path = /var/log/dovecot-lmtp.log
lmtp_save_to_detail_mailbox = yes
mail_plugins = quota sieve
postmaster_address = postmaster
recipient_delimiter = +
}
protocol imap {
imap_client_workarounds = tb-extra-mailbox-sep
mail_plugins = quota fts imap_quota
}
protocol pop3 {
mail_plugins = quota fts
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %08Xu%08Xv
}
On Tue, Jan 27, 2015 at 4:02 PM, Thomas Leuxner t...@leuxner.net wrote:
* Kevin Laurie superinterstel...@gmail.com 2015.01.27 03:24:
. search text dear
* OK Searched 9% of the mailbox, ETA 1:39
* OK Searched 20% of the mailbox, ETA 1:18
* OK Searched 29% of the mailbox, ETA 1:12
* OK Searched 38% of the mailbox, ETA 1:03
* OK Searched 50% of the mailbox, ETA 0:49
* OK Searched 60% of the mailbox, ETA 0:39
* OK Searched 69% of the mailbox, ETA 0:30
* OK Searched 79% of the mailbox, ETA 0:20
I guess we need to start looking at your version and the
format/mail_location configuration, best to post your 'doveconf -n'.
Emulating the same search on my box looks a lot better:
$ /usr/lib/dovecot/imap -u t...@leuxner.net
* PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS
LIST-EXTENDED
Re: Indexing Mail faster
Dear Thomas,
I have removed the fts plugin.
See below:-
# dovecot -n
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS
auth_master_user_separator = *
auth_mechanisms = PLAIN LOGIN
dict {
acl = mysql:/etc/dovecot/dovecot-share-folder.conf
quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf
}
first_valid_uid = 2000
last_valid_uid = 2000
listen = *
log_path = /var/log/dovecot.log
mail_debug = yes
mail_gid = 2000
mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
mail_plugins = quota
mail_uid = 2000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
namespace {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Sent Messages {
auto = no
special_use = \Sent
}
mailbox Spam {
auto = no
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = /
type = private
}
namespace {
list = children
location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u
prefix = Shared/%%u/
separator = /
subscriptions = yes
type = shared
}
passdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
passdb {
args = /etc/dovecot/dovecot-master-users-password
driver = passwd-file
master = yes
}
plugin {
acl = vfile
acl_shared_dict = proxy::acl
auth_socket_path = /var/run/dovecot/auth-master
quota = dict:user::proxy::quotadict
quota_rule = *:storage=1G
quota_warning = storage=85%% quota-warning 85 %u
quota_warning2 = storage=90%% quota-warning 90 %u
quota_warning3 = storage=95%% quota-warning 95 %u
sieve = /%Lh/sieve/dovecot.sieve
sieve_default = /var/vmail/sieve/dovecot.sieve
sieve_dir = /%Lh/sieve
sieve_global_dir = /var/vmail/sieve
}
protocols = pop3 imap sieve lmtp
service auth {
unix_listener /var/spool/postfix/private/dovecot-auth {
group = postfix
mode = 0666
user = postfix
}
unix_listener auth-master {
group = vmail
mode = 0666
user = vmail
}
unix_listener auth-userdb {
group = vmail
mode = 0660
user = vmail
}
}
service dict {
unix_listener dict {
group = vmail
mode = 0660
user = vmail
}
}
service imap-login {
process_limit = 500
service_count = 1
}
service lmtp {
executable = lmtp -L
inet_listener lmtp {
port = 24
}
process_min_avail = 5
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
user = vmail
}
service pop3-login {
service_count = 1
}
service quota-warning {
executable = script /usr/local/bin/dovecot-quota-warning.sh
unix_listener quota-warning {
group = vmail
mode = 0660
user = vmail
}
}
ssl = required
ssl_cert = /etc/ssl/certs/sicl.net.crt
ssl_key = /etc/ssl/private/sicl.net.key
userdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
lda_mailbox_autocreate = yes
log_path = /var/log/dovecot-sieve.log
mail_plugins = quota sieve
postmaster_address = root
}
protocol lmtp {
info_log_path = /var/log/dovecot-lmtp.log
lmtp_save_to_detail_mailbox = yes
mail_plugins = quota sieve
postmaster_address = postmaster
recipient_delimiter = +
}
protocol imap {
imap_client_workarounds = tb-extra-mailbox-sep
mail_plugins = quota imap_quota
}
protocol pop3 {
mail_plugins = quota
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %08Xu%08Xv
}
The search is still the same even after removal. Do you think I have a
broken search dovecot?
. OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN
CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE QUOTA] Logged in
. select inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded
\*)] Flags permitted.
* 49255 EXISTS
* 30 RECENT
* OK [UNSEEN 46791] First unseen.
* OK [UIDVALIDITY 1414214135] UIDs valid
* OK [UIDNEXT 106490] Predicted next UID
* OK [NOMODSEQ] No permanent modsequences
. OK [READ-WRITE] Select completed (0.110 secs).
. search dear
. BAD Error in IMAP command SEARCH: Unknown argument DEAR
. search text hello
* OK Searched 8% of the mailbox, ETA 1:53
*
On Tue, Jan 27, 2015 at 6:47 PM, Thomas Leuxner t...@leuxner.net wrote:
* Kevin Laurie superinterstel...@gmail.com 2015.01.27 11:15:
Hi Kevin,
Re: Indexing Mail faster
* Kevin Laurie superinterstel...@gmail.com 2015.01.27 11:15: Hi Kevin, # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS Although this was released back in 2013, let's assume the core search is not substantially broken for this release. mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/ mail_plugins = quota fts As the mail storage looks okay, could you try with the parameter 'fts' removed? We would then have the search hit a vanilla Dovecot without plugins involved... Regards Thomas signature.asc Description: Digital signature
Re: Indexing Mail faster
Also tried to reload the config but still getting slow search.
Just tried to reindex the box by running
dovecot index -u u...@email.net inbox
I am still getting searches close to 2:45 minutes. It seems strange.
A smaller inbox took less time:-
. OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN
CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE QUOTA] Logged in
. search text hello
. BAD No mailbox selected.
. select inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags
permitted.
* 4825 EXISTS
* 66 RECENT
* OK [UNSEEN 3667] First unseen.
* OK [UIDVALIDITY 1414486440] UIDs valid
* OK [UIDNEXT 4826] Predicted next UID
* OK [NOMODSEQ] No permanent modsequences
. OK [READ-WRITE] Select completed (0.068 secs).
. search text hello
* OK Searched 50% of the mailbox, ETA 0:09
* OK Searched 95% of the mailbox, ETA 0:00
* SEARCH 20 21 22 23 24 32 39 40 41 42 45 48 49 50 59 60 270 373 428 576
585 624 749 756 770 776 798 829 849 917 927 928 934 971 981 983 984 1049
1062 1084 1095 1108 1186 1202 1203 1204 1212 1214 1278 1320 1345 1364 1365
1366 1367 1368 1384 1431 1456 1463 1480 1484 1485 1486 1507 1510 1525 1549
1550 1567 1574 1607 1608 1612 1613 1649 1701 1705 1711 1722 1765 1777 1826
1846 1856 1885 1889 1895 1983 1985 1986 2020 2022 2025 2064 2171 2175 2176
2177 2190 2224 2273 2274 2287 2299 2300 2313 2329 2338 2433 2434 2437 2440
2499 2574 2590 2595 2617 2618 2624 2625 2626 2627 2650 2652 2657 2701 2735
2785 2791 2792 2858 2872 2873 2875 2878 2929 3196 3242 3279 3325 3326 3327
3328 3345 3406 3408 3455 3456 3655 3679 3722 3723 3731 3792 3798 3812 3841
4085 4137 4192 4379 4708
. OK Search completed (21.478 secs).
On Wed, Jan 28, 2015 at 10:11 AM, Kevin Laurie superinterstel...@gmail.com
wrote:
Dear Thomas,
I have removed the fts plugin.
See below:-
# dovecot -n
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS
auth_master_user_separator = *
auth_mechanisms = PLAIN LOGIN
dict {
acl = mysql:/etc/dovecot/dovecot-share-folder.conf
quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf
}
first_valid_uid = 2000
last_valid_uid = 2000
listen = *
log_path = /var/log/dovecot.log
mail_debug = yes
mail_gid = 2000
mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
mail_plugins = quota
mail_uid = 2000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
namespace {
inbox = yes
location =
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Sent Messages {
auto = no
special_use = \Sent
}
mailbox Spam {
auto = no
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = /
type = private
}
namespace {
list = children
location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u
prefix = Shared/%%u/
separator = /
subscriptions = yes
type = shared
}
passdb {
args = /etc/dovecot/dovecot-mysql.conf
driver = sql
}
passdb {
args = /etc/dovecot/dovecot-master-users-password
driver = passwd-file
master = yes
}
plugin {
acl = vfile
acl_shared_dict = proxy::acl
auth_socket_path = /var/run/dovecot/auth-master
quota = dict:user::proxy::quotadict
quota_rule = *:storage=1G
quota_warning = storage=85%% quota-warning 85 %u
quota_warning2 = storage=90%% quota-warning 90 %u
quota_warning3 = storage=95%% quota-warning 95 %u
sieve = /%Lh/sieve/dovecot.sieve
sieve_default = /var/vmail/sieve/dovecot.sieve
sieve_dir = /%Lh/sieve
sieve_global_dir = /var/vmail/sieve
}
protocols = pop3 imap sieve lmtp
service auth {
unix_listener /var/spool/postfix/private/dovecot-auth {
group = postfix
mode = 0666
user = postfix
}
unix_listener auth-master {
group = vmail
mode = 0666
user = vmail
}
unix_listener auth-userdb {
group = vmail
mode = 0660
user = vmail
}
}
service dict {
unix_listener dict {
group = vmail
mode = 0660
user = vmail
}
}
service imap-login {
process_limit = 500
service_count = 1
}
service lmtp {
executable = lmtp -L
inet_listener lmtp {
port = 24
}
process_min_avail = 5
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user =
Re: Indexing Mail faster
Hi Thomas, That's very interesting. Below is my search result. Any idea why is my result so slow:- Appreciate if you could advise. Thanks Kevin . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE QUOTA] Logged in . select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded \*)] Flags permitted. * 49225 EXISTS * 73 RECENT * OK [UNSEEN 46791] First unseen. * OK [UIDVALIDITY 1414214135] UIDs valid * OK [UIDNEXT 106460] Predicted next UID * OK [NOMODSEQ] No permanent modsequences . OK [READ-WRITE] Select completed (0.548 secs). . search text dear * OK Searched 9% of the mailbox, ETA 1:39 * OK Searched 20% of the mailbox, ETA 1:18 * OK Searched 29% of the mailbox, ETA 1:12 * OK Searched 38% of the mailbox, ETA 1:03 * OK Searched 50% of the mailbox, ETA 0:49 * OK Searched 60% of the mailbox, ETA 0:39 * OK Searched 69% of the mailbox, ETA 0:30 * OK Searched 79% of the mailbox, ETA 0:20 On Sun, Jan 25, 2015 at 3:04 AM, Thomas Leuxner t...@leuxner.net wrote: * Kevin Laurie superinterstel...@gmail.com 2015.01.24 19:41: Currently the time it takes to search 25,000mails is 4mins. If indexed how much faster are we looking at? With a current version of Dovecot a search is pretty fast _without_ using external indexes. I have a view defined (virtual plugin) with around 22.000 messages in it, and searching the full view only takes 2.5 seconds: $ /usr/lib/dovecot/imap -u t...@leuxner.net * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NOTIFY QUOTA ACL RIGHTS=texk] Logged in as t...@leuxner.net a select Virtual/Postfix-Users * FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old \*)] Flags permitted. * 22094 EXISTS * 0 RECENT * OK [UIDVALIDITY 1420123174] UIDs valid * OK [UIDNEXT 44036] Predicted next UID * OK [HIGHESTMODSEQ 679] Highest a OK [READ-WRITE] Select completed (0.028 secs). b search charset utf-8 body SQL * SEARCH 51 52 54 83 87 102 108 126 172 173 195 223 234 238 249 250 252 254 260 263 268 274 293 294 316 317 324 343 412 414 431 432 434 435 436 438 439 440 441 442 443 450 451 454 457 458 464 478 493 496 497 505 590 617 619 625 628 632 636 637 638 639 644 645 646 650 655 657 659 671 676 677 699 702 757 759 762 767 769 770 790 791 792 793 796 799 839 840 841 843 844 845 846 847 848 849 850 864 868 882 905 907 929 953 955 959 1129 1133 1136 1137 1138 1139 1140 1143 1145 1168 1174 1175 1176 1177 1178 1179 1182 1205 1208 1244 1247 1248 1251 1316 1320 1334 1336 1352 1356 1392 1394 1395 1397 1398 1403 1407 1409 1410 1411 1433 1434 1435 1451 1468 1469 1470 1511 1512 1546 1547 1548 1560 1561 1562 1563 1567 1569 1570 1574 1578 1587 1588 1590 1601 1606 1648 1661 1669 1670 1671 1726 1748 1749 1750 1751 1752 1753 1755 1771 1773 1774 1792 1800 1808 1819 1822 1850 1885 1913 1932 1933 1980 1982 1999 2004 2026 2102 2104 2144 2145 2162 2184 2192 2193 2194 2204 2205 2217 2237 2272 2336 2337 2339 2341 2342 2343 2344 2346 2348 2368 2369 2380 2409 2426 2429 2455 2461 2499 2605 2607 2609 2611 2615 2618 2620 2635 2638 2639 2640 2642 2673 2784 2797 2799 2800 2821 2830 2849 2857 2863 2894 2895 2943 2944 2953 2954 2955 3049 3050 3052 3195 3244 3418 3419 3420 3421 3425 3465 3470 3474 3530 3543 3544 3546 3549 3550 3569 3572 3593 3602 3643 3647 3648 3649 3650 3654 3655 3656 3673 3674 3721 3722 3723 3724 3728 3740 3741 3772 3795 3797 3799 3803 3804 3806 3830 3832 3837 3838 3839 3840 3841 3859 3865 3866 3867 3875 3896 3898 3907 3915 3919 3924 3925 3927 3949 3951 4029 4031 4043 4073 4083 4084 4085 4087 4089 4091 4094 4096 4185 4186 4350 4354 4400 4401 4403 4404 4405 4406 4417 4419 4422 4423 4424 4425 4427 4431 4463 4501 4554 4556 4583 4615 4640 4642 4650 4683 4684 4686 4687 4688 4690 4691 4696 4697 4698 4699 4702 4705 4713 4714 4724 4725 4726 4727 4728 4729 4730 4734 4746 4754 4770 4772 4785 4787 4792 4794 4798 4800 4912 4922 4955 4956 4958 4966 4972 4973 4974 4975 4978 4979 4988 4990 4991 4992 4998 5009 5021 5025 5026 5027 5029 5030 5031 5036 5037 5109 5110 5111 5112 5149 5150 5151 5156 5159 5202 5248 5249 5365 5366 5367 5402 5428 5441 5442 5447 5449 5483 5484 5486 5487 5489 5493 5494 5495 5496 5497 5498 5499 5501 5502 5503 5504 5506 5508 5604 5681 5721 5722 5723 5724 5725 5727 5728 5729 5730 5731 5737 5738 5761 5778 5794 5796 5817 5818 5860 5863
Re: Indexing Mail faster
Thomas Leuxner: I have a view defined (virtual plugin) with around 22.000 messages in it interesting. I assume you drop multiple mailing lists together in a mailbox and separate them using the virtual plugin, right? could publish how you configured dovecot virtual plugin? I never had success on using the plugin in a useful way :-/ Thanks Andreas
Re: Indexing Mail faster
* A. Schulze s...@andreasschulze.de 2015.01.25 16:58:
once setup correctly it works like expected :-)
Good!
how do you separate older postings into the Archive?
something like doveadm search + move ?
Yes basically with move. I'd like to have them archived by year e.g.
Public/Archive/Mailing-Lists/Dovecot/2014. Being lazy when it comes to
repetitive tasks, I wrote a little script for it. It gives a good idea how to
do it (hopefully). Please note that this only has minimal error handling as I'm
setting most parameters in the script rather than the shell - butterfingers...
No bashism comments please :)
$ cat dovearchive.sh
#!/bin/bash
# Archive old posts before certain date to 'Public/Archive/Mailbox/Year'
# Set basic parameters accordingly:
# archive=2011, before_date=$year-mm-dd
# source_mailbox_base=Public/Newsletters
# dest_mailbox_base=Public/Archive/Newsletters
# Actual Mailbox is read from command line
set -e
archive=2014
let year=$archive+1
before_date=$year-01-01
mailbox_owner=t...@leuxner.net
source_mailbox_base='Public/Mailing-Lists'
dest_mailbox_base='Public/Archive/Mailing-Lists'
#acl_admin_group=owner
acl_admin_group='group=PublicMailboxAdmins'
acl_unlock_seq=$acl_admin_group delete expunge insert lookup post read write
write-seen write-deleted
acl_lock_seq=$acl_admin_group insert lookup post read write write-seen
acl_lock_archive=$acl_admin_group insert lookup read write write-seen
debug_acl_mailbox_path='path: '
msg_formatted() {
echo $(date +%b %d %H:%M:%S) $*
}
if [ $# -eq 0 ]; then
echo usage: $0 mailbox 2
exit 1
fi
# Mailbox exists?
doveadm acl get -u $mailbox_owner $source_mailbox_base/$1 || { echo 'Mailbox
not found.'; exit 1; }
# Create New Archive Mailbox
doveadm mailbox create -u $mailbox_owner $dest_mailbox_base/$1/$archive
# Modify ACL, expunge mail and revert ACL
msg_formatted [] Archiving \$dest_mailbox_base/$1/$archive\
doveadm acl set -u $mailbox_owner $source_mailbox_base/$1 $acl_unlock_seq
doveadm move -u $mailbox_owner $dest_mailbox_base/$1/$archive mailbox
$source_mailbox_base/$1 before $before_date
doveadm acl set -u $mailbox_owner $source_mailbox_base/$1 $acl_lock_seq
# Get actual path of dovecot-acl file and remove it as it is handled by global
ACL
acl_path=$(doveadm acl debug -u $mailbox_owner $source_mailbox_base/$1 21 |
sed -n s/\(.*\)$debug_acl_mailbox_path//p)
[ -f $acl_path/dovecot-acl ] rm $acl_path/dovecot-acl
msg_formatted '[ Complete ]'
signature.asc
Description: Digital signature
Re: Indexing Mail faster
Thomas Leuxner:
namespace {
location = virtual:~/mdbox/virtual
prefix = Virtual/
separator = /
}
$ cat virtual/Flagged/dovecot-virtual
*
Public/*
flagged
once setup correctly it works like expected :-)
Another example, the one I used in the original reply, is 'gluing'
together archives as a single view:
$ cat virtual/Postfix-Users/dovecot-virtual
Public/Archive/Mailing-Lists/Postfix-Users/*
Public/Mailing-Lists/Postfix-Users
all
how do you separate older postings into the Archive?
something like doveadm search + move ?
Andreas
Re: Indexing Mail faster
On Jan 25, 2015, at 5:05 AM, Thomas Leuxner t...@leuxner.net wrote:
The namespace Virtual will look for its configuration in the user's home
directory:
namespace {
location = virtual:~/mdbox/virtual
prefix = Virtual/
separator = /
}
Inside the ~/mdbox/virtual directory the actual query is defined in directory
'Flagged' (that will show up in the MUA as sub-dir). The actual search/query
is defined in file 'dovecot-virtual' which will create the magic:
$ cat virtual/Flagged/dovecot-virtual
*
Public/*
flagged
Another example, the one I used in the original reply, is 'gluing' together
archives as a single view:
$ cat virtual/Postfix-Users/dovecot-virtual
Public/Archive/Mailing-Lists/Postfix-Users/*
Public/Mailing-Lists/Postfix-Users
all
This all sounds interesting. Is there anything i can read to understand what
the purpose of Virtual is and why this would help? It sounds like you are using
a shared space for mailing lists (which I can see would be useful if you had
many people subscribing to the same lists). If so, doesn’t each user lose their
indications of what messages are read or replied to?
--
'But you ain't part of it, are you?' said Granny conversationally. 'You
try, but you always find yourself watchin' yourself watchin' people, eh?
Never quite believin' anything? Thinkin' the wrong thoughts?'
Dovecot Virtual Plugin Use Cases (was Re: Indexing Mail faster)
* LuKreme krem...@kreme.com 2015.01.25 19:32: Another example, the one I used in the original reply, is 'gluing' together archives as a single view: $ cat virtual/Postfix-Users/dovecot-virtual Public/Archive/Mailing-Lists/Postfix-Users/* Public/Mailing-Lists/Postfix-Users all This all sounds interesting. Is there anything i can read to understand what the purpose of Virtual is and why this would help? It sounds like you are using a shared space for mailing lists (which I can see would be useful if you had many people subscribing to the same lists). If so, doesn’t each user lose their indications of what messages are read or replied to? http://wiki2.dovecot.org/Plugins/Virtual It's a bit hard to wrap your head around it in the beginning, but once you understand what it can do, I'm sure you'll find a use case. Think of it as special view for all messages in Dovecot. You define which set of messages you want to display as a mailbox/folder. This view (or multiple views) is being updated dynamically going forward. For instance you can set up a certain filter/selection like you can with a modern MUA. The difference however is: your Virtual view(s) will be available regardless which client you are using, whether they have integrated filters capabilities - or not. signature.asc Description: Digital signature
Re: Indexing Mail faster
* Andreas Schulze s...@andreasschulze.de 2015.01.25 12:47:
I assume you drop multiple mailing lists together in a mailbox and separate
them using the virtual plugin, right?
could publish how you configured dovecot virtual plugin?
The namespace Virtual will look for its configuration in the user's home
directory:
namespace {
location = virtual:~/mdbox/virtual
prefix = Virtual/
separator = /
}
Inside the ~/mdbox/virtual directory the actual query is defined in directory
'Flagged' (that will show up in the MUA as sub-dir). The actual search/query is
defined in file 'dovecot-virtual' which will create the magic:
$ cat virtual/Flagged/dovecot-virtual
*
Public/*
flagged
Another example, the one I used in the original reply, is 'gluing' together
archives as a single view:
$ cat virtual/Postfix-Users/dovecot-virtual
Public/Archive/Mailing-Lists/Postfix-Users/*
Public/Mailing-Lists/Postfix-Users
all
HTH
signature.asc
Description: Digital signature
Re: Indexing Mail faster
* Kevin Laurie superinterstel...@gmail.com 2015.01.24 19:41: Currently the time it takes to search 25,000mails is 4mins. If indexed how much faster are we looking at? With a current version of Dovecot a search is pretty fast _without_ using external indexes. I have a view defined (virtual plugin) with around 22.000 messages in it, and searching the full view only takes 2.5 seconds: $ /usr/lib/dovecot/imap -u t...@leuxner.net * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NOTIFY QUOTA ACL RIGHTS=texk] Logged in as t...@leuxner.net a select Virtual/Postfix-Users * FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old \*)] Flags permitted. * 22094 EXISTS * 0 RECENT * OK [UIDVALIDITY 1420123174] UIDs valid * OK [UIDNEXT 44036] Predicted next UID * OK [HIGHESTMODSEQ 679] Highest a OK [READ-WRITE] Select completed (0.028 secs). b search charset utf-8 body SQL * SEARCH 51 52 54 83 87 102 108 126 172 173 195 223 234 238 249 250 252 254 260 263 268 274 293 294 316 317 324 343 412 414 431 432 434 435 436 438 439 440 441 442 443 450 451 454 457 458 464 478 493 496 497 505 590 617 619 625 628 632 636 637 638 639 644 645 646 650 655 657 659 671 676 677 699 702 757 759 762 767 769 770 790 791 792 793 796 799 839 840 841 843 844 845 846 847 848 849 850 864 868 882 905 907 929 953 955 959 1129 1133 1136 1137 1138 1139 1140 1143 1145 1168 1174 1175 1176 1177 1178 1179 1182 1205 1208 1244 1247 1248 1251 1316 1320 1334 1336 1352 1356 1392 1394 1395 1397 1398 1403 1407 1409 1410 1411 1433 1434 1435 1451 1468 1469 1470 1511 1512 1546 1547 1548 1560 1561 1562 1563 1567 1569 1570 1574 1578 1587 1588 1590 1601 1606 1648 1661 1669 1670 1671 1726 1748 1749 1750 1751 1752 1753 1755 1771 1773 1774 1792 1800 1808 1819 1822 1850 1885 1913 1932 1933 1980 1982 1999 2004 2026 2102 2104 2144 2145 2162 2184 2192 2193 2194 2204 2205 2217 2237 2272 2336 2337 2339 2341 2342 2343 2344 2346 2348 2368 2369 2380 2409 2426 2429 2455 2461 2499 2605 2607 2609 2611 2615 2618 2620 2635 2638 2639 2640 2642 2673 2784 2797 2799 2800 2821 2830 2849 2857 2863 2894 2895 2943 2944 2953 2954 2955 3049 3050 3052 3195 3244 3418 3419 3420 3421 3425 3465 3470 3474 3530 3543 3544 3546 3549 3550 3569 3572 3593 3602 3643 3647 3648 3649 3650 3654 3655 3656 3673 3674 3721 3722 3723 3724 3728 3740 3741 3772 3795 3797 3799 3803 3804 3806 3830 3832 3837 3838 3839 3840 3841 3859 3865 3866 3867 3875 3896 3898 3907 3915 3919 3924 3925 3927 3949 3951 4029 4031 4043 4073 4083 4084 4085 4087 4089 4091 4094 4096 4185 4186 4350 4354 4400 4401 4403 4404 4405 4406 4417 4419 4422 4423 4424 4425 4427 4431 4463 4501 4554 4556 4583 4615 4640 4642 4650 4683 4684 4686 4687 4688 4690 4691 4696 4697 4698 4699 4702 4705 4713 4714 4724 4725 4726 4727 4728 4729 4730 4734 4746 4754 4770 4772 4785 4787 4792 4794 4798 4800 4912 4922 4955 4956 4958 4966 4972 4973 4974 4975 4978 4979 4988 4990 4991 4992 4998 5009 5021 5025 5026 5027 5029 5030 5031 5036 5037 5109 5110 5111 5112 5149 5150 5151 5156 5159 5202 5248 5249 5365 5366 5367 5402 5428 5441 5442 5447 5449 5483 5484 5486 5487 5489 5493 5494 5495 5496 5497 5498 5499 5501 5502 5503 5504 5506 5508 5604 5681 5721 5722 5723 5724 5725 5727 5728 5729 5730 5731 5737 5738 5761 5778 5794 5796 5817 5818 5860 5863 5864 5865 5867 5989 5990 5991 6006 6028 6075 6078 6080 6081 6082 6083 6090 6092 6093 6103 6104 6115 6122 6193 6194 6197 6198 6235 6243 6258 6259 6260 6268 6312 6317 6321 6408 6409 6471 6472 6485 6546 6551 6620 6622 6623 6624 6630 6637 6652 6694 6698 6710 6768 6770 6772 6797 6851 6853 6868 6899 6900 6949 7092 7093 7095 7156 7219 7226 7227 7265 7278 7280 7281 7282 7283 7294 7295 7305 7307 7310 7317 7319 7327 7338 7341 7342 7358 7363 7365 7370 7378 7381 7382 7434 7447 7448 7449 7450 7452 7509 7516 7517 7518 7520 7521 7522 7523 7524 7525 7526 7544 7585 7614 7616 7617 7637 7644 7670 7671 7794 7795 7796 7885 7886 7887 7889 7895 7898 7900 7953 8016 8052 8054 8055 8056 8057 8059 8063 8064 8066 8069 8079 8119 8132 8156 8175 8232 8233 8234 8235 8236 8242 8289 8354 8420 8506 8507 8508 8511 8546 8622 8666 8667 8709 8873 8874 8875 8876 8877 8879 8887 8892 8898 8899 8927 8956 8957 8984 8988 8993 9056 9166 9207 9225 9226 9227 9229 9230 9231 9232 9262 9263 9264 9265 9266 9267 9268 9269 9272 9357 9366 9369 9371 9378 9381 9390 9392 9393 9402 9403 9409 9416 9417 9426 9430 9431 9432 9455 9458 9540 9553 9554 9574 9575 9577 9593 9599 9601 9610 9611 9612 9613 9618 9619 9625 9627 9641 9642 9659 9685 9694 9735 9737 9772 9773 9774 9775 9776 9777 9781 9795 9802 9803 9806 9827 9829 9844 9866 9867 9875 9914 9949 9954 9957 9958 9959 9960 9961 9962 9967 9968
