Re: index problem with only 1 folder of 1 box
Hello, Once again, sorry for my poor english skills... I upgrade dovecot to 2.2.13, and... the problem is always here. :-( So i purge (once again) solr entirely and reinstall it. But I forget to install the schema.xml in solr. I have only the original schema which is given with the package. == All is OK (but a bit slow : 5s against less than a second) I reinstall the schema.xml of dovecot, it's very fast but... I have the problem again... Now i know the source of the problem! ;) But i'm not able to configure correctly the schema at this moment... :-( So i will preserve the original schema for the moment. Good bye Guillaume Le 10/10/2014 14:46, Guillaume a écrit : Thank you for your answer. I will try to upgrade dovecot. By the way, I erase cache of solr ( deletequery*:*/query/delete ) and the problem is always here... When I will can upgrade dovecot, i will write back to say if its good or not. Sincerely yours, Guillaume Le 10/10/2014 14:15, Teemu Huovila a écrit : Hello On 10/10/2014 02:50 PM, Guillaume wrote: The biggest trouble for me is : Is it a solr problem or a dovecot problem? In my opinion, it's more a dovecot problem because the first research after a solr reindex give the good answer. If at all possible, you should try a newer version of Dovecot. There have been quite a few changes to FTS and the SOLR backend since version 2.2.9. See attached log for HG log of changes. Hope this helps. br, Teemu Huovila
Re: index problem with only 1 folder of 1 box
Hello, I retry. Nobody does have an idea about my index problem? The biggest trouble for me is : Is it a solr problem or a dovecot problem? In my opinion, it's more a dovecot problem because the first research after a solr reindex give the good answer. Can someone explains to me how dovecot creates and uses cache please? Once again, sorry for my poor english level. Thank you in advance Guillaume Le 07/10/2014 12:23, Guillaume a écrit : Hello, First of all, sorry for my poor english level. I pass to solr indexing. All is OK, except for the inbox of one of the mailboxes. For this inbox (other folders of the same mailbox have no problems), when i do text search, there is always no response. Others mailboxes have no problems. If i do a : doveadm -Dv fts rescan -u my...@domain.tld, i can do one and only one search with the correct response. It index the folder, take a few seconds and give me the good answer. If i redo the same search, it's extremly quick with no answers. It's like a cache problem but i don't know where and why... Logs are not very verbose... Here is an example of command line search : $ nc 127.0.0.1 143 * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Bienvenue sur le serveur IMAP de Domain.tld 10 LOGIN my...@domain.tld MyBigSecret 10 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE SEARCH=FUZZY QUOTA] Logged in 20 SELECT INBOX * FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-2 unknown-9 unknown-0 unknown-10 unknown-1 unknown-4 $MDNSent unknown-3 unknown-8 $Forwarded NonJunk) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-2 unknown-9 unknown-0 unknown-10 unknown-1 unknown-4 $MDNSent unknown-3 unknown-8 $Forwarded NonJunk \*)] Flags permitted. * 8437 EXISTS * 0 RECENT * OK [UIDVALIDITY 1412669180] UIDs valid * OK [UIDNEXT 8439] Predicted next UID * OK [HIGHESTMODSEQ 20] Highest 20 OK [READ-WRITE] Select completed (0.000 secs). 30 SEARCH text coucou * OK Indexed 14% of the mailbox, ETA 0:57 * OK Indexed 29% of the mailbox, ETA 0:47 * OK Indexed 72% of the mailbox, ETA 0:11 * OK Mailbox indexing finished * SEARCH 35 108 153 175 176 182 348 357 360 431 472 479 520 574 575 576 577 578 586 949 984 1025 1032 1033 1034 1087 1099 1129 1284 1313 1314 1315 1366 1372 1465 1538 1711 1726 1758 1759 1969 1981 1983 1984 2036 2077 2104 2116 2162 2181 2193 2437 2504 2587 2663 2808 2809 2854 2861 2862 3076 3087 3088 3137 3188 3189 3190 3191 3192 3193 3196 3197 3250 3251 3252 3327 3328 3329 3330 3331 3332 3379 3464 3481 3494 3521 3528 3582 3589 3593 3596 3651 3667 3672 3695 3787 3789 3939 3940 3941 3942 3943 3944 3945 3946 4019 4020 4034 4043 4134 4146 4175 4241 4346 4379 4422 4490 4491 4537 4540 4542 4543 4544 4550 4627 4727 4742 4767 4790 4817 4828 4831 4839 4862 4888 4899 4901 4906 4911 4916 4957 4999 5003 5004 5008 5009 5010 5011 5012 5013 5039 5062 5081 5092 5119 5138 5145 5150 5152 5185 5186 5187 5216 5246 5247 5282 5283 5314 5316 5320 5322 5325 5337 5353 5360 5363 5385 5432 5447 5472 5485 5496 5498 5500 5528 5551 5564 5567 5595 5598 5642 5794 5841 5896 5933 5960 5961 5994 6001 6005 6016 6017 6051 6075 6125 6152 6163 6215 6238 6239 6240 6241 6287 6311 6342 6350 6353 6357 6369 6431 6472 6689 6928 6945 6951 7026 7030 7045 7046 7115 7210 7212 7214 7230 7256 7295 7298 7340 7489 7490 7492 7654 7655 7658 7660 7676 7767 7768 7800 7801 7913 7937 7956 8075 8085 8102 8111 8144 8266 8329 8333 8394 8395 8431 30 OK Search completed (36.029 secs). 40 SEARCH text coucou * SEARCH 40 OK Search completed (0.002 secs). 50 SEARCH text blabla * SEARCH 50 OK Search completed (0.002 secs). 60 LOGOUT * BYE Logging out 60 OK Logout completed. To close this question, here is dovecot -n: # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.13.0-36-generic x86_64 Ubuntu 14.04.1 LTS ext4 auth_mechanisms = plain login debug_log_path = /var/log/dovecot-debug.log disable_plaintext_auth = no first_valid_uid = 5000 hostname = mailserver.domain.tld info_log_path = /var/log/dovecot-info.log last_valid_uid = 5000 lda_mailbox_autocreate = yes lda_original_recipient_header = X-Original-To listen = * log_path = /var/log/dovecot-err.log login_greeting = Bienvenue sur le serveur IMAP de Domain.tld mail_debug = yes mail_gid = 5000 mail_location = maildir:/data/mails/%d/%n mail_plugins = quota fts fts_solr mail_uid = 5000 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body
Re: index problem with only 1 folder of 1 box
Hello On 10/10/2014 02:50 PM, Guillaume wrote: The biggest trouble for me is : Is it a solr problem or a dovecot problem? In my opinion, it's more a dovecot problem because the first research after a solr reindex give the good answer. If at all possible, you should try a newer version of Dovecot. There have been quite a few changes to FTS and the SOLR backend since version 2.2.9. See attached log for HG log of changes. Hope this helps. br, Teemu Huovila 2014-09-16 15:23 +0300 Timo Sirainen t...@iki.fi (8c2cb7d01a78) * src/plugins/fts-solr/fts-backend-solr.c, src/plugins/fts/fts-api- private.h, src/plugins/fts/fts-api.c: doveadm fts rescan: For virtual namespaces just mark the last indexed UID to 0. 2014-09-16 14:32 +0300 Timo Sirainen t...@iki.fi (e82ad7f1c58f) * src/plugins/fts/fts-expunge-log.c: fts: dovecot-expunges.log wasn't closed at deinit 2014-08-08 16:27 +0300 Timo Sirainen t...@iki.fi (1ea3da40ea8f) * src/plugins/fts/fts-storage.c: fts: fts_no_autofuzzy shouldn't disable fuzzying when FUZZY search parameter is set. 2014-08-08 16:20 +0300 Timo Sirainen t...@iki.fi (cdf4edcc6256) * src/plugins/fts-lucene/fts-backend-lucene.c, src/plugins/fts-lucene /lucene-wrapper.cc, src/plugins/fts-lucene/lucene-wrapper.h, src/plugins/fts-solr/fts-backend-solr-old.c, src/plugins/fts-solr /fts-backend-solr.c, src/plugins/fts-squat/fts-backend-squat.c, src/plugins/fts/fts-api-private.h, src/plugins/fts/fts-api.c, src/plugins/fts/fts-api.h, src/plugins/fts/fts-search.c, src/plugins/fts/fts-storage.c, src/plugins/fts/fts-storage.h: fts: Added fts_no_autofuzzy setting to require exact matches for found results. This is done by using the FTS search results as only filters on which the regular non-FTS search is done. 2014-07-03 14:37 +0300 Timo Sirainen t...@iki.fi (9c6643daae98) * src/plugins/fts/fts-expunge-log.c: fts: If we detect corrupted fts expunge log, unlink it. This avoids the same error repeating forever. 2014-06-30 17:25 +0300 Timo Sirainen t...@iki.fi (2c2b94840ff3) * src/plugins/fts/fts-parser-tika.c: fts-tika: Hiden Unsupported Media Type errors. Log HTTP status code on errors. 2014-06-30 16:41 +0300 Timo Sirainen t...@iki.fi (49dfc6da1786) * src/plugins/fts/fts-parser-tika.c: fts-tika: Fixed crash if Tika returned 200 reply without payload. 2014-06-16 15:35 +0300 Timo Sirainen t...@iki.fi (fc40b1a6e962) * src/plugins/fts/xml2text.c: xml2text: Check for read()/write() failures and exit if they fail. 2014-06-13 02:19 +0300 Timo Sirainen t...@iki.fi (b67c1c9bf1a5) * src/auth/mech-winbind.c, src/auth/userdb-passwd-file.c, src/config /config-parser.c, src/doveadm/doveadm-director.c, src/doveadm /doveadm-dump-dbox.c, src/doveadm/doveadm-log.c, src/doveadm /doveadm-penalty.c, src/doveadm/doveadm-replicator.c, src/doveadm /doveadm-stats.c, src/doveadm/doveadm-who.c, src/doveadm/doveadm- zlib.c, src/lib-compression/test-compression.c, src/lib-imap-urlauth /imap-urlauth-connection.c, src/lib-lda/smtp-client.c, src/lib- master/master-instance.c, src/lib-master/mountpoint-list.c, src/lib- settings/settings-parser.c, src/lib-settings/settings.c, src/lib- storage/index/cydir/cydir-mail.c, src/lib-storage/index/dbox-common /dbox-file.c, src/lib-storage/index/imapc/imapc-save.c, src/lib- storage/index/maildir/maildir-mail.c, src/lib-storage/index/raw/raw- storage.c, src/lib-storage/list/subscription-file.c, src/lib /iostream-temp.c, src/lib/istream-seekable.c, src/plugins/fts/fts- parser-script.c, src/plugins/zlib/zlib-plugin.c, src/replication/replicator/replicator-queue.c, src/ssl- params/main.c, src/util/rawlog.c: Use the new [io]_stream_create_fd_*autoclose() functions wherever possible. 2014-06-13 01:11 +0300 Timo Sirainen t...@iki.fi (54f1beb8d071) * src/plugins/fts/doveadm-dump-fts-expunge-log.c: fts: Improved doveadm fts dump for corrupted expunge log Although we may still be trying to allocate up to 2 GB of memory, but at least no more than that now. Found by Coverity 2014-06-13 00:46 +0300 Timo Sirainen t...@iki.fi (0fc86de05ccf) * src/plugins/fts/doveadm-dump-fts-expunge-log.c: fts: Minor code cleanup: Don't increment NULL pointer. 2014-05-27 21:17 +0300 Phil Carmody p...@dovecot.fi (ad028a950248) * src/plugins/fts/fts-parser-html.c: fts: parser-html - parser can fail on attributes='with values in single quotes' If that value were to contain an odd number of double quotes, then the HTML_STATE_TAG_(D)QUOTED state would be entered and not exited. The two quoting types behave basically the same, so just add two new cases and duplicate the state transition code. 2014-05-27 21:17 +0300 Phil Carmody p...@dovecot.fi (54e508b71dcd) * src/plugins/fts/fts-parser-html.c: fts: parser-html - parse_tag_name returns wrong value for comments This function returns 1 more than the number of additional characters to be swallowed up by the state transition. 2014-05-27 21:17 +0300 Phil
Re: index problem with only 1 folder of 1 box
Thank you for your answer. I will try to upgrade dovecot. By the way, I erase cache of solr ( deletequery*:*/query/delete ) and the problem is always here... When I will can upgrade dovecot, i will write back to say if its good or not. Sincerely yours, Guillaume Le 10/10/2014 14:15, Teemu Huovila a écrit : Hello On 10/10/2014 02:50 PM, Guillaume wrote: The biggest trouble for me is : Is it a solr problem or a dovecot problem? In my opinion, it's more a dovecot problem because the first research after a solr reindex give the good answer. If at all possible, you should try a newer version of Dovecot. There have been quite a few changes to FTS and the SOLR backend since version 2.2.9. See attached log for HG log of changes. Hope this helps. br, Teemu Huovila
index problem with only 1 folder of 1 box
Hello,
First of all, sorry for my poor english level.
I pass to solr indexing.
All is OK, except for the inbox of one of the mailboxes. For this inbox
(other folders of the same mailbox have no problems), when i do text
search, there is always no response.
Others mailboxes have no problems.
If i do a : doveadm -Dv fts rescan -u my...@domain.tld, i can do one
and only one search with the correct response. It index the folder, take
a few seconds and give me the good answer.
If i redo the same search, it's extremly quick with no answers. It's
like a cache problem but i don't know where and why...
Logs are not very verbose...
Here is an example of command line search :
$ nc 127.0.0.1 143
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Bienvenue sur le serveur IMAP de
Domain.tld
10 LOGIN my...@domain.tld MyBigSecret
10 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS
THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH
ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY
MOVE SEARCH=FUZZY QUOTA] Logged in
20 SELECT INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-2 unknown-9
unknown-0 unknown-10 unknown-1 unknown-4 $MDNSent unknown-3 unknown-8
$Forwarded NonJunk)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-2
unknown-9 unknown-0 unknown-10 unknown-1 unknown-4 $MDNSent unknown-3
unknown-8 $Forwarded NonJunk \*)] Flags permitted.
* 8437 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1412669180] UIDs valid
* OK [UIDNEXT 8439] Predicted next UID
* OK [HIGHESTMODSEQ 20] Highest
20 OK [READ-WRITE] Select completed (0.000 secs).
30 SEARCH text coucou
* OK Indexed 14% of the mailbox, ETA 0:57
* OK Indexed 29% of the mailbox, ETA 0:47
* OK Indexed 72% of the mailbox, ETA 0:11
* OK Mailbox indexing finished
* SEARCH 35 108 153 175 176 182 348 357 360 431 472 479 520 574 575 576
577 578 586 949 984 1025 1032 1033 1034 1087 1099 1129 1284 1313 1314
1315 1366 1372 1465 1538 1711 1726 1758 1759 1969 1981 1983 1984 2036
2077 2104 2116 2162 2181 2193 2437 2504 2587 2663 2808 2809 2854 2861
2862 3076 3087 3088 3137 3188 3189 3190 3191 3192 3193 3196 3197 3250
3251 3252 3327 3328 3329 3330 3331 3332 3379 3464 3481 3494 3521 3528
3582 3589 3593 3596 3651 3667 3672 3695 3787 3789 3939 3940 3941 3942
3943 3944 3945 3946 4019 4020 4034 4043 4134 4146 4175 4241 4346 4379
4422 4490 4491 4537 4540 4542 4543 4544 4550 4627 4727 4742 4767 4790
4817 4828 4831 4839 4862 4888 4899 4901 4906 4911 4916 4957 4999 5003
5004 5008 5009 5010 5011 5012 5013 5039 5062 5081 5092 5119 5138 5145
5150 5152 5185 5186 5187 5216 5246 5247 5282 5283 5314 5316 5320 5322
5325 5337 5353 5360 5363 5385 5432 5447 5472 5485 5496 5498 5500 5528
5551 5564 5567 5595 5598 5642 5794 5841 5896 5933 5960 5961 5994 6001
6005 6016 6017 6051 6075 6125 6152 6163 6215 6238 6239 6240 6241 6287
6311 6342 6350 6353 6357 6369 6431 6472 6689 6928 6945 6951 7026 7030
7045 7046 7115 7210 7212 7214 7230 7256 7295 7298 7340 7489 7490 7492
7654 7655 7658 7660 7676 7767 7768 7800 7801 7913 7937 7956 8075 8085
8102 8111 8144 8266 8329 8333 8394 8395 8431
30 OK Search completed (36.029 secs).
40 SEARCH text coucou
* SEARCH
40 OK Search completed (0.002 secs).
50 SEARCH text blabla
* SEARCH
50 OK Search completed (0.002 secs).
60 LOGOUT
* BYE Logging out
60 OK Logout completed.
To close this question, here is dovecot -n:
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.13.0-36-generic x86_64 Ubuntu 14.04.1 LTS ext4
auth_mechanisms = plain login
debug_log_path = /var/log/dovecot-debug.log
disable_plaintext_auth = no
first_valid_uid = 5000
hostname = mailserver.domain.tld
info_log_path = /var/log/dovecot-info.log
last_valid_uid = 5000
lda_mailbox_autocreate = yes
lda_original_recipient_header = X-Original-To
listen = *
log_path = /var/log/dovecot-err.log
login_greeting = Bienvenue sur le serveur IMAP de Domain.tld
mail_debug = yes
mail_gid = 5000
mail_location = maildir:/data/mails/%d/%n
mail_plugins = quota fts fts_solr
mail_uid = 5000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
passdb {
driver = pam
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
fts = solr
fts_autoindex = yes
fts_solr = break-imap-search url=http://localhost:8080/solr/
quota = maildir:User quota
sieve = /home/vmail/managesieve/.%n@%d.sieve
sieve_dir = /home/vmail/managesieve/%d/%n
sieve_global_dir = /home/vmail/sieve/
}
