[
https://issues.apache.org/jira/browse/LUCENE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781477#comment-13781477
]
Uwe Schindler edited comment on LUCENE-5240 at 9/29/13 7:02 PM:
----------------------------------------------------------------
OK! Thanks for removing TrieTokencorrumpter! Just one question: Is the
TrieTokenizer really not needed for highlighting? I know, the TrieField is now
"tokenized=false", but I have no idea how this affects the highlighter. On the
other hand, highlighting a trie field is completely useless :-)
I know, ElasticSearch has cloned TrieTokenizer and it uses it for
highlighting... Adrien knows more, he did that porting.
For 4.x, we have to add TrieTokenizerFactory to the backwards breaks, because
somebody might have used it... We should suggest to use a real TrieField in
that case!
was (Author: thetaphi):
OK! Thanks for removing TrieTokencorrumpter! Just one question: If the
TrieTokenizer really not needed for highlighting? I know, the TrieField is now
"tokenized=false", but I have no idea how this affects the highlighter. On the
other hand, highlighting a trie field is completely useless :-)
I know, ElasticSearch has cloned TrieTokenizer and it uses it for
highlighting...
For 4.x, we have to add TrieTokenizerFactory to the backwards breaks, because
somebody might have used it... We should suggest to use a real TrieField in
that case!
> additional safety in Tokenizer state machine
> --------------------------------------------
>
> Key: LUCENE-5240
> URL: https://issues.apache.org/jira/browse/LUCENE-5240
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Robert Muir
> Attachments: LUCENE-5240.patch, LUCENE-5240.patch, LUCENE-5240.patch
>
>
> {code}
> * <b>NOTE:</b>
> * The default implementation closes the input Reader, so
> * be sure to call <code>super.close()</code> when overriding this method.
> */
> @Override
> public void close() throws IOException {
> {code}
> We can add a simple check for this easily now in setReader. I found a few
> bugs, and fixed all except TrieTokenizer in solr (I am lost here... somewhere
> i have a patch to remove this thing).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
