[
https://issues.apache.org/jira/browse/LUCENE-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605483#action_12605483
]
Hiroaki Kawai commented on LUCENE-1306:
---------------------------------------
After thinking for a week, I think this idea is nice.
IMHO, this might be renamed to NGramTokenizer simply. A general n-gram
tokenizer accepts a sequence that has no gap in it. By the concept, TokenFilter
accepts a tokien stream (gapped sequence), and current NGramTokenFilter does
not work well in that sense. CombinedNGramTokenFilter filles the gap with
prefix(^) and suffix($), and the token stream becomes a simple stream again
virtually, n-gram works nice agian.
Comments:
1. prefix and suffix chars should be configurable. Because user must choose a
char that is not used in the terms.
2. prefix and suffix might be a white space. Because most of the users are not
interested in whitespace itself.
3. If you want to do a phrase query (for example, "This is"), we have to
generate $^ token in the gap to make the positions valid.
4. n-gram algorithm should be rewritten to make the positions valid. Please see
LUCENE-1225
I think "^h" is OK, because prefix and suffix are the chars that was introduced
as a workaround.
> CombinedNGramTokenFilter
> ------------------------
>
> Key: LUCENE-1306
> URL: https://issues.apache.org/jira/browse/LUCENE-1306
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/analyzers
> Reporter: Karl Wettin
> Assignee: Karl Wettin
> Priority: Trivial
> Attachments: LUCENE-1306.txt
>
>
> Alternative NGram filter that produce tokens with composite prefix and suffix
> markers.
> {code:java}
> ts = new WhitespaceTokenizer(new StringReader("hello"));
> ts = new CombinedNGramTokenFilter(ts, 2, 2);
> assertNext(ts, "^h");
> assertNext(ts, "he");
> assertNext(ts, "el");
> assertNext(ts, "ll");
> assertNext(ts, "lo");
> assertNext(ts, "o$");
> assertNull(ts.next());
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
