[
https://issues.apache.org/jira/browse/LUCENE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand updated LUCENE-4955:
---------------------------------
Attachment: LUCENE-4955.patch
I tried to iterate on Simon's patch:
* NGramTokenFilter doesn't modify offsets and emits all n-grams of a single
term at the same position
* NGramTokenizer uses a sliding window.
* NGramTokenizer and NGramTokenFilter removed from TestRandomChains exclusions.
It was very hard to add the compatibility version support to NGramTokenizer so
there are now two distinct classes and the factory picks the right one
depending on the Lucene match version.
Simon's highlighting test now fails because the highlighted content is
different, but not because of a broken token stream.
> NGramTokenFilter increments positions for each gram
> ---------------------------------------------------
>
> Key: LUCENE-4955
> URL: https://issues.apache.org/jira/browse/LUCENE-4955
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
> Affects Versions: 4.3
> Reporter: Simon Willnauer
> Fix For: 5.0, 4.4
>
> Attachments: highlighter-test.patch, highlighter-test.patch,
> LUCENE-4955.patch, LUCENE-4955.patch
>
>
> NGramTokenFilter increments positions for each gram rather for the actual
> token which can lead to rather funny problems especially with highlighting.
> if this filter should be used for highlighting is a different story but today
> this seems to be a common practice in many situations to highlight sub-term
> matches.
> I have a test for highlighting that uses ngram failing with a StringIOOB
> since tokens are sorted by position which causes offsets to be mixed up due
> to ngram token filter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
