[
https://issues.apache.org/jira/browse/SOLR-8606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124873#comment-15124873
]
Jean-Baptiste Lespiau edited comment on SOLR-8606 at 1/30/16 12:23 PM:
-----------------------------------------------------------------------
Can someone move it to LUCENE ?
Here is a patch to add many more tests to the plugin, since its behavior is
underspecified. The tests passes, but may not be all "normal" behavior.
It uses a new class for debuging introduced by LUCENE-7003
was (Author: gueust):
Can someone move it to LUCENE ?
Here is a patch to add many more tests to the plugin, since its behavior is
underspecified. The tests passes, but may not be all "normal" behavior.
> Duplicate tokens using WordDelimiterFilter for a specific configuration
> -----------------------------------------------------------------------
>
> Key: SOLR-8606
> URL: https://issues.apache.org/jira/browse/SOLR-8606
> Project: Solr
> Issue Type: Bug
> Reporter: Jean-Baptiste Lespiau
> Priority: Minor
> Attachments: 0001-Adding-more-tests-to-WordDelimiterTK.patch,
> wdf-analysis.png
>
>
> When using both the options PRESERVE_ORIGINAL| SPLIT_ON_CASE_CHANGE and
> CONCATENATE_ALL|CATENATE_WORDS using the WordDelimiterFilter, we have
> duplicate tokens on strings contaning only case changes.
> When using the SPLIT_ON_CASE_CHANGE option, "abcDef" is split into "abc",
> "Def".
> When having PRESERVE_ORIGINAL, we keep "abcDef".
> However, when one uses CONCATENATE_ALL or CATENATE_WORDS, it also adds an
> other token built from the concatenation of the splited words, giving
> "abcDef" again.
> I'm not 100% certain that token filters should not produce duplicate tokens
> (same word, same start and end positions). Can someone confirm this is a bug ?
> I supply a patch that gives a test explosing the incorrect behavior.
> I'm willing to work on the following days to fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
