[
https://issues.apache.org/jira/browse/LUCENE-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15551382#comment-15551382
]
ASF subversion and git services commented on LUCENE-7468:
---------------------------------------------------------
Commit 28d187acd1e391723eb6e1b5445f22abf5580a80 in lucene-solr's branch
refs/heads/master from [~jpountz]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=28d187a ]
LUCENE-7468: ASCIIFoldingFilter should not emit duplicated tokens when preserve
original is on.
> ASCIIFoldingFilter should not emit duplicated tokens when preserve original
> is on
> ---------------------------------------------------------------------------------
>
> Key: LUCENE-7468
> URL: https://issues.apache.org/jira/browse/LUCENE-7468
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
> Affects Versions: 4.7, 5.x, trunk, 6.x
> Reporter: David Causse
> Attachments: LUCENE-7468.patch
>
>
> The ASCIIFoldingFilter seems to make the bold assumption that any tokens that
> contain a char outside the ASCII range will be folded.
> The problem is that when preserve original is true we capture and restore the
> state even if the token remains unmodified.
> This causes term frequencies to double for such words and probably extra
> space used when positions/offsets are stored in the postings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
