[
https://issues.apache.org/jira/browse/LUCENE-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771525#action_12771525
]
Michael McCandless commented on LUCENE-2016:
--------------------------------------------
bq. Finally, I completely disagree with the nontrivial performance comment. The
trick is to make sure the execution branch / checks for the process-internal
characters outside the bmp, only occurs for surrogate pairs. They are
statistically very rare and if done right, it will not affect performance of
BMP content.
OK I agree, you're right: we could in fact do this with negligible impact to
performance.
bq. Its my understanding Lucene indexes should be portable to different
programming languages: perhaps my implementation in C/perl/python decides to
use a different process-internal character, this is allowed by Unicode and I
think we should adhere to it, I don't think its being anal.
But if we forcefully map all invalid-for-interchange unicode characters to the
replacement character (I think that's what's being proposed, right?), then your
app no longer has any characters it can use for its own "internal" purposes?
Can you open a new issue to track this? This is a wider discussion than
preventing index corruption :)
> replace invalid U+FFFF character during indexing
> ------------------------------------------------
>
> Key: LUCENE-2016
> URL: https://issues.apache.org/jira/browse/LUCENE-2016
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.4, 2.4.1, 2.9
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.9.1, 3.0
>
> Attachments: LUCENE-2016.patch
>
>
> If the invalid U+FFFF character is embedded in a token, it actually causes
> indexing to silently corrupt the index by writing duplicate terms into the
> terms dict. CheckIndex will catch the error, and merging will hit exceptions
> (I think).
> We already replace invalid surrogate pairs with the replacement character
> U+FFFD, so I'll just do the same with U+FFFF.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
