[
https://issues.apache.org/jira/browse/LUCENE-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487322#comment-16487322
]
ASF subversion and git services commented on LUCENE-8325:
---------------------------------------------------------
Commit 55858d7ba72f857ded79035430855e511a8e319d in lucene-solr's branch
refs/heads/master from [~jimczi]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=55858d7 ]
LUCENE-8325: Fixed the smartcn tokenizer to not split UTF-16 surrogate pairs.
> smartcn analyzer can't handle SURROGATE char
> --------------------------------------------
>
> Key: LUCENE-8325
> URL: https://issues.apache.org/jira/browse/LUCENE-8325
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: chengpohi
> Priority: Minor
> Labels: newbie, patch
> Attachments: handle_surrogate_char_for_smartcn_2018-05-23.patch
>
>
> This issue is from [https://github.com/elastic/elasticsearch/issues/30739]
> smartcn analyzer can't handle SURROGATE char, Example:
>
>
> {code:java}
> Analyzer ca = new SmartChineseAnalyzer();
> String sentence = "\uD862\uDE0F"; // 𨨏 a surrogate char
> TokenStream tokenStream = ca.tokenStream("", sentence);
> CharTermAttribute charTermAttribute =
> tokenStream.addAttribute(CharTermAttribute.class);
> tokenStream.reset();
> while (tokenStream.incrementToken()) {
> String term = charTermAttribute.toString();
> System.out.println(term);
> }
> {code}
>
> In the above code snippet will output:
>
> {code:java}
> ?
> ?
> {code}
>
> and I have created a *PATCH* to try to fix this, please help review(since
> *smartcn* only support *GBK* char, so it's only just handle it as a *single
> char*).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
