[
https://issues.apache.org/jira/browse/SOLR-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461573#comment-13461573
]
Lance Norskog commented on SOLR-3653:
-------------------------------------
Another note: one trigram is the number 15. There are several conventions for
representing integers, including regional quirks. There is no 'number
canonicalizer' in the Smart Chinese toolkit. This could be a problem with
formal documents: historical, government docs, treaties and the like.
[http://en.wikipedia.org/wiki/Chinese_numerals#Whole_numbers]
> Custom bigramming filter for to handle Smart Chinese edge cases
> ---------------------------------------------------------------
>
> Key: SOLR-3653
> URL: https://issues.apache.org/jira/browse/SOLR-3653
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Lance Norskog
> Attachments: SmartChineseType.pdf, SOLR-3653.patch,
> translations_450.five2thirteen.txt, translations_first_500.quad.txt,
> translations_first_500.trigrams.txt
>
>
> The "Smart" Simplified Chinese toolkit in lucene/analysis/smartcn does not
> work in some edge cases. It fails to split certain words which were not part
> of the dictionary or training corpus.
> This patch supplies a bigramming class to handle these occasional mistakes.
> The algorithm creates bigrams out of all "words" longer than two ideograms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
