[
https://issues.apache.org/jira/browse/OPENNLP-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121290#comment-18121290
]
ASF GitHub Bot commented on OPENNLP-1974:
-----------------------------------------
rzo1 commented on PR #1313:
URL: https://github.com/apache/opennlp/pull/1313#issuecomment-5917437319
Note for the 3.0.0 release notes: since `CompatibilityMode` defaults to
`CURRENT` from 3.0.0 on, the `EmojiCharSequenceNormalizer` in the default
language detector chain produces different features than the one
`langdetect-183.bin` was trained with (hyphens, fullwidth forms, private use
and non-emoji supplementary characters are no longer stripped). So
`langdetect-183.bin` needs to be re-trained for 3.0.0 ;-)
Until then, users of the existing model can get the old features back with
`-Dopennlp.compat.mode=LEGACY`.
Also worth noting: `opennlp-eval-tests` has no language detector eval, so
the green eval run doesn't cover this change.
> RegEx removal part 8: EmojiCharSequenceNormalizer blanks hyphens and BMP
> characters
> -----------------------------------------------------------------------------------
>
> Key: OPENNLP-1974
> URL: https://issues.apache.org/jira/browse/OPENNLP-1974
> Project: OpenNLP
> Issue Type: Bug
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1973.
> The pattern of {{EmojiCharSequenceNormalizer}} was parsed as the range
> U\+D83C to U\+10FC00 plus a literal hyphen, so it also blanked the hyphen and
> the BMP characters from U\+D83C to U\+FFFF \(fullwidth and halfwidth forms,
> CJK compatibility ideographs, Arabic presentation forms, the private use
> area, U\+FE0F, unpaired surrogates\) and kept U\+10FC01 to U\+10FFFF.
> The normalizer now replaces each run of supplementary code points \(U\+10000
> and above\) with one space, using a code point scan instead of the regular
> expression. It is part of the default language detector chain, so models
> trained with an earlier release see different input; under
> {{\-Dopennlp.compat.mode=LEGACY}} the earlier output is produced.
> Before merge: language detector eval build.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
