[
https://issues.apache.org/jira/browse/OPENNLP-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118093#comment-18118093
]
ASF GitHub Bot commented on OPENNLP-1974:
-----------------------------------------
rzo1 commented on PR #1313:
URL: https://github.com/apache/opennlp/pull/1313#issuecomment-5781275010
@krickert, two questions about the bundled emoji data before we merge:
1. **Diff in the bundled data:** The description originally described an
Emoji 17.0 inventory of 3,944 sequences. The head now ships
`EmojiSequences-18.0.txt` with 3,963 sequences, pulled from the Unicode 18.0.0
files with new pinned checksums and a 2026 copyright year. What's in those 19
extra sequences, and did anything else in the inventory or the 10 component
ranges change between 17.0 and 18.0? Could you post the generator's 17.0 → 18.0
diff here so reviewers can check it?
2. **Scope:** Should the Emoji 17 → 18 upgrade be in this PR at all? This is
a behavior-fix ticket for `EmojiCharSequenceNormalizer`, and the spec bump went
in without a mention in the description or the commit messages. It also leaves
us on mixed versions, since `WordBreakProperty`, `ExtendedPictographic` and the
emoji annotations are still on 17.0. I'd lean towards landing the scanner on
17.0 here and doing the 18.0 upgrade for all bundled Unicode data in a separate
issue, so the language-detector evaluation measures only the normalizer change.
Is there a reason it needs to be 18.0 in this PR?
> RegEx removal part 8: EmojiCharSequenceNormalizer blanks hyphens and BMP
> characters
> -----------------------------------------------------------------------------------
>
> Key: OPENNLP-1974
> URL: https://issues.apache.org/jira/browse/OPENNLP-1974
> Project: OpenNLP
> Issue Type: Bug
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1973.
> The pattern of {{EmojiCharSequenceNormalizer}} was parsed as the range
> U\+D83C to U\+10FC00 plus a literal hyphen, so it also blanked the hyphen and
> the BMP characters from U\+D83C to U\+FFFF \(fullwidth and halfwidth forms,
> CJK compatibility ideographs, Arabic presentation forms, the private use
> area, U\+FE0F, unpaired surrogates\) and kept U\+10FC01 to U\+10FFFF.
> The normalizer now replaces each run of supplementary code points \(U\+10000
> and above\) with one space, using a code point scan instead of the regular
> expression. It is part of the default language detector chain, so models
> trained with an earlier release see different input; under
> {{\-Dopennlp.compat.mode=LEGACY}} the earlier output is produced.
> Before merge: language detector eval build.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
