[
https://issues.apache.org/jira/browse/OPENNLP-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118096#comment-18118096
]
ASF GitHub Bot commented on OPENNLP-1974:
-----------------------------------------
krickert commented on PR #1313:
URL: https://github.com/apache/opennlp/pull/1313#issuecomment-5781466262
> @krickert, two questions about the bundled emoji data before we merge:
>
> 1. **Diff in the bundled data:** The description originally described an
Emoji 17.0 inventory of 3,944 sequences. The head now ships
`EmojiSequences-18.0.txt` with 3,963 sequences, pulled from the Unicode 18.0.0
files with new pinned checksums and a 2026 copyright year. What's in those 19
extra sequences, and did anything else in the inventory or the 10 component
ranges change between 17.0 and 18.0? Could you post the generator's 17.0 → 18.0
diff here so reviewers can check it?
I'll do a total audit to see what the dependency is. I remember seeing most
of them were thumbs of multiple colors, but I'll double check
> 2. **Scope:** Should the Emoji 17 → 18 upgrade be in this PR at all?
I think so, but totally not important (I don't think this is a heavily used
feature, but will become an incredibly helpful one in the future).
But any pushback on emjois I won't battle :) No problem - 17 is coming
back. Made it a draft for a quick sec to make sure the other nits are still in
though...
I thought of it as a quick property update, but even that should be it's own
PR. Noted.
> RegEx removal part 8: EmojiCharSequenceNormalizer blanks hyphens and BMP
> characters
> -----------------------------------------------------------------------------------
>
> Key: OPENNLP-1974
> URL: https://issues.apache.org/jira/browse/OPENNLP-1974
> Project: OpenNLP
> Issue Type: Bug
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1973.
> The pattern of {{EmojiCharSequenceNormalizer}} was parsed as the range
> U\+D83C to U\+10FC00 plus a literal hyphen, so it also blanked the hyphen and
> the BMP characters from U\+D83C to U\+FFFF \(fullwidth and halfwidth forms,
> CJK compatibility ideographs, Arabic presentation forms, the private use
> area, U\+FE0F, unpaired surrogates\) and kept U\+10FC01 to U\+10FFFF.
> The normalizer now replaces each run of supplementary code points \(U\+10000
> and above\) with one space, using a code point scan instead of the regular
> expression. It is part of the default language detector chain, so models
> trained with an earlier release see different input; under
> {{\-Dopennlp.compat.mode=LEGACY}} the earlier output is produced.
> Before merge: language detector eval build.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
