This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch feat/GH-1347-unicode-emoji-18 in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit f24386d88309b54cae5c6c2e0725797af8c8f69f Author: Kristian Rickert <[email protected]> AuthorDate: Thu Oct 29 07:35:08 2026 -0400 GH-1347: Update the bundled Unicode Emoji data to 18.0 Regenerate EmojiSequences.txt from the Emoji 18.0 emoji-test.txt and emoji-data.txt: 4992 qualified sequences (3963 fully-qualified and 1029 minimally-qualified), 19 added since 17.0, and the same 10 Emoji_Component ranges. The generator records the 18.0 checksums and count and accepts the "18.0.0" version header of the 18.0 emoji-data.txt. The Javadoc, README, LICENSE and NOTICE files name 18.0 and the 2026 Unicode copyright year. Add emoji-annotations.txt records for the 19 new emoji: cracking face, the leftwards and rightwards thumb signs with each of their skin tone sequences, monarch butterfly, pickle, lighthouse, meteor, eraser and net with handle, with short names and group headers copied from the 18.0 emoji-test.txt. EmojiEntityType adds constants for the thumb sign, sky and weather, tool and writing subgroups the table did not cover. The tests were written first. On the 17.0 data the normalizer test failed with 21 failures (expected 4992 but was 4973, and each 18.0 sequence was kept), and the annotation audit failed with 20 failures (expected 59 records but was 40, and no record for any of the 19 sequences). The other bundled Unicode tables remain at 17.0. --- LICENSE | 2 +- NOTICE | 6 +- dev/README-unicode-emoji-sequences.md | 27 ++++--- dev/UnicodeEmojiSequenceGenerator.java | 18 +++-- .../src/main/appended-resources/META-INF/LICENSE | 2 +- .../src/main/appended-resources/META-INF/NOTICE | 14 ++-- .../normalizer/EmojiCharSequenceNormalizer.java | 2 +- .../tools/util/normalizer/EmojiEntityType.java | 11 ++- .../util/normalizer/UnicodeEmojiSequences.java | 2 +- .../tools/util/normalizer/EmojiSequences.txt | 27 ++++++- .../tools/util/normalizer/emoji-annotations.txt | 88 +++++++++++++++++++++- .../util/normalizer/EmojiAnnotationsTest.java | 63 +++++++++++++++- .../EmojiCharSequenceNormalizerTest.java | 37 ++++----- .../normalizer/UnicodeEmojiSequencesParseTest.java | 2 +- opennlp-distr/src/main/readme/LICENSE | 2 +- opennlp-distr/src/main/readme/NOTICE | 6 +- src/license/NOTICE.template | 6 +- 17 files changed, 244 insertions(+), 71 deletions(-) diff --git a/LICENSE b/LICENSE index 8c18490f6..cfbb7d8db 100644 --- a/LICENSE +++ b/LICENSE @@ -387,7 +387,7 @@ opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/em COPYRIGHT AND PERMISSION NOTICE - Copyright (c) 1991-2025 Unicode, Inc. + Copyright (c) 1991-2026 Unicode, Inc. NOTICE TO USER: Carefully read the following legal agreement. BY DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR diff --git a/NOTICE b/NOTICE index 9540f26b5..c89b80e45 100644 --- a/NOTICE +++ b/NOTICE @@ -120,7 +120,7 @@ by Unicode, Inc. (https://www.unicode.org/Public/). * opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt is a generated inventory of fully-qualified and minimally-qualified emoji sequences from emoji-test.txt and Emoji_Component ranges from - emoji-data.txt, both Unicode Emoji version 17.0. + emoji-data.txt, both Unicode Emoji version 18.0. The original Unicode copyright and license header is preserved verbatim at the top of each copied or filtered data file. Generated inventories identify their @@ -133,11 +133,11 @@ opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/em is licensed under the Apache License 2.0, but its name values (the CLDR short names) and its entity-type/category derivations are transcribed from the upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51, -version 17.0), distributed under the Unicode License V3 reproduced in the +version 18.0), distributed under the Unicode License V3 reproduced in the LICENSE file. Its sentiment scores are original project judgments; no third-party sentiment data set is included. -Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. +Copyright (c) 1991-2026 Unicode, Inc. All rights reserved. ============================================================================ diff --git a/dev/README-unicode-emoji-sequences.md b/dev/README-unicode-emoji-sequences.md index 3f09c6706..c4a4bdaaa 100644 --- a/dev/README-unicode-emoji-sequences.md +++ b/dev/README-unicode-emoji-sequences.md @@ -19,18 +19,21 @@ limitations under the License. `UnicodeEmojiSequenceGenerator.java` derives the bundled `opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt` -from these official Unicode Emoji 17.0 files: - -* `https://www.unicode.org/Public/17.0.0/emoji/emoji-test.txt` - (`1d8a944f88d7952f7ef7c5167fef3c67995bcae24543949710231b03a201acda`) -* `https://www.unicode.org/Public/17.0.0/ucd/emoji/emoji-data.txt` - (`2cb2bb9455cda83e8481541ecf5b6dfda66a3bb89efa3fa7c5297eccf607b72b`) - -The generator verifies both SHA-256 checksums and both `# Version:` headers against the release -pinned in its `RELEASE` table, so a version bump changes that table, the data file and the NOTICE -files. It keeps the exact `fully-qualified` and `minimally-qualified` sequences from -`emoji-test.txt` (the `S;` records) and the `Emoji_Component` ranges from `emoji-data.txt` (the `C;` records). The component ranges only -tell the normalizer which stray joiners, modifiers, selectors and tags belong to a neighboring +from these official Unicode Emoji 18.0 files: + +* `https://www.unicode.org/Public/18.0.0/emoji/emoji-test.txt` + (`8f3735cda1f92a779d78af67cf86066bb1f07143dc22f2ac29394d9bc57ab21a`) +* `https://www.unicode.org/Public/18.0.0/ucd/emoji/emoji-data.txt` + (`80d00f8e616a0ef27fd6b8de3b758c06383b5d917e2977709578e68baf733bf1`) + +The generator verifies both SHA-256 checksums and both `# Version:` headers (`18.0`, or `18.0.0` +as emoji-data.txt writes it) against the release pinned in its `RELEASE` table, so a version bump +changes that table, the data file and the NOTICE files. The other bundled Unicode tables +(WordBreakProperty.txt, CaseFolding.txt, confusables.txt, ExtendedPictographic.txt) are separate +copies of Unicode 17.0 files and are not touched by this generator. It keeps the exact +`fully-qualified` and `minimally-qualified` sequences from `emoji-test.txt` (the `S;` records) +and the `Emoji_Component` ranges from `emoji-data.txt` (the `C;` records). The component ranges +only tell the normalizer which stray joiners, modifiers, selectors and tags belong to a neighboring emoji; a component on its own is never treated as an emoji. Run the generator from the repository root: diff --git a/dev/UnicodeEmojiSequenceGenerator.java b/dev/UnicodeEmojiSequenceGenerator.java index 54f39c4fc..1684f94bc 100644 --- a/dev/UnicodeEmojiSequenceGenerator.java +++ b/dev/UnicodeEmojiSequenceGenerator.java @@ -35,11 +35,11 @@ import java.util.List; */ public final class UnicodeEmojiSequenceGenerator { - /** Unicode Emoji 17.0, the version of the other bundled Unicode data files. */ - private static final Release RELEASE = new Release("17.0", "2025", - "1d8a944f88d7952f7ef7c5167fef3c67995bcae24543949710231b03a201acda", - "2cb2bb9455cda83e8481541ecf5b6dfda66a3bb89efa3fa7c5297eccf607b72b", - 4973); + /** Unicode Emoji 18.0. The other bundled Unicode data files are still Unicode 17.0. */ + private static final Release RELEASE = new Release("18.0", "2026", + "8f3735cda1f92a779d78af67cf86066bb1f07143dc22f2ac29394d9bc57ab21a", + "80d00f8e616a0ef27fd6b8de3b758c06383b5d917e2977709578e68baf733bf1", + 4992); private static final String OUTPUT_NAME = "EmojiSequences.txt"; private static final String VERSION_LINE = "# Version: "; @@ -250,8 +250,9 @@ public final class UnicodeEmojiSequenceGenerator { /** * Reads a source file once and checks its SHA-256 and its {@code # Version:} header line - * against {@link #RELEASE}. Both emoji files use the two part form, for example - * {@code # Version: 17.0}. + * against {@link #RELEASE}. The header is the two part form, for example + * {@code # Version: 18.0}, or the three part form with a zero update number, for example + * {@code # Version: 18.0.0}, which the 18.0 emoji-data.txt uses. * * @param source The file to read. * @param expectedSha256 The expected SHA-256 as lower case hex. @@ -272,7 +273,8 @@ public final class UnicodeEmojiSequenceGenerator { + ", found " + actual); } List<String> lines = new String(bytes, StandardCharsets.UTF_8).lines().toList(); - if (!lines.contains(VERSION_LINE + RELEASE.version())) { + String twoPart = VERSION_LINE + RELEASE.version(); + if (!lines.contains(twoPart) && !lines.contains(twoPart + ".0")) { throw new IOException(source + " is not Unicode Emoji version " + RELEASE.version()); } return lines; diff --git a/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/LICENSE b/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/LICENSE index 0d943f6c3..adcd0c84f 100644 --- a/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/LICENSE +++ b/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/LICENSE @@ -13,7 +13,7 @@ UNICODE LICENSE V3 COPYRIGHT AND PERMISSION NOTICE -Copyright (c) 1991-2025 Unicode, Inc. +Copyright (c) 1991-2026 Unicode, Inc. NOTICE TO USER: Carefully read the following legal agreement. BY DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR diff --git a/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/NOTICE b/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/NOTICE index 52a7fdf85..e01bda94b 100644 --- a/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/NOTICE +++ b/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/NOTICE @@ -1,6 +1,6 @@ This product bundles Unicode Character Database (UCD), Unicode Security -Mechanisms and Unicode Emoji data, version 17.0, published by Unicode, Inc. -(https://www.unicode.org/Public/): +Mechanisms and Unicode Emoji data in the versions listed below, published by +Unicode, Inc. (https://www.unicode.org/Public/): * opennlp/tools/tokenize/uax29/WordBreakProperty.txt is the upstream WordBreakProperty-17.0.0.txt, unmodified except for the file name. @@ -14,15 +14,15 @@ Mechanisms and Unicode Emoji data, version 17.0, published by Unicode, Inc. * opennlp/tools/util/normalizer/EmojiSequences.txt is a generated inventory of the fully-qualified and minimally-qualified emoji sequences from emoji-test.txt and the Emoji_Component ranges from emoji-data.txt, both - Unicode Emoji 17.0: - https://www.unicode.org/Public/17.0.0/emoji/emoji-test.txt - https://www.unicode.org/Public/17.0.0/ucd/emoji/emoji-data.txt + Unicode Emoji 18.0: + https://www.unicode.org/Public/18.0.0/emoji/emoji-test.txt + https://www.unicode.org/Public/18.0.0/ucd/emoji/emoji-data.txt The project-authored annotation table opennlp/tools/util/normalizer/emoji-annotations.txt is licensed under the Apache License 2.0, but its name values (the CLDR short names) and its entity-type/category derivations are transcribed from the upstream -emoji-test.txt (version 17.0). +emoji-test.txt (version 18.0). -Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. +Copyright (c) 1991-2026 Unicode, Inc. All rights reserved. Distributed under the Unicode License V3 reproduced in LICENSE. diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizer.java index d0c0bb890..47ed90d75 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizer.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizer.java @@ -23,7 +23,7 @@ import opennlp.tools.util.ParamChecks; * A {@link CharSequenceNormalizer} implementation that replaces each run of complete emoji with * one space. * - * <p>An emoji is a fully-qualified or minimally-qualified sequence of Unicode Emoji 17.0 in the + * <p>An emoji is a fully-qualified or minimally-qualified sequence of Unicode Emoji 18.0 in the * sense of <a href="https://www.unicode.org/reports/tr51/";>UTS #51</a>: a single emoji, an emoji * with U+FE0F, a flag, a keycap, a skin tone modifier sequence, a ZWJ sequence or a tag * sequence, as listed in {@code emoji-test.txt}. A minimally-qualified ZWJ sequence lacks U+FE0F diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java index 5c43c7840..5e271ad8d 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java @@ -49,9 +49,18 @@ public enum EmojiEntityType { /** A vehicle, from the {@code transport-*} subgroups. */ VEHICLE, - /** A landmark or building, from the {@code place-*} subgroups. */ + /** A landmark or building, from the {@code place-*} subgroups and the lighthouse. */ LANDMARK, + /** A hand gesture, from the {@code hand-*} subgroups. */ + HAND, + + /** A sky or weather phenomenon, from the {@code sky & weather} subgroup. */ + SKY_AND_WEATHER, + + /** A hand tool or writing implement, from the {@code tool} and {@code writing} subgroups. */ + TOOL, + /** A flag; assigned by the derived layer from the code point sequence itself. */ FLAG } diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UnicodeEmojiSequences.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UnicodeEmojiSequences.java index 0106bca28..14f31fb5f 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UnicodeEmojiSequences.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UnicodeEmojiSequences.java @@ -26,7 +26,7 @@ import java.util.BitSet; import java.util.List; /** - * Finds complete emoji at a position of a text, from the bundled inventory of Unicode Emoji 17.0 + * Finds complete emoji at a position of a text, from the bundled inventory of Unicode Emoji 18.0 * sequences. * * <p>A sequence is <em>fully-qualified</em> in the sense of diff --git a/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt index f91667437..c00ff4780 100644 --- a/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt +++ b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt @@ -1,12 +1,12 @@ # EmojiSequences.txt -# Copyright (c) 1991-2025 Unicode, Inc. +# Copyright (c) 1991-2026 Unicode, Inc. # For terms of use and license, see https://www.unicode.org/license.txt # # Unicode Emoji fully-qualified and minimally-qualified sequences, derived from # emoji-test.txt and emoji-data.txt under Unicode License V3. -# Version: 17.0 -# emoji-test.txt SHA-256: 1d8a944f88d7952f7ef7c5167fef3c67995bcae24543949710231b03a201acda -# emoji-data.txt SHA-256: 2cb2bb9455cda83e8481541ecf5b6dfda66a3bb89efa3fa7c5297eccf607b72b +# Version: 18.0 +# emoji-test.txt SHA-256: 8f3735cda1f92a779d78af67cf86066bb1f07143dc22f2ac29394d9bc57ab21a +# emoji-data.txt SHA-256: 80d00f8e616a0ef27fd6b8de3b758c06383b5d917e2977709578e68baf733bf1 # S records are exact sequences; C records are Emoji_Component ranges. # Generated by dev/UnicodeEmojiSequenceGenerator.java; do not edit. S;1F600 @@ -20,6 +20,7 @@ S;1F602 S;1F642 S;1F643 S;1FAE0 +S;1FAEB S;1F609 S;1F60A S;1F607 @@ -358,6 +359,18 @@ S;1F44E 1F3FC S;1F44E 1F3FD S;1F44E 1F3FE S;1F44E 1F3FF +S;1FAF9 +S;1FAF9 1F3FB +S;1FAF9 1F3FC +S;1FAF9 1F3FD +S;1FAF9 1F3FE +S;1FAF9 1F3FF +S;1FAFA +S;1FAFA 1F3FB +S;1FAFA 1F3FC +S;1FAFA 1F3FD +S;1FAFA 1F3FE +S;1FAFA 1F3FF S;270A S;270A 1F3FB S;270A 1F3FC @@ -3742,6 +3755,7 @@ S;1F991 S;1F9AA S;1F40C S;1F98B +S;1FACC S;1F41B S;1F41C S;1F41D @@ -3813,6 +3827,7 @@ S;1F33D S;1F336 FE0F S;1FAD1 S;1F952 +S;1FADD S;1F96C S;1F966 S;1F9C4 @@ -4033,6 +4048,7 @@ S;1F6A5 S;1F6A6 S;1F6D1 S;1F6A7 +S;1F6D9 S;2693 S;1F6DF S;26F5 @@ -4132,6 +4148,7 @@ S;2744 FE0F S;2603 FE0F S;26C4 S;2604 FE0F +S;1FA8B S;1F525 S;1F4A7 S;1F30A @@ -4382,6 +4399,7 @@ S;1F58A FE0F S;1F58C FE0F S;1F58D FE0F S;1F4DD +S;1FA8C S;1F4BC S;1F4C1 S;1F4C2 @@ -4411,6 +4429,7 @@ S;1F50F S;1F510 S;1F511 S;1F5DD FE0F +S;1FA8D S;1F528 S;1FA93 S;26CF FE0F diff --git a/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt index ea416c164..5ed66be10 100644 --- a/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt +++ b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt @@ -23,13 +23,14 @@ # # Attributes and their provenance: # name The CLDR short name as printed in the upstream emoji-test.txt (Emoji -# Keyboard/Display Test Data for UTS #51, version 17.0); source CLDR:annotation. +# Keyboard/Display Test Data for UTS #51, version 18.0); source CLDR:annotation. # sentiment A project-authored coarse polarity score, an integer in -2..2; source UNSPECIFIED, # the marker for a project judgment. No third-party sentiment data set is copied # here; in particular the Emoji Sentiment Ranking (CC BY-SA) is NOT used in any form. # entityType A coarse entity type (an EmojiEntityType constant), assigned by this project from # the subgroup header of the upstream emoji-test.txt; source UCD:emoji-test, the -# subgroup is cited in the notes. +# subgroup is cited in the notes. An entity type that departs from the subgroup is +# a project judgment, source UNSPECIFIED, with the subgroup still cited in the notes. # category A document-category hint (an EmojiCategory constant), the group header of the # upstream emoji-test.txt; source UCD:emoji-test, the group is cited in the notes. # @@ -38,7 +39,8 @@ # every record carries all four attributes. Flag emoji have no rows: their region is derived from # the code point sequence (see EmojiFlags), not bundled. # -# Code points, names, and group/subgroup headers checked against emoji-test.txt version 17.0. +# Code points, names, and group/subgroup headers checked against emoji-test.txt version 18.0; +# the notes column records which emoji-test.txt version a name was transcribed from. # # A line starting with '#' is a comment; data rows carry no inline comments. The notes column is # the fifth and final field, so it may contain ';'. @@ -172,6 +174,60 @@ 1F494 ; sentiment ; -2 ; UNSPECIFIED ; project judgment 1F494 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart 1F494 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1FAEB ; name ; cracking face ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAEB ; sentiment ; -1 ; UNSPECIFIED ; project judgment +1FAEB ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling +1FAEB ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +# +# People & Body +1FAF9 ; name ; leftwards thumb sign ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAF9 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAF9 ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAF9 ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAF9 1F3FB ; name ; leftwards thumb sign: light skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAF9 1F3FB ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAF9 1F3FB ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAF9 1F3FB ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAF9 1F3FC ; name ; leftwards thumb sign: medium-light skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAF9 1F3FC ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAF9 1F3FC ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAF9 1F3FC ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAF9 1F3FD ; name ; leftwards thumb sign: medium skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAF9 1F3FD ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAF9 1F3FD ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAF9 1F3FD ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAF9 1F3FE ; name ; leftwards thumb sign: medium-dark skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAF9 1F3FE ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAF9 1F3FE ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAF9 1F3FE ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAF9 1F3FF ; name ; leftwards thumb sign: dark skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAF9 1F3FF ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAF9 1F3FF ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAF9 1F3FF ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAFA ; name ; rightwards thumb sign ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAFA ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAFA ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAFA ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAFA 1F3FB ; name ; rightwards thumb sign: light skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAFA 1F3FB ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAFA 1F3FB ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAFA 1F3FB ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAFA 1F3FC ; name ; rightwards thumb sign: medium-light skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAFA 1F3FC ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAFA 1F3FC ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAFA 1F3FC ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAFA 1F3FD ; name ; rightwards thumb sign: medium skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAFA 1F3FD ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAFA 1F3FD ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAFA 1F3FD ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAFA 1F3FE ; name ; rightwards thumb sign: medium-dark skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAFA 1F3FE ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAFA 1F3FE ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAFA 1F3FE ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body +1FAFA 1F3FF ; name ; rightwards thumb sign: dark skin tone ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FAFA 1F3FF ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FAFA 1F3FF ; entityType ; HAND ; UCD:emoji-test ; subgroup hand-fingers-closed +1FAFA 1F3FF ; category ; PEOPLE_AND_BODY ; UCD:emoji-test ; group People & Body # # Animals & Nature 1F436 ; name ; dog face ; CLDR:annotation ; emoji-test.txt v17.0 short name @@ -182,6 +238,10 @@ 1F431 ; sentiment ; 0 ; UNSPECIFIED ; project judgment 1F431 ; entityType ; ANIMAL ; UCD:emoji-test ; subgroup animal-mammal 1F431 ; category ; ANIMALS_AND_NATURE ; UCD:emoji-test ; group Animals & Nature +1FACC ; name ; monarch butterfly ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FACC ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FACC ; entityType ; ANIMAL ; UCD:emoji-test ; subgroup animal-bug +1FACC ; category ; ANIMALS_AND_NATURE ; UCD:emoji-test ; group Animals & Nature # # Food & Drink 1F355 ; name ; pizza ; CLDR:annotation ; emoji-test.txt v17.0 short name @@ -192,6 +252,10 @@ 1F34E ; sentiment ; 0 ; UNSPECIFIED ; project judgment 1F34E ; entityType ; FOOD ; UCD:emoji-test ; subgroup food-fruit 1F34E ; category ; FOOD_AND_DRINK ; UCD:emoji-test ; group Food & Drink +1FADD ; name ; pickle ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FADD ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FADD ; entityType ; FOOD ; UCD:emoji-test ; subgroup food-vegetable +1FADD ; category ; FOOD_AND_DRINK ; UCD:emoji-test ; group Food & Drink # # Travel & Places 1F697 ; name ; automobile ; CLDR:annotation ; emoji-test.txt v17.0 short name @@ -210,3 +274,21 @@ 1F5FC ; sentiment ; 0 ; UNSPECIFIED ; project judgment 1F5FC ; entityType ; LANDMARK ; UCD:emoji-test ; subgroup place-building 1F5FC ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places +1F6D9 ; name ; lighthouse ; CLDR:annotation ; emoji-test.txt v18.0 short name +1F6D9 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F6D9 ; entityType ; LANDMARK ; UNSPECIFIED ; project judgment, UCD subgroup transport-water +1F6D9 ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places +1FA8B ; name ; meteor ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FA8B ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FA8B ; entityType ; SKY_AND_WEATHER ; UCD:emoji-test ; subgroup sky & weather +1FA8B ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places +# +# Objects +1FA8C ; name ; eraser ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FA8C ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FA8C ; entityType ; TOOL ; UCD:emoji-test ; subgroup writing +1FA8C ; category ; OBJECTS ; UCD:emoji-test ; group Objects +1FA8D ; name ; net with handle ; CLDR:annotation ; emoji-test.txt v18.0 short name +1FA8D ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1FA8D ; entityType ; TOOL ; UCD:emoji-test ; subgroup tool +1FA8D ; category ; OBJECTS ; UCD:emoji-test ; group Objects diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java index 51beda7e6..d1fbd1ccf 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java @@ -27,6 +27,7 @@ import java.util.stream.Stream; import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; import org.junit.jupiter.params.provider.MethodSource; import static opennlp.tools.util.normalizer.NormalizerTestUtil.cp; @@ -69,7 +70,9 @@ public class EmojiAnnotationsTest { // The licensing audit: names come from the CLDR short names, entity type and category from // the emoji-test.txt group/subgroup headers, and sentiment is ALWAYS a project judgment // (UNSPECIFIED). A sentiment row with any other source would mean third-party sentiment data - // (for example the CC BY-SA Emoji Sentiment Ranking) leaked into the bundled file. + // (for example the CC BY-SA Emoji Sentiment Ranking) leaked into the bundled file. An entity + // type that departs from the subgroup is a project judgment too, and its notes must still + // cite the UCD subgroup it departs from. final Map<String, String> expectedSource = Map.of( EmojiAnnotation.NAME, "CLDR:annotation", EmojiAnnotation.SENTIMENT, "UNSPECIFIED", @@ -79,6 +82,12 @@ public class EmojiAnnotationsTest { for (final Map.Entry<String, EmojiAnnotation.Value> entry : annotation.attributes().entrySet()) { final EmojiAnnotation.Value value = entry.getValue(); + if (EmojiAnnotation.ENTITY_TYPE.equals(entry.getKey()) + && "UNSPECIFIED".equals(value.source())) { + assertTrue(value.notes().contains("UCD subgroup "), + "Project-judged entity type without its UCD subgroup: " + annotation.symbol()); + continue; + } assertEquals(expectedSource.get(entry.getKey()), value.source(), "Unexpected source for " + annotation.symbol() + " " + entry.getKey()); assertFalse(value.notes().isEmpty(), @@ -111,12 +120,60 @@ public class EmojiAnnotationsTest { // Locks the bundled counts so a data edit trips this test for a conscious bump, the same // discipline as the emoji-emoticons.txt and CaseFolding.txt audits. final Map<String, EmojiAnnotation> annotations = bundled(); - assertEquals(40, annotations.size()); + assertEquals(59, annotations.size()); int rowCount = 0; for (final EmojiAnnotation annotation : annotations.values()) { rowCount += annotation.attributes().size(); } - assertEquals(160, rowCount); + assertEquals(236, rowCount); + } + + /** + * The nine pictographs and ten skin tone sequences that Emoji 18.0 added, with the short names + * printed in the 18.0 emoji-test.txt, each bundled as its own record. + */ + private static Stream<Arguments> emoji18Additions() { + return Stream.of( + Arguments.of(cp(0x1FAEB), "cracking face", EmojiCategory.SMILEYS_AND_EMOTION), + Arguments.of(cp(0x1FAF9), "leftwards thumb sign", EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAF9, 0x1F3FB), "leftwards thumb sign: light skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAF9, 0x1F3FC), "leftwards thumb sign: medium-light skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAF9, 0x1F3FD), "leftwards thumb sign: medium skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAF9, 0x1F3FE), "leftwards thumb sign: medium-dark skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAF9, 0x1F3FF), "leftwards thumb sign: dark skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAFA), "rightwards thumb sign", EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAFA, 0x1F3FB), "rightwards thumb sign: light skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAFA, 0x1F3FC), "rightwards thumb sign: medium-light skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAFA, 0x1F3FD), "rightwards thumb sign: medium skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAFA, 0x1F3FE), "rightwards thumb sign: medium-dark skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FAFA, 0x1F3FF), "rightwards thumb sign: dark skin tone", + EmojiCategory.PEOPLE_AND_BODY), + Arguments.of(cp(0x1FACC), "monarch butterfly", EmojiCategory.ANIMALS_AND_NATURE), + Arguments.of(cp(0x1FADD), "pickle", EmojiCategory.FOOD_AND_DRINK), + Arguments.of(cp(0x1F6D9), "lighthouse", EmojiCategory.TRAVEL_AND_PLACES), + Arguments.of(cp(0x1FA8B), "meteor", EmojiCategory.TRAVEL_AND_PLACES), + Arguments.of(cp(0x1FA8C), "eraser", EmojiCategory.OBJECTS), + Arguments.of(cp(0x1FA8D), "net with handle", EmojiCategory.OBJECTS)); + } + + @ParameterizedTest(name = "{1}") + @MethodSource("emoji18Additions") + void everyEmoji18AdditionIsAnnotated(String symbol, String name, EmojiCategory category) { + final EmojiAnnotation annotation = EmojiAnnotations.lookup(symbol).orElseThrow( + () -> new AssertionError("No annotation record for Emoji 18.0 addition: " + name)); + assertEquals(name, annotation.name().orElseThrow()); + assertEquals(category, annotation.category().orElseThrow()); + annotation.sentiment().orElseThrow(); + annotation.entityType().orElseThrow(); } @Test diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizerTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizerTest.java index 5017e267c..7c5d09c56 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizerTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizerTest.java @@ -42,10 +42,10 @@ public class EmojiCharSequenceNormalizerTest { EmojiCharSequenceNormalizer.getInstance(); /** - * The number of fully-qualified and minimally-qualified sequences in the Emoji 17.0 + * The number of fully-qualified and minimally-qualified sequences in the Emoji 18.0 * emoji-test.txt, the same number dev/UnicodeEmojiSequenceGenerator.java pins for that release. */ - private static final int EMOJI_17_SEQUENCE_COUNT = 4973; + private static final int EMOJI_18_SEQUENCE_COUNT = 4992; private static Stream<Arguments> emojiSequences() { return Stream.of( @@ -68,11 +68,11 @@ public class EmojiCharSequenceNormalizerTest { } /** - * The nine code points and ten modifier sequences that Emoji 18.0 added on top of 17.0. The - * bundled inventory is Emoji 17.0, the same version as the other bundled Unicode data, so - * these are not emoji to this normalizer yet. + * The nine code points and ten modifier sequences that Emoji 18.0 added on top of 17.0, the + * difference between the qualified lines of the two emoji-test.txt files. The bundled + * inventory is Emoji 18.0, so each of them is one complete emoji to this normalizer. */ - private static Stream<String> emoji18OnlySequences() { + private static Stream<String> emoji18AddedSequences() { return Stream.of(cp(0x1FAEB), cp(0x1FACC), cp(0x1FADD), cp(0x1F6D9), cp(0x1FA8B), cp(0x1FA8C), cp(0x1FA8D), cp(0x1FAF9), cp(0x1FAFA), cp(0x1FAF9, 0x1F3FB), cp(0x1FAF9, 0x1F3FC), cp(0x1FAF9, 0x1F3FD), @@ -82,9 +82,9 @@ public class EmojiCharSequenceNormalizerTest { } @ParameterizedTest - @MethodSource("emoji18OnlySequences") - void normalizeKeepsSequencesAddedAfterEmoji17(String sequence) { - Assertions.assertEquals("a" + sequence + "b", NORMALIZER.normalize("a" + sequence + "b")); + @MethodSource("emoji18AddedSequences") + void normalizeRemovesSequencesAddedInEmoji18(String sequence) { + Assertions.assertEquals("a b", NORMALIZER.normalize("a" + sequence + "b")); } @Test @@ -244,11 +244,12 @@ public class EmojiCharSequenceNormalizerTest { } /** - * Known Emoji 17.0 sequences, one or two per emoji-test.txt group, written down from the - * published list rather than read from the bundled file, so a wrong or truncated inventory - * fails here. The last seven are new in 17.0. + * Known emoji sequences up to Emoji 17.0, one or two per emoji-test.txt group, written down + * from the published list rather than read from the bundled file, so a wrong or truncated + * inventory fails here. The last seven are new in 17.0; {@link #emoji18AddedSequences()} + * covers the 18.0 additions. */ - private static Stream<Arguments> knownEmoji17Sequences() { + private static Stream<Arguments> knownEmojiSequences() { return Stream.of( Arguments.of(cp(0x1F600), "Smileys & Emotion: grinning face"), Arguments.of(cp(0x1F636, 0x200D, 0x1F32B, 0xFE0F), "Smileys & Emotion: face in clouds"), @@ -280,14 +281,14 @@ public class EmojiCharSequenceNormalizerTest { } @ParameterizedTest(name = "{1}") - @MethodSource("knownEmoji17Sequences") - void normalizeRemovesKnownEmoji17Sequences(String emoji, String description) { + @MethodSource("knownEmojiSequences") + void normalizeRemovesKnownEmojiSequences(String emoji, String description) { Assertions.assertEquals("a b", NORMALIZER.normalize("a" + emoji + "b")); } @Test - void bundledInventoryHoldsTheEmoji17SequenceCount() { - Assertions.assertEquals(EMOJI_17_SEQUENCE_COUNT, + void bundledInventoryHoldsTheEmoji18SequenceCount() { + Assertions.assertEquals(EMOJI_18_SEQUENCE_COUNT, UnicodeEmojiSequences.getInstance().sequenceCount()); } @@ -309,7 +310,7 @@ public class EmojiCharSequenceNormalizerTest { count++; } } - Assertions.assertEquals(EMOJI_17_SEQUENCE_COUNT, count); + Assertions.assertEquals(EMOJI_18_SEQUENCE_COUNT, count); } @Test diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/UnicodeEmojiSequencesParseTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/UnicodeEmojiSequencesParseTest.java index 274430de4..5c37300eb 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/UnicodeEmojiSequencesParseTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/UnicodeEmojiSequencesParseTest.java @@ -33,7 +33,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue; class UnicodeEmojiSequencesParseTest { private static final String WELL_FORMED = "# EmojiSequences.txt\n" - + "# Version: 17.0\n" + + "# Version: 18.0\n" + "S;1F600\n" + "S;1F44D 1F3FD\n" + "\n" diff --git a/opennlp-distr/src/main/readme/LICENSE b/opennlp-distr/src/main/readme/LICENSE index e1783ae8f..edb471f7b 100644 --- a/opennlp-distr/src/main/readme/LICENSE +++ b/opennlp-distr/src/main/readme/LICENSE @@ -625,7 +625,7 @@ opennlp/tools/util/normalizer/emoji-annotations.txt: COPYRIGHT AND PERMISSION NOTICE - Copyright (c) 1991-2025 Unicode, Inc. + Copyright (c) 1991-2026 Unicode, Inc. NOTICE TO USER: Carefully read the following legal agreement. BY DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR diff --git a/opennlp-distr/src/main/readme/NOTICE b/opennlp-distr/src/main/readme/NOTICE index 9540f26b5..c89b80e45 100644 --- a/opennlp-distr/src/main/readme/NOTICE +++ b/opennlp-distr/src/main/readme/NOTICE @@ -120,7 +120,7 @@ by Unicode, Inc. (https://www.unicode.org/Public/). * opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt is a generated inventory of fully-qualified and minimally-qualified emoji sequences from emoji-test.txt and Emoji_Component ranges from - emoji-data.txt, both Unicode Emoji version 17.0. + emoji-data.txt, both Unicode Emoji version 18.0. The original Unicode copyright and license header is preserved verbatim at the top of each copied or filtered data file. Generated inventories identify their @@ -133,11 +133,11 @@ opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/em is licensed under the Apache License 2.0, but its name values (the CLDR short names) and its entity-type/category derivations are transcribed from the upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51, -version 17.0), distributed under the Unicode License V3 reproduced in the +version 18.0), distributed under the Unicode License V3 reproduced in the LICENSE file. Its sentiment scores are original project judgments; no third-party sentiment data set is included. -Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. +Copyright (c) 1991-2026 Unicode, Inc. All rights reserved. ============================================================================ diff --git a/src/license/NOTICE.template b/src/license/NOTICE.template index b4809bb49..b32c80043 100644 --- a/src/license/NOTICE.template +++ b/src/license/NOTICE.template @@ -120,7 +120,7 @@ by Unicode, Inc. (https://www.unicode.org/Public/). * opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt is a generated inventory of fully-qualified and minimally-qualified emoji sequences from emoji-test.txt and Emoji_Component ranges from - emoji-data.txt, both Unicode Emoji version 17.0. + emoji-data.txt, both Unicode Emoji version 18.0. The original Unicode copyright and license header is preserved verbatim at the top of each copied or filtered data file. Generated inventories identify their @@ -133,10 +133,10 @@ opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/em is licensed under the Apache License 2.0, but its name values (the CLDR short names) and its entity-type/category derivations are transcribed from the upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51, -version 17.0), distributed under the Unicode License V3 reproduced in the +version 18.0), distributed under the Unicode License V3 reproduced in the LICENSE file. Its sentiment scores are original project judgments; no third-party sentiment data set is included. -Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. +Copyright (c) 1991-2026 Unicode, Inc. All rights reserved. ============================================================================
