This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch feat/GH-1347-unicode-emoji-18 in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit bcec97f6f0a6e109f2fdaca2a3b07f9c11f17ed9 Author: Kristian Rickert <[email protected]> AuthorDate: Thu Oct 1 18:33:34 2026 -0400 GH-1347: Update the bundled Unicode Emoji data to 18.0 Regenerate EmojiSequences.txt from the Emoji 18.0 emoji-test.txt and emoji-data.txt (4992 qualified sequences: 3963 fully-qualified and 1029 minimally-qualified, 19 added since 17.0, the same 10 Emoji_Component ranges), pin the 18.0 checksums and count in the generator, accept the "18.0.0" version header of the 18.0 emoji-data.txt, and name 18.0 and the 2026 Unicode copyright year in the Javadoc, README, LICENSE and NOTICE files. The other bundled Unicode tables remain at 17.0. --- LICENSE | 2 +- NOTICE | 4 ++-- dev/README-unicode-emoji-sequences.md | 27 ++++++++++++---------- dev/UnicodeEmojiSequenceGenerator.java | 18 ++++++++------- .../src/main/appended-resources/META-INF/LICENSE | 2 +- .../src/main/appended-resources/META-INF/NOTICE | 12 +++++----- .../normalizer/EmojiCharSequenceNormalizer.java | 2 +- .../util/normalizer/UnicodeEmojiSequences.java | 2 +- .../tools/util/normalizer/EmojiSequences.txt | 27 ++++++++++++++++++---- opennlp-distr/src/main/readme/LICENSE | 2 +- opennlp-distr/src/main/readme/NOTICE | 4 ++-- src/license/NOTICE.template | 4 ++-- 12 files changed, 65 insertions(+), 41 deletions(-) diff --git a/LICENSE b/LICENSE index 8c18490f6..cfbb7d8db 100644 --- a/LICENSE +++ b/LICENSE @@ -387,7 +387,7 @@ opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/em COPYRIGHT AND PERMISSION NOTICE - Copyright (c) 1991-2025 Unicode, Inc. + Copyright (c) 1991-2026 Unicode, Inc. NOTICE TO USER: Carefully read the following legal agreement. BY DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR diff --git a/NOTICE b/NOTICE index 9540f26b5..48b13bb6c 100644 --- a/NOTICE +++ b/NOTICE @@ -120,7 +120,7 @@ by Unicode, Inc. (https://www.unicode.org/Public/). * opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt is a generated inventory of fully-qualified and minimally-qualified emoji sequences from emoji-test.txt and Emoji_Component ranges from - emoji-data.txt, both Unicode Emoji version 17.0. + emoji-data.txt, both Unicode Emoji version 18.0. The original Unicode copyright and license header is preserved verbatim at the top of each copied or filtered data file. Generated inventories identify their @@ -137,7 +137,7 @@ version 17.0), distributed under the Unicode License V3 reproduced in the LICENSE file. Its sentiment scores are original project judgments; no third-party sentiment data set is included. -Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. +Copyright (c) 1991-2026 Unicode, Inc. All rights reserved. ============================================================================ diff --git a/dev/README-unicode-emoji-sequences.md b/dev/README-unicode-emoji-sequences.md index 3f09c6706..c4a4bdaaa 100644 --- a/dev/README-unicode-emoji-sequences.md +++ b/dev/README-unicode-emoji-sequences.md @@ -19,18 +19,21 @@ limitations under the License. `UnicodeEmojiSequenceGenerator.java` derives the bundled `opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt` -from these official Unicode Emoji 17.0 files: - -* `https://www.unicode.org/Public/17.0.0/emoji/emoji-test.txt` - (`1d8a944f88d7952f7ef7c5167fef3c67995bcae24543949710231b03a201acda`) -* `https://www.unicode.org/Public/17.0.0/ucd/emoji/emoji-data.txt` - (`2cb2bb9455cda83e8481541ecf5b6dfda66a3bb89efa3fa7c5297eccf607b72b`) - -The generator verifies both SHA-256 checksums and both `# Version:` headers against the release -pinned in its `RELEASE` table, so a version bump changes that table, the data file and the NOTICE -files. It keeps the exact `fully-qualified` and `minimally-qualified` sequences from -`emoji-test.txt` (the `S;` records) and the `Emoji_Component` ranges from `emoji-data.txt` (the `C;` records). The component ranges only -tell the normalizer which stray joiners, modifiers, selectors and tags belong to a neighboring +from these official Unicode Emoji 18.0 files: + +* `https://www.unicode.org/Public/18.0.0/emoji/emoji-test.txt` + (`8f3735cda1f92a779d78af67cf86066bb1f07143dc22f2ac29394d9bc57ab21a`) +* `https://www.unicode.org/Public/18.0.0/ucd/emoji/emoji-data.txt` + (`80d00f8e616a0ef27fd6b8de3b758c06383b5d917e2977709578e68baf733bf1`) + +The generator verifies both SHA-256 checksums and both `# Version:` headers (`18.0`, or `18.0.0` +as emoji-data.txt writes it) against the release pinned in its `RELEASE` table, so a version bump +changes that table, the data file and the NOTICE files. The other bundled Unicode tables +(WordBreakProperty.txt, CaseFolding.txt, confusables.txt, ExtendedPictographic.txt) are separate +copies of Unicode 17.0 files and are not touched by this generator. It keeps the exact +`fully-qualified` and `minimally-qualified` sequences from `emoji-test.txt` (the `S;` records) +and the `Emoji_Component` ranges from `emoji-data.txt` (the `C;` records). The component ranges +only tell the normalizer which stray joiners, modifiers, selectors and tags belong to a neighboring emoji; a component on its own is never treated as an emoji. Run the generator from the repository root: diff --git a/dev/UnicodeEmojiSequenceGenerator.java b/dev/UnicodeEmojiSequenceGenerator.java index 54f39c4fc..1684f94bc 100644 --- a/dev/UnicodeEmojiSequenceGenerator.java +++ b/dev/UnicodeEmojiSequenceGenerator.java @@ -35,11 +35,11 @@ import java.util.List; */ public final class UnicodeEmojiSequenceGenerator { - /** Unicode Emoji 17.0, the version of the other bundled Unicode data files. */ - private static final Release RELEASE = new Release("17.0", "2025", - "1d8a944f88d7952f7ef7c5167fef3c67995bcae24543949710231b03a201acda", - "2cb2bb9455cda83e8481541ecf5b6dfda66a3bb89efa3fa7c5297eccf607b72b", - 4973); + /** Unicode Emoji 18.0. The other bundled Unicode data files are still Unicode 17.0. */ + private static final Release RELEASE = new Release("18.0", "2026", + "8f3735cda1f92a779d78af67cf86066bb1f07143dc22f2ac29394d9bc57ab21a", + "80d00f8e616a0ef27fd6b8de3b758c06383b5d917e2977709578e68baf733bf1", + 4992); private static final String OUTPUT_NAME = "EmojiSequences.txt"; private static final String VERSION_LINE = "# Version: "; @@ -250,8 +250,9 @@ public final class UnicodeEmojiSequenceGenerator { /** * Reads a source file once and checks its SHA-256 and its {@code # Version:} header line - * against {@link #RELEASE}. Both emoji files use the two part form, for example - * {@code # Version: 17.0}. + * against {@link #RELEASE}. The header is the two part form, for example + * {@code # Version: 18.0}, or the three part form with a zero update number, for example + * {@code # Version: 18.0.0}, which the 18.0 emoji-data.txt uses. * * @param source The file to read. * @param expectedSha256 The expected SHA-256 as lower case hex. @@ -272,7 +273,8 @@ public final class UnicodeEmojiSequenceGenerator { + ", found " + actual); } List<String> lines = new String(bytes, StandardCharsets.UTF_8).lines().toList(); - if (!lines.contains(VERSION_LINE + RELEASE.version())) { + String twoPart = VERSION_LINE + RELEASE.version(); + if (!lines.contains(twoPart) && !lines.contains(twoPart + ".0")) { throw new IOException(source + " is not Unicode Emoji version " + RELEASE.version()); } return lines; diff --git a/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/LICENSE b/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/LICENSE index 0d943f6c3..adcd0c84f 100644 --- a/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/LICENSE +++ b/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/LICENSE @@ -13,7 +13,7 @@ UNICODE LICENSE V3 COPYRIGHT AND PERMISSION NOTICE -Copyright (c) 1991-2025 Unicode, Inc. +Copyright (c) 1991-2026 Unicode, Inc. NOTICE TO USER: Carefully read the following legal agreement. BY DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR diff --git a/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/NOTICE b/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/NOTICE index 52a7fdf85..1d6e00212 100644 --- a/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/NOTICE +++ b/opennlp-core/opennlp-runtime/src/main/appended-resources/META-INF/NOTICE @@ -1,6 +1,6 @@ This product bundles Unicode Character Database (UCD), Unicode Security -Mechanisms and Unicode Emoji data, version 17.0, published by Unicode, Inc. -(https://www.unicode.org/Public/): +Mechanisms and Unicode Emoji data in the versions listed below, published by +Unicode, Inc. (https://www.unicode.org/Public/): * opennlp/tools/tokenize/uax29/WordBreakProperty.txt is the upstream WordBreakProperty-17.0.0.txt, unmodified except for the file name. @@ -14,9 +14,9 @@ Mechanisms and Unicode Emoji data, version 17.0, published by Unicode, Inc. * opennlp/tools/util/normalizer/EmojiSequences.txt is a generated inventory of the fully-qualified and minimally-qualified emoji sequences from emoji-test.txt and the Emoji_Component ranges from emoji-data.txt, both - Unicode Emoji 17.0: - https://www.unicode.org/Public/17.0.0/emoji/emoji-test.txt - https://www.unicode.org/Public/17.0.0/ucd/emoji/emoji-data.txt + Unicode Emoji 18.0: + https://www.unicode.org/Public/18.0.0/emoji/emoji-test.txt + https://www.unicode.org/Public/18.0.0/ucd/emoji/emoji-data.txt The project-authored annotation table opennlp/tools/util/normalizer/emoji-annotations.txt is licensed under the @@ -24,5 +24,5 @@ Apache License 2.0, but its name values (the CLDR short names) and its entity-type/category derivations are transcribed from the upstream emoji-test.txt (version 17.0). -Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. +Copyright (c) 1991-2026 Unicode, Inc. All rights reserved. Distributed under the Unicode License V3 reproduced in LICENSE. diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizer.java index d0c0bb890..47ed90d75 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizer.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCharSequenceNormalizer.java @@ -23,7 +23,7 @@ import opennlp.tools.util.ParamChecks; * A {@link CharSequenceNormalizer} implementation that replaces each run of complete emoji with * one space. * - * <p>An emoji is a fully-qualified or minimally-qualified sequence of Unicode Emoji 17.0 in the + * <p>An emoji is a fully-qualified or minimally-qualified sequence of Unicode Emoji 18.0 in the * sense of <a href="https://www.unicode.org/reports/tr51/";>UTS #51</a>: a single emoji, an emoji * with U+FE0F, a flag, a keycap, a skin tone modifier sequence, a ZWJ sequence or a tag * sequence, as listed in {@code emoji-test.txt}. A minimally-qualified ZWJ sequence lacks U+FE0F diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UnicodeEmojiSequences.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UnicodeEmojiSequences.java index 0106bca28..14f31fb5f 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UnicodeEmojiSequences.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UnicodeEmojiSequences.java @@ -26,7 +26,7 @@ import java.util.BitSet; import java.util.List; /** - * Finds complete emoji at a position of a text, from the bundled inventory of Unicode Emoji 17.0 + * Finds complete emoji at a position of a text, from the bundled inventory of Unicode Emoji 18.0 * sequences. * * <p>A sequence is <em>fully-qualified</em> in the sense of diff --git a/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt index f91667437..c00ff4780 100644 --- a/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt +++ b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt @@ -1,12 +1,12 @@ # EmojiSequences.txt -# Copyright (c) 1991-2025 Unicode, Inc. +# Copyright (c) 1991-2026 Unicode, Inc. # For terms of use and license, see https://www.unicode.org/license.txt # # Unicode Emoji fully-qualified and minimally-qualified sequences, derived from # emoji-test.txt and emoji-data.txt under Unicode License V3. -# Version: 17.0 -# emoji-test.txt SHA-256: 1d8a944f88d7952f7ef7c5167fef3c67995bcae24543949710231b03a201acda -# emoji-data.txt SHA-256: 2cb2bb9455cda83e8481541ecf5b6dfda66a3bb89efa3fa7c5297eccf607b72b +# Version: 18.0 +# emoji-test.txt SHA-256: 8f3735cda1f92a779d78af67cf86066bb1f07143dc22f2ac29394d9bc57ab21a +# emoji-data.txt SHA-256: 80d00f8e616a0ef27fd6b8de3b758c06383b5d917e2977709578e68baf733bf1 # S records are exact sequences; C records are Emoji_Component ranges. # Generated by dev/UnicodeEmojiSequenceGenerator.java; do not edit. S;1F600 @@ -20,6 +20,7 @@ S;1F602 S;1F642 S;1F643 S;1FAE0 +S;1FAEB S;1F609 S;1F60A S;1F607 @@ -358,6 +359,18 @@ S;1F44E 1F3FC S;1F44E 1F3FD S;1F44E 1F3FE S;1F44E 1F3FF +S;1FAF9 +S;1FAF9 1F3FB +S;1FAF9 1F3FC +S;1FAF9 1F3FD +S;1FAF9 1F3FE +S;1FAF9 1F3FF +S;1FAFA +S;1FAFA 1F3FB +S;1FAFA 1F3FC +S;1FAFA 1F3FD +S;1FAFA 1F3FE +S;1FAFA 1F3FF S;270A S;270A 1F3FB S;270A 1F3FC @@ -3742,6 +3755,7 @@ S;1F991 S;1F9AA S;1F40C S;1F98B +S;1FACC S;1F41B S;1F41C S;1F41D @@ -3813,6 +3827,7 @@ S;1F33D S;1F336 FE0F S;1FAD1 S;1F952 +S;1FADD S;1F96C S;1F966 S;1F9C4 @@ -4033,6 +4048,7 @@ S;1F6A5 S;1F6A6 S;1F6D1 S;1F6A7 +S;1F6D9 S;2693 S;1F6DF S;26F5 @@ -4132,6 +4148,7 @@ S;2744 FE0F S;2603 FE0F S;26C4 S;2604 FE0F +S;1FA8B S;1F525 S;1F4A7 S;1F30A @@ -4382,6 +4399,7 @@ S;1F58A FE0F S;1F58C FE0F S;1F58D FE0F S;1F4DD +S;1FA8C S;1F4BC S;1F4C1 S;1F4C2 @@ -4411,6 +4429,7 @@ S;1F50F S;1F510 S;1F511 S;1F5DD FE0F +S;1FA8D S;1F528 S;1FA93 S;26CF FE0F diff --git a/opennlp-distr/src/main/readme/LICENSE b/opennlp-distr/src/main/readme/LICENSE index e1783ae8f..edb471f7b 100644 --- a/opennlp-distr/src/main/readme/LICENSE +++ b/opennlp-distr/src/main/readme/LICENSE @@ -625,7 +625,7 @@ opennlp/tools/util/normalizer/emoji-annotations.txt: COPYRIGHT AND PERMISSION NOTICE - Copyright (c) 1991-2025 Unicode, Inc. + Copyright (c) 1991-2026 Unicode, Inc. NOTICE TO USER: Carefully read the following legal agreement. BY DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR diff --git a/opennlp-distr/src/main/readme/NOTICE b/opennlp-distr/src/main/readme/NOTICE index 9540f26b5..48b13bb6c 100644 --- a/opennlp-distr/src/main/readme/NOTICE +++ b/opennlp-distr/src/main/readme/NOTICE @@ -120,7 +120,7 @@ by Unicode, Inc. (https://www.unicode.org/Public/). * opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt is a generated inventory of fully-qualified and minimally-qualified emoji sequences from emoji-test.txt and Emoji_Component ranges from - emoji-data.txt, both Unicode Emoji version 17.0. + emoji-data.txt, both Unicode Emoji version 18.0. The original Unicode copyright and license header is preserved verbatim at the top of each copied or filtered data file. Generated inventories identify their @@ -137,7 +137,7 @@ version 17.0), distributed under the Unicode License V3 reproduced in the LICENSE file. Its sentiment scores are original project judgments; no third-party sentiment data set is included. -Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. +Copyright (c) 1991-2026 Unicode, Inc. All rights reserved. ============================================================================ diff --git a/src/license/NOTICE.template b/src/license/NOTICE.template index b4809bb49..a6b7ab035 100644 --- a/src/license/NOTICE.template +++ b/src/license/NOTICE.template @@ -120,7 +120,7 @@ by Unicode, Inc. (https://www.unicode.org/Public/). * opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/EmojiSequences.txt is a generated inventory of fully-qualified and minimally-qualified emoji sequences from emoji-test.txt and Emoji_Component ranges from - emoji-data.txt, both Unicode Emoji version 17.0. + emoji-data.txt, both Unicode Emoji version 18.0. The original Unicode copyright and license header is preserved verbatim at the top of each copied or filtered data file. Generated inventories identify their @@ -137,6 +137,6 @@ version 17.0), distributed under the Unicode License V3 reproduced in the LICENSE file. Its sentiment scores are original project judgments; no third-party sentiment data set is included. -Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. +Copyright (c) 1991-2026 Unicode, Inc. All rights reserved. ============================================================================
