rzo1 commented on code in PR #1513:
URL: https://github.com/apache/opennlp/pull/1513#discussion_r4440809091
##########
NOTICE:
##########
@@ -130,14 +130,16 @@ in the LICENSE file accompanying this distribution.
The project-authored annotation table
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
-is licensed under the Apache License 2.0, but its name values (the CLDR short
-names) and its entity-type/category derivations are transcribed from the
-upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51,
-version 17.0), distributed under the Unicode License V3 reproduced in the
-LICENSE file. Its sentiment scores are original project judgments; no
-third-party sentiment data set is included.
-
-Copyright (c) 1991-2025 Unicode, Inc. All rights reserved.
+is licensed under the Apache License 2.0. It is an independent project comment
+table (not a full Unicode Emoji inventory): coverage is limited to pictographs
+folded by emoji-emoticons.txt plus the Unicode Emoji 18.0 additions needed for
+parity with the bundled sequences. Its CLDR short names and
entity-type/category
+derivations were transcribed from upstream emoji-test.txt (Emoji
Keyboard/Display
Review Comment:
U+1F6D9 -> LANDMARK isn't from emoji-test.txt (upstream is transport-water).
"derived from, with documented exceptions" fits better.
##########
NOTICE:
##########
@@ -130,14 +130,16 @@ in the LICENSE file accompanying this distribution.
The project-authored annotation table
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
-is licensed under the Apache License 2.0, but its name values (the CLDR short
-names) and its entity-type/category derivations are transcribed from the
-upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51,
-version 17.0), distributed under the Unicode License V3 reproduced in the
-LICENSE file. Its sentiment scores are original project judgments; no
-third-party sentiment data set is included.
-
-Copyright (c) 1991-2025 Unicode, Inc. All rights reserved.
+is licensed under the Apache License 2.0. It is an independent project comment
Review Comment:
The sentence saying the CLDR names and subgroups are under the Unicode
License V3 got dropped, so the Unicode copyright line below has no license
statement anymore. Same in src/license/NOTICE.template and
opennlp-distr/src/main/readme/NOTICE. Also "comment table" -> "annotation
table"?
##########
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/uax29/ExtendedPictographicTest.java:
##########
@@ -45,6 +45,23 @@ void testNonPictographicCodePoints(int codePoint) {
assertFalse(ExtendedPictographic.is(codePoint));
}
+ /**
+ * Unicode Emoji 18.0 dropped these code points from Extended_Pictographic
(they were
+ * reserved ranges in 17.0). Word-break must treat them as non-pictographic.
+ */
+ @ParameterizedTest
+ @ValueSource(ints = {
+ 0x1F1AE,
Review Comment:
18 code points were dropped, this checks 7. Can you generate the full list
(1F1AE, 1F7DA-1F7DB, 1F7F1-1F7FF) in a MethodSource? A WordBreak case for X ZWJ
X would also help.
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java:
##########
@@ -49,9 +49,18 @@ public enum EmojiEntityType {
/** A vehicle, from the {@code transport-*} subgroups. */
VEHICLE,
- /** A landmark or building, from the {@code place-*} subgroups. */
+ /** A landmark or building, from the {@code place-*} subgroups and the
lighthouse. */
LANDMARK,
+ /** A hand gesture, from the {@code hand-*} subgroups. */
+ HAND,
Review Comment:
New constants in a public enum, needs a line in the release notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
