This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch OPENNLP-1870 in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit b7fbcd9bb9000287e2cecefe7a6b9a40be37056d Author: Kristian Rickert <[email protected]> AuthorDate: Mon Jul 6 22:00:41 2026 -0400 OPENNLP-1870: Emoji annotation record store with bundled facts and audit Adds the bundled-facts layer of the emoji annotation record store: a provenance-tagged, project-authored attribute table (emoji-annotations.txt) loaded by EmojiAnnotations into per-symbol EmojiAnnotation records. - Attribute-row format (codepoints ; attribute ; value ; source ; notes), so adding an attribute later is new rows plus loader support instead of a file format break, and every value carries its own provenance. - Attributes: name (CLDR short name, CLDR:annotation), sentiment (project judgment on a coarse -2..2 scale, UNSPECIFIED), entityType and category (assigned from the emoji-test.txt subgroup/group headers, UCD:emoji-test). No third-party sentiment data set is copied; in particular the CC BY-SA Emoji Sentiment Ranking is not used in any form. - The loader fails loud on an unknown attribute or malformed row (IllegalArgumentException naming the resource and line): the data and the code move together, so an unrecognized attribute is corruption, not extensibility. Values are validated into their typed domains at load. - Lookups strip U+FE0F, since presentation selection does not change a symbol's identity. - Audit tests pin coverage (every EMOJI-direction source of emoji-emoticons.txt has a complete four-attribute record), per-attribute provenance tags, typed domains, and the record/row counts (40/160). - LICENSE/NOTICE/NOTICE.template attribution for the Unicode-derived values (CLDR short names, emoji-test.txt group/subgroup classifications) in the otherwise project-authored table. Flag emoji intentionally have no bundled rows; their region decodes from the code point sequence itself in the derived-facts layer (next commit), and gazetteer identifiers are never baked into bundled data. Part of the OPENNLP-1852 epic. Stacks on OPENNLP-1869. --- LICENSE | 8 +- NOTICE | 9 + .../tools/util/normalizer/EmojiAnnotation.java | 185 ++++++++++++++ .../tools/util/normalizer/EmojiAnnotations.java | 254 +++++++++++++++++++ .../tools/util/normalizer/EmojiCategory.java | 62 +++++ .../tools/util/normalizer/EmojiEntityType.java | 57 +++++ .../tools/util/normalizer/emoji-annotations.txt | 223 +++++++++++++++++ .../util/normalizer/EmojiAnnotationsTest.java | 269 +++++++++++++++++++++ src/license/NOTICE.template | 9 + 9 files changed, 1074 insertions(+), 2 deletions(-) diff --git a/LICENSE b/LICENSE index 1b92a4dc6..e77ddad5f 100644 --- a/LICENSE +++ b/LICENSE @@ -375,9 +375,13 @@ The following license applies to the bundled Unicode data files in opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/tokenize/uax29 (WordBreakProperty.txt, ExtendedPictographic.txt), opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer -(confusables.txt, CaseFolding.txt), and +(confusables.txt, CaseFolding.txt), opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/tokenize/uax29 -(WordBreakTest.txt): +(WordBreakTest.txt), and to the Unicode-derived values (the CLDR short names +and the emoji-test.txt group/subgroup classifications) in the otherwise +project-authored +opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt +(see NOTICE): UNICODE LICENSE V3 diff --git a/NOTICE b/NOTICE index bca534b1f..c503a5f8f 100644 --- a/NOTICE +++ b/NOTICE @@ -123,6 +123,15 @@ top of each bundled file. These files are distributed under the Unicode License V3, the full text of which is reproduced in the LICENSE file accompanying this distribution. +The project-authored annotation table +opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt +is licensed under the Apache License 2.0, but its name values (the CLDR short +names) and its entity-type/category derivations are transcribed from the +upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51, +version 17.0), distributed under the Unicode License V3 reproduced in the +LICENSE file. Its sentiment scores are original project judgments; no +third-party sentiment data set is included. + Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. ============================================================================ diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java new file mode 100644 index 000000000..b19ce8a4e --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.util.normalizer; + +import java.util.Map; +import java.util.Optional; +import java.util.OptionalInt; + +/** + * The annotation record of one emoji: an extensible per-symbol store of attribute values, each + * carrying its own provenance. A record is assembled from up to three strictly separated layers: + * bundled facts from {@code emoji-annotations.txt} (see {@link EmojiAnnotations}), derived facts + * computed from the code point sequence itself, and joined facts resolved at run time against + * user-installed data. The typed accessors ({@link #name()}, {@link #sentiment()}, + * {@link #entityType()}, {@link #category()}) are projections of the {@link #attributes()} map, + * so an attribute added later is new rows plus an accessor, not a shape change. + * + * <p>Every attribute is optional: a consumer that only wants one dimension pays nothing for the + * rest, and a record never fabricates a value it has no source for. Instances are immutable and + * thread-safe.</p> + * + * <p>These annotations are per-symbol <em>metadata</em>, not text transforms, which is why they + * are not {@link Dimension} constants; see {@link EmojiAnnotations} for the design note.</p> + * + * @param symbol The annotated code point sequence, without the U+FE0F presentation selector. + * @param attributes The attribute values keyed by attribute name ({@link #NAME}, + * {@link #SENTIMENT}, {@link #ENTITY_TYPE}, {@link #CATEGORY}). + */ +public record EmojiAnnotation(String symbol, Map<String, Value> attributes) { + + /** The attribute key of the human-readable name (the CLDR short name). */ + public static final String NAME = "name"; + + /** The attribute key of the project-authored coarse sentiment score, an integer in -2..2. */ + public static final String SENTIMENT = "sentiment"; + + /** The attribute key of the coarse {@link EmojiEntityType entity type}. */ + public static final String ENTITY_TYPE = "entityType"; + + /** The attribute key of the {@link EmojiCategory document-category hint}. */ + public static final String CATEGORY = "category"; + + /** + * One attribute value with its provenance, mirroring one row of the bundled data file. + * + * @param value The attribute value. Must not be {@code null} or empty. + * @param source The provenance tag, for example {@code CLDR:annotation}, {@code UCD:emoji-test}, + * or {@code UNSPECIFIED} (the explicit marker for a project judgment). Must not be + * {@code null} or empty. + * @param notes Free-text notes, for example the upstream subgroup a value was derived from. + * Must not be {@code null}; may be empty. + */ + public record Value(String value, String source, String notes) { + + public Value { + if (value == null || value.isEmpty()) { + throw new IllegalArgumentException("Value must not be null or empty"); + } + if (source == null || source.isEmpty()) { + throw new IllegalArgumentException("Source must not be null or empty"); + } + if (notes == null) { + throw new IllegalArgumentException("Notes must not be null"); + } + } + } + + /** + * Creates a record; normally records come from {@link EmojiAnnotations#lookup(CharSequence)}. + * + * @param symbol The annotated code point sequence. Must not be {@code null} or empty. + * @param attributes The attribute values keyed by attribute name. Must not be {@code null} or + * contain {@code null} keys or values; it is defensively copied. + * @throws IllegalArgumentException if {@code symbol} is {@code null} or empty, or if + * {@code attributes} is {@code null} or contains a {@code null} key or value. + */ + public EmojiAnnotation { + if (symbol == null || symbol.isEmpty()) { + throw new IllegalArgumentException("Symbol must not be null or empty"); + } + if (attributes == null) { + throw new IllegalArgumentException("Attributes must not be null"); + } + for (final Map.Entry<String, Value> entry : attributes.entrySet()) { + if (entry.getKey() == null || entry.getValue() == null) { + throw new IllegalArgumentException( + "Attributes must not contain a null key or value, got: " + attributes); + } + } + attributes = Map.copyOf(attributes); + } + + /** + * Returns the value of one attribute with its provenance. + * + * @param name The attribute name, for example {@link #NAME}. Must not be {@code null}. + * @return The value, or empty when this record does not carry the attribute. + * @throws IllegalArgumentException if {@code name} is {@code null}. + */ + public Optional<Value> attribute(String name) { + if (name == null) { + throw new IllegalArgumentException("Name must not be null"); + } + return Optional.ofNullable(attributes.get(name)); + } + + /** + * {@return the human-readable name (the CLDR short name), or empty when not annotated} + */ + public Optional<String> name() { + final Value value = attributes.get(NAME); + return value == null ? Optional.empty() : Optional.of(value.value()); + } + + /** + * {@return the project-authored coarse sentiment score in -2..2, or empty when not annotated} + * + * @throws IllegalStateException if the stored value is not an integer, which cannot happen for + * records loaded from the bundled data (the loader validates it). + */ + public OptionalInt sentiment() { + final Value value = attributes.get(SENTIMENT); + if (value == null) { + return OptionalInt.empty(); + } + try { + return OptionalInt.of(Integer.parseInt(value.value())); + } catch (NumberFormatException e) { + throw new IllegalStateException("Sentiment value '" + value.value() + "' of symbol '" + + symbol + "' is not an integer", e); + } + } + + /** + * {@return the coarse entity type, or empty when not annotated} + * + * @throws IllegalStateException if the stored value is not an {@link EmojiEntityType} constant, + * which cannot happen for records loaded from the bundled data (the loader validates it). + */ + public Optional<EmojiEntityType> entityType() { + final Value value = attributes.get(ENTITY_TYPE); + if (value == null) { + return Optional.empty(); + } + try { + return Optional.of(EmojiEntityType.valueOf(value.value())); + } catch (IllegalArgumentException e) { + throw new IllegalStateException("Entity type value '" + value.value() + "' of symbol '" + + symbol + "' is not an EmojiEntityType constant", e); + } + } + + /** + * {@return the document-category hint, or empty when not annotated} + * + * @throws IllegalStateException if the stored value is not an {@link EmojiCategory} constant, + * which cannot happen for records loaded from the bundled data (the loader validates it). + */ + public Optional<EmojiCategory> category() { + final Value value = attributes.get(CATEGORY); + if (value == null) { + return Optional.empty(); + } + try { + return Optional.of(EmojiCategory.valueOf(value.value())); + } catch (IllegalArgumentException e) { + throw new IllegalStateException("Category value '" + value.value() + "' of symbol '" + + symbol + "' is not an EmojiCategory constant", e); + } + } +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java new file mode 100644 index 000000000..a191f803f --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java @@ -0,0 +1,254 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.util.normalizer; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +/** + * The bundled-facts layer of the emoji annotation record store: license-clean, provenance-tagged + * attributes intrinsic to a pictograph (name, coarse sentiment, entity type, document category), + * loaded once from the project-authored {@code emoji-annotations.txt} resource. Each row of that + * file is one attribute of one symbol ({@code codepoints ; attribute ; value ; source ; notes}), + * so every value carries its own provenance and adding an attribute later is new rows plus loader + * support instead of a file-format break. The loader fails loud on an unknown attribute or a + * malformed row: the data and the code move together, so an unrecognized attribute is corruption, + * not extensibility. + * + * <p>Data licensing: the name values are the CLDR short names and the entity-type/category values + * are derived from the group and subgroup headers of the upstream {@code emoji-test.txt} + * (UTS #51, Unicode License V3, see the NOTICE file); the sentiment scores are original + * project judgments tagged {@code UNSPECIFIED}. No third-party sentiment data set is copied; in + * particular the Emoji Sentiment Ranking (CC BY-SA) is not used in any form.</p> + * + * <p>Design note: these annotations are surfaced beside the {@link Term} model rather than as new + * {@link Dimension} constants. A {@code Dimension} is a character-level text <em>transform</em> + * whose result is another string layer of the token (each layer feeds the next, and + * {@link Term#peel()}/{@link Term#normalized()} walk that stack), while an annotation is typed, + * per-symbol <em>metadata</em> about the original pictograph: it has no place in the transform + * pipeline, would not compose with the layers below it, and would force typed facts through a + * stringly {@code Map<Dimension, String>}. The parallel accessor surface is + * {@code EmojiAnnotator}, which consumes a {@code Term} and returns the record.</p> + * + * <p>Lookups strip U+FE0F VARIATION SELECTOR-16 (the emoji presentation selector) from the queried + * sequence, because presentation selection does not change a symbol's identity; the bundled rows + * are keyed without it. Flag emoji intentionally have no bundled rows: their region decodes from + * the code point sequence itself (the derived-facts layer), and gazetteer identifiers are never + * baked into bundled data.</p> + */ +public final class EmojiAnnotations { + + private static final String RESOURCE = "emoji-annotations.txt"; + + // The records keyed by code point sequence, loaded lazily on first use and cached. Volatile so + // the fully built, immutable map is safely published to every thread that observes it non-null. + private static volatile Map<String, EmojiAnnotation> annotations; + + private EmojiAnnotations() { + } + + /** + * Returns the bundled annotation record of one emoji. + * + * @param symbol The code point sequence of one symbol, for example one {@link Term#original()} + * token. U+FE0F presentation selectors are ignored. Must not be {@code null}. + * @return The record, or empty when the bundled data does not annotate the symbol. + * @throws IllegalArgumentException if {@code symbol} is {@code null}. + * @throws IllegalStateException if the bundled data resource is missing. + * @throws UncheckedIOException if the bundled data resource cannot be read. + */ + public static Optional<EmojiAnnotation> lookup(CharSequence symbol) { + if (symbol == null) { + throw new IllegalArgumentException("Symbol must not be null"); + } + final String key = stripPresentationSelector(symbol); + if (key.isEmpty()) { + return Optional.empty(); + } + return Optional.ofNullable(annotations().get(key)); + } + + // Removes every U+FE0F VARIATION SELECTOR-16; allocation-free when none is present. + private static String stripPresentationSelector(CharSequence symbol) { + final int length = symbol.length(); + int i = 0; + while (i < length && symbol.charAt(i) != 0xFE0F) { + i++; + } + if (i == length) { + return symbol.toString(); + } + final StringBuilder stripped = new StringBuilder(length - 1); + stripped.append(symbol, 0, i); + for (int k = i + 1; k < length; k++) { + final char c = symbol.charAt(k); + if (c != 0xFE0F) { + stripped.append(c); + } + } + return stripped.toString(); + } + + private static Map<String, EmojiAnnotation> annotations() { + Map<String, EmojiAnnotation> map = annotations; + if (map == null) { + synchronized (EmojiAnnotations.class) { + map = annotations; + if (map == null) { + map = load(); + annotations = map; + } + } + } + return map; + } + + private static Map<String, EmojiAnnotation> load() { + try (InputStream in = EmojiAnnotations.class.getResourceAsStream(RESOURCE)) { + if (in == null) { + throw new IllegalStateException("Missing emoji annotation data resource: " + RESOURCE); + } + return parse(in); + } catch (IOException e) { + throw new UncheckedIOException("Unable to read emoji annotation data resource " + RESOURCE, e); + } + } + + // Package-private so the malformed-data handling can be exercised without the bundled resource. + // Parses rows of "codepoints ; attribute ; value ; source ; notes" with space-separated + // hexadecimal code points; '#' starts a comment line. The notes column is the fifth and final + // field, so it may contain ';'. + static Map<String, EmojiAnnotation> parse(InputStream in) throws IOException { + final Map<String, Map<String, EmojiAnnotation.Value>> rows = new HashMap<>(); + try (BufferedReader reader = + new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8))) { + String line; + int lineNumber = 0; + while ((line = reader.readLine()) != null) { + lineNumber++; + final String content = line.strip(); + if (content.isEmpty() || content.startsWith("#")) { + continue; + } + // Bounded split: only the first four separators are structural. + final String[] fields = content.split(";", 5); + if (fields.length != 5) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": expected 5 fields, got " + fields.length + + " in: " + content); + } + final String symbol = decode(fields[0], lineNumber, content); + final String attribute = fields[1].strip(); + final String value = fields[2].strip(); + final String source = fields[3].strip(); + final String notes = fields[4].strip(); + validate(attribute, value, source, lineNumber, content); + final Map<String, EmojiAnnotation.Value> record = + rows.computeIfAbsent(symbol, k -> new HashMap<>()); + if (record.putIfAbsent(attribute, new EmojiAnnotation.Value(value, source, notes)) != null) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": duplicate attribute '" + attribute + + "' in: " + content); + } + } + } + final Map<String, EmojiAnnotation> records = new HashMap<>(rows.size()); + for (final Map.Entry<String, Map<String, EmojiAnnotation.Value>> entry : rows.entrySet()) { + records.put(entry.getKey(), new EmojiAnnotation(entry.getKey(), entry.getValue())); + } + return Map.copyOf(records); + } + + // Fails loud on an unknown attribute and on a value the attribute's type does not admit, so a + // corrupted or drifted data file cannot load quietly. + private static void validate(String attribute, String value, String source, + int lineNumber, String content) { + if (value.isEmpty()) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": empty value in: " + content); + } + if (source.isEmpty()) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": empty source in: " + content); + } + switch (attribute) { + case EmojiAnnotation.NAME: + break; + case EmojiAnnotation.SENTIMENT: + final int score; + try { + score = Integer.parseInt(value); + } catch (NumberFormatException e) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": sentiment value '" + value + + "' is not an integer in: " + content, e); + } + if (score < -2 || score > 2) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": sentiment value " + score + + " is outside -2..2 in: " + content); + } + break; + case EmojiAnnotation.ENTITY_TYPE: + try { + EmojiEntityType.valueOf(value); + } catch (IllegalArgumentException e) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": unrecognized entityType value '" + value + + "' in: " + content, e); + } + break; + case EmojiAnnotation.CATEGORY: + try { + EmojiCategory.valueOf(value); + } catch (IllegalArgumentException e) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": unrecognized category value '" + value + + "' in: " + content, e); + } + break; + default: + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": unknown attribute '" + attribute + "' in: " + content); + } + } + + private static String decode(String hexCodePoints, int lineNumber, String content) { + final String stripped = hexCodePoints.strip(); + if (stripped.isEmpty()) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": empty code point sequence in: " + content); + } + try { + final StringBuilder decoded = new StringBuilder(); + for (final String hex : stripped.split(" ")) { + decoded.appendCodePoint(Integer.parseInt(hex, 16)); + } + return decoded.toString(); + } catch (NumberFormatException e) { + throw new IllegalArgumentException("Malformed emoji annotation data in " + RESOURCE + + " at line " + lineNumber + ": " + content, e); + } + } +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java new file mode 100644 index 000000000..cfedf4018 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.util.normalizer; + +/** + * The document-category hint of an annotated emoji, one constant per group header of the upstream + * Unicode {@code emoji-test.txt} (Emoji Keyboard/Display Test Data for + * <a href="https://www.unicode.org/reports/tr51/";>UTS #51</a>). The group level of that file + * is already a document-category-shaped taxonomy, so this enum mirrors it one to one rather than + * inventing a competing one; the constant a bundled record carries is provenance-tagged + * {@code UCD:emoji-test} in {@code emoji-annotations.txt}. + * + * <p>All upstream groups are represented, including those the bundled data does not populate yet, + * so growing the data never needs an enum change. See {@link EmojiEntityType} for the finer, + * subgroup-derived projection of the same upstream table.</p> + */ +public enum EmojiCategory { + + /** Upstream group {@code Smileys & Emotion}: faces, hearts, and other emotion symbols. */ + SMILEYS_AND_EMOTION, + + /** Upstream group {@code People & Body}: people, body parts, hand gestures, roles. */ + PEOPLE_AND_BODY, + + /** Upstream group {@code Component}: skin tone and hair components, not free-standing symbols. */ + COMPONENT, + + /** Upstream group {@code Animals & Nature}: animals, plants, and nature symbols. */ + ANIMALS_AND_NATURE, + + /** Upstream group {@code Food & Drink}: food, drink, and dishware. */ + FOOD_AND_DRINK, + + /** Upstream group {@code Travel & Places}: places, buildings, transport, sky and weather. */ + TRAVEL_AND_PLACES, + + /** Upstream group {@code Activities}: events, sports, games, and arts. */ + ACTIVITIES, + + /** Upstream group {@code Objects}: clothing, tools, instruments, and other objects. */ + OBJECTS, + + /** Upstream group {@code Symbols}: signs, arrows, punctuation-like and abstract symbols. */ + SYMBOLS, + + /** Upstream group {@code Flags}: flag emoji; assigned by the derived layer, not by bundled rows. */ + FLAGS +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java new file mode 100644 index 000000000..5c43c7840 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java @@ -0,0 +1,57 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.util.normalizer; + +/** + * The coarse entity type of an annotated emoji, a small project-authored taxonomy assigned from + * the subgroup headers of the upstream Unicode {@code emoji-test.txt} (Emoji Keyboard/Display Test + * Data for <a href="https://www.unicode.org/reports/tr51/";>UTS #51</a>): for example the + * {@code face-smiling} and {@code face-concerned} subgroups map to {@link #FACE} and + * {@code transport-ground} maps to {@link #VEHICLE}. The subgroup a bundled record was derived + * from is cited in the notes column of {@code emoji-annotations.txt}, provenance-tagged + * {@code UCD:emoji-test}. + * + * <p>The set is deliberately coarse and grows with the bundled data; a new constant is a + * compatible addition. {@link #FLAG} is assigned by the derived layer (regional-indicator and tag + * sequence decoding), never by a bundled row, so flag coverage does not depend on data + * rows.</p> + * + * @see EmojiCategory + */ +public enum EmojiEntityType { + + /** A face pictograph, from the {@code face-*} subgroups. */ + FACE, + + /** A heart pictograph, from the {@code heart} subgroup. */ + HEART, + + /** An animal, from the {@code animal-*} subgroups. */ + ANIMAL, + + /** Food, from the {@code food-*} subgroups. */ + FOOD, + + /** A vehicle, from the {@code transport-*} subgroups. */ + VEHICLE, + + /** A landmark or building, from the {@code place-*} subgroups. */ + LANDMARK, + + /** A flag; assigned by the derived layer from the code point sequence itself. */ + FLAG +} diff --git a/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt new file mode 100644 index 000000000..d95a83f14 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt @@ -0,0 +1,223 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Bundled emoji annotation facts, authored by this project. This is the bundled-facts layer of +# the emoji annotation record store (loaded by EmojiAnnotations): one row per attribute of one +# pictograph, so adding an attribute later is new rows plus loader support instead of a file +# format break, and every value carries its own provenance. Row format: +# +# codepoints ; attribute ; value ; source ; notes +# +# codepoints is the space-separated hexadecimal code point sequence of the pictograph, keyed +# without U+FE0F VARIATION SELECTOR-16 (the emoji presentation selector); lookups strip it, +# because presentation selection does not change a symbol's identity. +# +# Attributes and their provenance: +# name The CLDR short name as printed in the upstream emoji-test.txt (Emoji +# Keyboard/Display Test Data for UTS #51, version 17.0); source CLDR:annotation. +# sentiment A project-authored coarse polarity score, an integer in -2..2; original work, +# source UNSPECIFIED, the explicit marker for a project judgment. No third-party +# sentiment data set is copied here; in particular the Emoji Sentiment Ranking +# (CC BY-SA) is NOT used in any form. +# entityType A coarse entity type (an EmojiEntityType constant), assigned by this project +# from the subgroup header of the upstream emoji-test.txt; source UCD:emoji-test, +# the subgroup is cited in the notes. +# category A document-category hint (an EmojiCategory constant), the group header of the +# upstream emoji-test.txt; source UCD:emoji-test, the group is cited in the notes. +# +# The loader fails loud on an unknown attribute or a malformed row: the data and the code move +# together, so an unrecognized attribute is corruption, not extensibility. +# +# Coverage claim (audited by EmojiAnnotationsTest): every pictograph that emoji-emoticons.txt +# folds (every EMOJI-direction source, presentation selector stripped) has a record here, and +# every record carries all four attributes. +# +# Flag emoji deliberately have no rows: their region decodes algorithmically from the code point +# sequence (the derived-facts layer), and gazetteer identifiers are never baked into bundled data +# (they churn, the gazetteer is an optional download, and baked identifiers would couple this +# file's version to the gazetteer's version; the joined-facts layer resolves them at run time). +# +# Code points, names, and group/subgroup headers checked against emoji-test.txt version 17.0. +# +# A line starting with '#' is a comment; data rows carry no inline comments. The notes column is +# the fifth and final field, so it may contain ';'. +# +# Smileys & Emotion +1F600 ; name ; grinning face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F600 ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F600 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling +1F600 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F603 ; name ; grinning face with big eyes ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F603 ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F603 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling +1F603 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F604 ; name ; grinning face with smiling eyes ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F604 ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F604 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling +1F604 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F601 ; name ; beaming face with smiling eyes ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F601 ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F601 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling +1F601 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F642 ; name ; slightly smiling face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F642 ; sentiment ; 1 ; UNSPECIFIED ; project judgment +1F642 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling +1F642 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F609 ; name ; winking face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F609 ; sentiment ; 1 ; UNSPECIFIED ; project judgment +1F609 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling +1F609 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F60A ; name ; smiling face with smiling eyes ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F60A ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F60A ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling +1F60A ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +263A ; name ; smiling face ; CLDR:annotation ; emoji-test.txt v17.0 short name +263A ; sentiment ; 1 ; UNSPECIFIED ; project judgment +263A ; entityType ; FACE ; UCD:emoji-test ; subgroup face-affection +263A ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F617 ; name ; kissing face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F617 ; sentiment ; 1 ; UNSPECIFIED ; project judgment +1F617 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-affection +1F617 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F618 ; name ; face blowing a kiss ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F618 ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F618 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-affection +1F618 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F61B ; name ; face with tongue ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F61B ; sentiment ; 1 ; UNSPECIFIED ; project judgment +1F61B ; entityType ; FACE ; UCD:emoji-test ; subgroup face-tongue +1F61B ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F61C ; name ; winking face with tongue ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F61C ; sentiment ; 1 ; UNSPECIFIED ; project judgment +1F61C ; entityType ; FACE ; UCD:emoji-test ; subgroup face-tongue +1F61C ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F61D ; name ; squinting face with tongue ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F61D ; sentiment ; 1 ; UNSPECIFIED ; project judgment +1F61D ; entityType ; FACE ; UCD:emoji-test ; subgroup face-tongue +1F61D ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F610 ; name ; neutral face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F610 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F610 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-neutral-skeptical +1F610 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F611 ; name ; expressionless face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F611 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F611 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-neutral-skeptical +1F611 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F615 ; name ; confused face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F615 ; sentiment ; -1 ; UNSPECIFIED ; project judgment +1F615 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned +1F615 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F641 ; name ; slightly frowning face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F641 ; sentiment ; -1 ; UNSPECIFIED ; project judgment +1F641 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned +1F641 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +2639 ; name ; frowning face ; CLDR:annotation ; emoji-test.txt v17.0 short name +2639 ; sentiment ; -1 ; UNSPECIFIED ; project judgment +2639 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned +2639 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F62E ; name ; face with open mouth ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F62E ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F62E ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned +1F62E ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F632 ; name ; astonished face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F632 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F632 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned +1F632 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F622 ; name ; crying face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F622 ; sentiment ; -2 ; UNSPECIFIED ; project judgment +1F622 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned +1F622 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F62D ; name ; loudly crying face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F62D ; sentiment ; -2 ; UNSPECIFIED ; project judgment +1F62D ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned +1F62D ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +2764 ; name ; red heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +2764 ; sentiment ; 2 ; UNSPECIFIED ; project judgment +2764 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +2764 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F9E1 ; name ; orange heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F9E1 ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F9E1 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +1F9E1 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F49B ; name ; yellow heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F49B ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F49B ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +1F49B ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F49A ; name ; green heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F49A ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F49A ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +1F49A ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F499 ; name ; blue heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F499 ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F499 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +1F499 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F49C ; name ; purple heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F49C ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F49C ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +1F49C ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F90E ; name ; brown heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F90E ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F90E ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +1F90E ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F5A4 ; name ; black heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F5A4 ; sentiment ; 0 ; UNSPECIFIED ; project judgment, polarity ambiguous in running text +1F5A4 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +1F5A4 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F90D ; name ; white heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F90D ; sentiment ; 2 ; UNSPECIFIED ; project judgment +1F90D ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +1F90D ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +1F494 ; name ; broken heart ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F494 ; sentiment ; -2 ; UNSPECIFIED ; project judgment +1F494 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart +1F494 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys & Emotion +# +# Animals & Nature +1F436 ; name ; dog face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F436 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F436 ; entityType ; ANIMAL ; UCD:emoji-test ; subgroup animal-mammal +1F436 ; category ; ANIMALS_AND_NATURE ; UCD:emoji-test ; group Animals & Nature +1F431 ; name ; cat face ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F431 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F431 ; entityType ; ANIMAL ; UCD:emoji-test ; subgroup animal-mammal +1F431 ; category ; ANIMALS_AND_NATURE ; UCD:emoji-test ; group Animals & Nature +# +# Food & Drink +1F355 ; name ; pizza ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F355 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F355 ; entityType ; FOOD ; UCD:emoji-test ; subgroup food-prepared +1F355 ; category ; FOOD_AND_DRINK ; UCD:emoji-test ; group Food & Drink +1F34E ; name ; red apple ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F34E ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F34E ; entityType ; FOOD ; UCD:emoji-test ; subgroup food-fruit +1F34E ; category ; FOOD_AND_DRINK ; UCD:emoji-test ; group Food & Drink +# +# Travel & Places +1F697 ; name ; automobile ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F697 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F697 ; entityType ; VEHICLE ; UCD:emoji-test ; subgroup transport-ground +1F697 ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places +2708 ; name ; airplane ; CLDR:annotation ; emoji-test.txt v17.0 short name +2708 ; sentiment ; 0 ; UNSPECIFIED ; project judgment +2708 ; entityType ; VEHICLE ; UCD:emoji-test ; subgroup transport-air +2708 ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places +1F5FD ; name ; Statue of Liberty ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F5FD ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F5FD ; entityType ; LANDMARK ; UCD:emoji-test ; subgroup place-building +1F5FD ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places +1F5FC ; name ; Tokyo tower ; CLDR:annotation ; emoji-test.txt v17.0 short name +1F5FC ; sentiment ; 0 ; UNSPECIFIED ; project judgment +1F5FC ; entityType ; LANDMARK ; UCD:emoji-test ; subgroup place-building +1F5FC ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java new file mode 100644 index 000000000..27c76d735 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java @@ -0,0 +1,269 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.util.normalizer; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class EmojiAnnotationsTest { + + private static final String[] ATTRIBUTES = {EmojiAnnotation.NAME, EmojiAnnotation.SENTIMENT, + EmojiAnnotation.ENTITY_TYPE, EmojiAnnotation.CATEGORY}; + + private static String cp(int codePoint) { + return new String(Character.toChars(codePoint)); + } + + private static Map<String, EmojiAnnotation> bundled() throws IOException { + try (InputStream in = EmojiAnnotations.class.getResourceAsStream("emoji-annotations.txt")) { + return EmojiAnnotations.parse(in); + } + } + + private static Map<String, EmojiAnnotation> parse(String data) throws IOException { + return EmojiAnnotations.parse(new ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8))); + } + + // --- audits of the bundled data --- + + @Test + void everyBundledRecordCarriesAllFourAttributes() throws IOException { + // The coverage claim in the data file header: a record is complete, never a partial row set, + // so a consumer can rely on any of the four dimensions once a symbol is annotated at all. + for (final EmojiAnnotation annotation : bundled().values()) { + for (final String attribute : ATTRIBUTES) { + assertTrue(annotation.attribute(attribute).isPresent(), + "Missing attribute '" + attribute + "' for symbol: " + annotation.symbol()); + } + } + } + + @Test + void everyBundledValueCarriesItsPinnedProvenance() throws IOException { + // The licensing audit: names come from the CLDR short names, entity type and category from + // the emoji-test.txt group/subgroup headers, and sentiment is ALWAYS a project judgment + // (UNSPECIFIED). A sentiment row with any other source would mean third-party sentiment data + // (for example the CC BY-SA Emoji Sentiment Ranking) leaked into the bundled file. + final Map<String, String> expectedSource = Map.of( + EmojiAnnotation.NAME, "CLDR:annotation", + EmojiAnnotation.SENTIMENT, "UNSPECIFIED", + EmojiAnnotation.ENTITY_TYPE, "UCD:emoji-test", + EmojiAnnotation.CATEGORY, "UCD:emoji-test"); + for (final EmojiAnnotation annotation : bundled().values()) { + for (final Map.Entry<String, EmojiAnnotation.Value> entry : + annotation.attributes().entrySet()) { + final EmojiAnnotation.Value value = entry.getValue(); + assertEquals(expectedSource.get(entry.getKey()), value.source(), + "Unexpected source for " + annotation.symbol() + " " + entry.getKey()); + assertTrue(!value.notes().isEmpty(), + "Empty notes for " + annotation.symbol() + " " + entry.getKey()); + } + } + } + + @Test + void everyEmojiFoldSourceIsAnnotated() throws IOException { + // The cross-file coverage claim: every pictograph emoji-emoticons.txt can fold has a record + // here (presentation selector stripped), so the fold layer and the annotation layer never + // disagree about which symbols the project understands. + final Map<String, EmojiAnnotation> annotations = bundled(); + final EmojiEmoticons.Tables tables; + try (InputStream in = EmojiEmoticons.class.getResourceAsStream("emoji-emoticons.txt")) { + tables = EmojiEmoticons.parse(in); + } + for (final List<EmojiEmoticons.Mapping> candidates : tables.emojiToEmoticon().values()) { + for (final EmojiEmoticons.Mapping mapping : candidates) { + final String key = mapping.source().replace(cp(0xFE0F), ""); + assertTrue(annotations.containsKey(key), + "Emoji fold source has no annotation record: " + mapping); + } + } + } + + @Test + void bundledRecordAndRowCountsAreAudited() throws IOException { + // Locks the bundled counts so a data edit trips this test for a conscious bump, the same + // discipline as the emoji-emoticons.txt and CaseFolding.txt audits. + final Map<String, EmojiAnnotation> annotations = bundled(); + assertEquals(40, annotations.size()); + int rowCount = 0; + for (final EmojiAnnotation annotation : annotations.values()) { + rowCount += annotation.attributes().size(); + } + assertEquals(160, rowCount); + } + + @Test + void bundledValuesAreWithinTheirTypedDomains() throws IOException { + // The typed accessors must never throw for bundled data: sentiment parses into -2..2 and the + // enum-backed attributes resolve to constants. Also asserts values are printable ASCII, the + // same discipline as the data file itself. + for (final EmojiAnnotation annotation : bundled().values()) { + final int sentiment = annotation.sentiment().orElseThrow(); + assertTrue(sentiment >= -2 && sentiment <= 2, + "Sentiment outside -2..2 for " + annotation.symbol()); + annotation.entityType().orElseThrow(); + annotation.category().orElseThrow(); + final String name = annotation.name().orElseThrow(); + name.chars().forEach(c -> + assertTrue(c >= 0x20 && c < 0x7F, "Non-ASCII-printable name: " + name)); + } + } + + // --- lookup behavior --- + + @Test + void lookupResolvesATypedRecord() { + final EmojiAnnotation annotation = EmojiAnnotations.lookup(cp(0x1F642)).orElseThrow(); + assertEquals(cp(0x1F642), annotation.symbol()); + assertEquals("slightly smiling face", annotation.name().orElseThrow()); + assertEquals(1, annotation.sentiment().orElseThrow()); + assertEquals(EmojiEntityType.FACE, annotation.entityType().orElseThrow()); + assertEquals(EmojiCategory.SMILEYS_AND_EMOTION, annotation.category().orElseThrow()); + } + + @Test + void lookupStripsThePresentationSelector() { + // U+2764 U+FE0F (emoji presentation) and bare U+2764 are the same symbol; the bundled rows + // are keyed without the selector. + final EmojiAnnotation withSelector = + EmojiAnnotations.lookup(cp(0x2764) + cp(0xFE0F)).orElseThrow(); + final EmojiAnnotation bare = EmojiAnnotations.lookup(cp(0x2764)).orElseThrow(); + assertEquals(bare, withSelector); + assertEquals("red heart", withSelector.name().orElseThrow()); + } + + @Test + void lookupOfUnannotatedTextIsEmpty() { + assertEquals(Optional.empty(), EmojiAnnotations.lookup("word")); + assertEquals(Optional.empty(), EmojiAnnotations.lookup(":-)")); + assertEquals(Optional.empty(), EmojiAnnotations.lookup("")); + // A bare presentation selector strips to an empty key. + assertEquals(Optional.empty(), EmojiAnnotations.lookup(cp(0xFE0F))); + // Flags are handled by the derived layer, never by bundled rows. + assertEquals(Optional.empty(), EmojiAnnotations.lookup(cp(0x1F1E9) + cp(0x1F1EA))); + } + + @Test + void lookupFailsLoudOnNull() { + assertThrows(IllegalArgumentException.class, () -> EmojiAnnotations.lookup(null)); + } + + // --- fail-loud parsing --- + + @Test + void parseFailsLoudOnWrongFieldCount() { + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; name ; slightly smiling face ; CLDR:annotation\n")); + } + + @Test + void parseFailsLoudOnUnknownAttribute() { + // An unknown attribute is corruption, not extensibility: the data and the code move together. + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; shortName ; slightly smiling face ; CLDR:annotation ; n\n")); + } + + @Test + void parseFailsLoudOnMalformedHex() { + assertThrows(IllegalArgumentException.class, + () -> parse("1F64X ; name ; slightly smiling face ; CLDR:annotation ; n\n")); + } + + @Test + void parseFailsLoudOnEmptyFields() { + assertThrows(IllegalArgumentException.class, + () -> parse(" ; name ; slightly smiling face ; CLDR:annotation ; n\n")); + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; name ; ; CLDR:annotation ; n\n")); + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; name ; slightly smiling face ; ; n\n")); + } + + @Test + void parseFailsLoudOnSentimentOutsideTheCoarseScale() { + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; sentiment ; 3 ; UNSPECIFIED ; n\n")); + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; sentiment ; -3 ; UNSPECIFIED ; n\n")); + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; sentiment ; positive ; UNSPECIFIED ; n\n")); + } + + @Test + void parseFailsLoudOnUnrecognizedEnumValues() { + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; entityType ; SMILEY ; UCD:emoji-test ; n\n")); + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; category ; EMOTION ; UCD:emoji-test ; n\n")); + } + + @Test + void parseFailsLoudOnDuplicateAttributeRows() { + assertThrows(IllegalArgumentException.class, + () -> parse("1F642 ; sentiment ; 1 ; UNSPECIFIED ; first\n" + + "1F642 ; sentiment ; 2 ; UNSPECIFIED ; duplicate\n")); + } + + @Test + void parseSkipsCommentAndBlankLinesAndKeepsSemicolonsInNotes() throws IOException { + final Map<String, EmojiAnnotation> parsed = parse("# comment\n\n \n" + + "1F642 ; name ; slightly smiling face ; CLDR:annotation ; note; with semicolon\n"); + assertEquals(1, parsed.size()); + assertEquals("note; with semicolon", + parsed.get(cp(0x1F642)).attribute(EmojiAnnotation.NAME).orElseThrow().notes()); + } + + // --- record validation (records are also user-constructable, for tests and custom joins) --- + + @Test + void recordConstructionFailsLoudOnBadArguments() { + final Map<String, EmojiAnnotation.Value> attributes = Map.of(EmojiAnnotation.NAME, + new EmojiAnnotation.Value("slightly smiling face", "CLDR:annotation", "")); + assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation(null, attributes)); + assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation("", attributes)); + assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation(cp(0x1F642), null)); + assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation.Value("", "s", "")); + assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation.Value("v", "", "")); + assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation.Value("v", "s", null)); + } + + @Test + void typedAccessorsFailLoudOnValuesOutsideTheirDomain() { + // Only reachable through hand-built records; the loader validates bundled data. A quietly + // swallowed conversion here would surface as a silently missing feature downstream. + final EmojiAnnotation annotation = new EmojiAnnotation(cp(0x1F642), Map.of( + EmojiAnnotation.SENTIMENT, new EmojiAnnotation.Value("happy", "UNSPECIFIED", ""), + EmojiAnnotation.ENTITY_TYPE, new EmojiAnnotation.Value("SMILEY", "UNSPECIFIED", ""), + EmojiAnnotation.CATEGORY, new EmojiAnnotation.Value("EMOTION", "UNSPECIFIED", ""))); + assertThrows(IllegalStateException.class, annotation::sentiment); + assertThrows(IllegalStateException.class, annotation::entityType); + assertThrows(IllegalStateException.class, annotation::category); + assertThrows(IllegalArgumentException.class, () -> annotation.attribute(null)); + } +} diff --git a/src/license/NOTICE.template b/src/license/NOTICE.template index 593720c09..da75e2a6a 100644 --- a/src/license/NOTICE.template +++ b/src/license/NOTICE.template @@ -123,6 +123,15 @@ top of each bundled file. These files are distributed under the Unicode License V3, the full text of which is reproduced in the LICENSE file accompanying this distribution. +The project-authored annotation table +opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt +is licensed under the Apache License 2.0, but its name values (the CLDR short +names) and its entity-type/category derivations are transcribed from the +upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51, +version 17.0), distributed under the Unicode License V3 reproduced in the +LICENSE file. Its sentiment scores are original project judgments; no +third-party sentiment data set is included. + Copyright (c) 1991-2025 Unicode, Inc. All rights reserved. ============================================================================ \ No newline at end of file
