This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch OPENNLP-1870 in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 89742e06ca70726d36f0d9dc5851178ebbd539a9 Author: Kristian Rickert <[email protected]> AuthorDate: Mon Jul 6 22:08:54 2026 -0400 OPENNLP-1870: EmojiAnnotator, the opt-in Term-adjacent annotation surface Adds the surface that assembles one EmojiAnnotation per symbol from the three strictly separated layers: bundled facts (EmojiAnnotations), derived facts (EmojiFlags), and joined facts through the new EmojiAnnotationJoin hook. - Design choice, justified in the javadoc: annotations are a parallel accessor surface beside Term, not new Dimension constants. A Dimension is a character-level text transform whose result is another string form of the token inside an ordered pipeline; an annotation is typed, per-symbol metadata with provenance that describes the original pictograph and composes with nothing downstream. Dimension constants would force typed facts through the stringly layer cache and give them a meaningless pipeline position, so the surface stays opt-in and consumers that only want folded text pay nothing. - annotate(Term) reads the original layer: the fold dimensions exist for matching and may have rewritten the pictograph to ASCII. - Flags materialize entirely from the derived layer (isoRegion, entityType FLAG, category FLAGS, provenance UTS51:flag-sequence/tag-sequence); bundled rows would win if they ever existed. - EmojiAnnotationJoin is the run-time hook for gazetteer-resolved facts (coordinates, GeoNames/WOF identifiers) keyed by the stable ISO code. No implementation ships, and foreign identifiers are never bundled (identifier churn, optional datasets, version coupling). The annotator fails loud (IllegalStateException) on a contract-violating join (null return or attribute collision), and the join only augments records, it never creates them. - Bulk safety: annotation is total over arbitrary tokens; degenerate flag-shaped text yields no region while the strict fail-loud decoder remains on EmojiFlags.isoRegion. A pure bundled hit returns the cached record without copying. - Manual: new Emoji annotations section in the Text Normalization chapter. Part of the OPENNLP-1852 epic. --- .../tools/util/normalizer/EmojiAnnotation.java | 15 ++ .../tools/util/normalizer/EmojiAnnotationJoin.java | 54 +++++++ .../tools/util/normalizer/EmojiAnnotations.java | 3 +- .../tools/util/normalizer/EmojiAnnotator.java | 159 ++++++++++++++++++ .../tools/util/normalizer/EmojiAnnotatorTest.java | 180 +++++++++++++++++++++ opennlp-docs/src/docbkx/normalizer.xml | 53 +++++- 6 files changed, 461 insertions(+), 3 deletions(-) diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java index b19ce8a4e..628e54083 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java @@ -54,6 +54,12 @@ public record EmojiAnnotation(String symbol, Map<String, Value> attributes) { /** The attribute key of the {@link EmojiCategory document-category hint}. */ public static final String CATEGORY = "category"; + /** + * The attribute key of the ISO 3166 region code of a flag emoji. Never a bundled row: the value + * is decoded from the code point sequence by the derived layer (see {@code EmojiFlags}). + */ + public static final String ISO_REGION = "isoRegion"; + /** * One attribute value with its provenance, mirroring one row of the bundled data file. * @@ -126,6 +132,15 @@ public record EmojiAnnotation(String symbol, Map<String, Value> attributes) { return value == null ? Optional.empty() : Optional.of(value.value()); } + /** + * {@return the ISO 3166 code of a flag emoji ({@code DE}, {@code GB-ENG}), or empty when this + * record is not a flag} Populated by the derived layer through {@code EmojiAnnotator}. + */ + public Optional<String> isoRegion() { + final Value value = attributes.get(ISO_REGION); + return value == null ? Optional.empty() : Optional.of(value.value()); + } + /** * {@return the project-authored coarse sentiment score in -2..2, or empty when not annotated} * diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotationJoin.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotationJoin.java new file mode 100644 index 000000000..3ec1f8afa --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotationJoin.java @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.util.normalizer; + +import java.util.Map; + +/** + * The joined-facts layer of the emoji annotation record store: a hook through which facts that + * live in user-installed data, typically a gazetteer (coordinates, containment chain, GeoNames or + * Who's On First identifiers), are resolved at run time and merged into an + * {@link EmojiAnnotation}. Register an implementation on an + * {@link EmojiAnnotator#EmojiAnnotator(EmojiAnnotationJoin) EmojiAnnotator}; no implementation is + * shipped. + * + * <p>Foreign identifiers are never baked into the bundled data file, by design: they get + * deprecated and superseded upstream (identifier churn), the datasets they point into are + * optional downloads, and baking them in would couple the bundled table's version to the + * dataset's version. The stable join key is the ISO 3166 code decoded by {@link EmojiFlags}, so + * a flag emoji resolves against whatever region data the user has installed. Checking that a + * decoded code is an <em>assigned</em> region also happens here, not in bundled or derived + * data.</p> + */ +@FunctionalInterface +public interface EmojiAnnotationJoin { + + /** + * Resolves joined attribute values for one symbol. Called by {@link EmojiAnnotator} only while + * an annotation is being assembled, that is when the symbol already has bundled or derived + * facts; the join augments records, it never creates them. + * + * @param symbol The annotated code point sequence, without the U+FE0F presentation selector. + * @param isoRegion The ISO 3166 code decoded from the symbol, or {@code null} when the symbol + * is not a flag. + * @return The joined attribute values keyed by attribute name, empty when there is nothing to + * add; never {@code null}. Keys must not collide with attributes already on the record + * (the annotator fails loud on a collision), and every value carries its own provenance in + * {@link EmojiAnnotation.Value#source()}. + */ + Map<String, EmojiAnnotation.Value> join(String symbol, String isoRegion); +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java index a191f803f..6a45371e7 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java @@ -90,7 +90,8 @@ public final class EmojiAnnotations { } // Removes every U+FE0F VARIATION SELECTOR-16; allocation-free when none is present. - private static String stripPresentationSelector(CharSequence symbol) { + // Package-visible so EmojiAnnotator keys derived-only records the same way. + static String stripPresentationSelector(CharSequence symbol) { final int length = symbol.length(); int i = 0; while (i < length && symbol.charAt(i) != 0xFE0F) { diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java new file mode 100644 index 000000000..4b5050940 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java @@ -0,0 +1,159 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.util.normalizer; + +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Optional; + +/** + * The opt-in annotation surface of the emoji record store: assembles one {@link EmojiAnnotation} + * per symbol from the three strictly separated layers, bundled facts + * ({@link EmojiAnnotations}), derived facts ({@link EmojiFlags}), and joined facts (an optional + * {@link EmojiAnnotationJoin}). + * + * <p><strong>Design note: a parallel accessor surface beside {@link Term}, not new + * {@link Dimension} constants.</strong> A {@code Dimension} is a character-level text transform: + * each constant produces another <em>string form</em> of the token, the constants form an ordered + * pipeline in which every layer feeds the next, and {@link Term#normalized()}, + * {@link Term#at(Dimension)}, and {@link Term#peel()} all walk that stack of strings. An emoji + * annotation is neither a string form nor a pipeline stage: it is typed, per-symbol + * <em>metadata</em> (a score, two enums, a name, a region code, each with provenance) that + * describes the original pictograph and composes with nothing downstream of it. Modeling it as + * {@code Dimension} constants would force those typed facts through the stringly + * {@code Map<Dimension, String>} layer cache, would give them a meaningless position in the + * transform order, and would break the {@code Dimension} contract that every layer is applied on + * top of the previous one. So annotations stay a parallel, opt-in lookup keyed by the token: + * consumers that only want folded text never touch this class and pay nothing for it.</p> + * + * <p>{@link #annotate(Term)} reads {@link Term#original()}, the source of truth: the derived + * layers exist for matching and may have folded the pictograph away entirely (for example + * {@link Dimension#EMOJI_FOLD} rewrites it to an ASCII emoticon), while annotations describe the + * symbol the author actually wrote.</p> + * + * <p>Bulk safety: annotation is total over arbitrary token text. A token that is not an annotated + * symbol returns empty, and degenerate flag-shaped text (a lone regional indicator in damaged + * input) returns no region rather than throwing; the strict, fail-loud decoding contract lives on + * {@link EmojiFlags#isoRegion(CharSequence)} for callers that want it. An instance is immutable + * and thread-safe when its join is.</p> + */ +public final class EmojiAnnotator { + + // Provenance tags of the derived facts; the mechanisms are defined by UTS #51. + private static final String FLAG_SEQUENCE = "UTS51:flag-sequence"; + private static final String TAG_SEQUENCE = "UTS51:tag-sequence"; + private static final String DERIVED_NOTE = "decoded from the code point sequence"; + + private final EmojiAnnotationJoin join; + + /** + * Creates an annotator over the bundled and derived layers only. + */ + public EmojiAnnotator() { + this.join = null; + } + + /** + * Creates an annotator that also resolves joined facts through {@code join}. + * + * @param join The joined-facts hook, called while a record is assembled. Must not be + * {@code null}. + * @throws IllegalArgumentException if {@code join} is {@code null}. + */ + public EmojiAnnotator(EmojiAnnotationJoin join) { + if (join == null) { + throw new IllegalArgumentException("Join must not be null"); + } + this.join = join; + } + + /** + * Annotates one term, keyed on its {@link Term#original() original} text (see the class note on + * why the original layer is the one annotations describe). + * + * @param term The term to annotate. Must not be {@code null}. + * @return The assembled record, or empty when the term is not an annotated symbol. + * @throws IllegalArgumentException if {@code term} is {@code null}. + * @throws IllegalStateException if the configured join violates its contract (returns + * {@code null} or a key colliding with an existing attribute). + */ + public Optional<EmojiAnnotation> annotate(Term term) { + if (term == null) { + throw new IllegalArgumentException("Term must not be null"); + } + return annotate(term.original()); + } + + /** + * Annotates one symbol. + * + * @param symbol The code point sequence of one symbol (one token). U+FE0F presentation + * selectors are ignored. Must not be {@code null}. + * @return The assembled record, or empty when {@code symbol} is not an annotated symbol. + * @throws IllegalArgumentException if {@code symbol} is {@code null}. + * @throws IllegalStateException if the configured join violates its contract (returns + * {@code null} or a key colliding with an existing attribute). + */ + public Optional<EmojiAnnotation> annotate(CharSequence symbol) { + if (symbol == null) { + throw new IllegalArgumentException("Symbol must not be null"); + } + final EmojiAnnotation bundled = EmojiAnnotations.lookup(symbol).orElse(null); + // The total predicate, not the strict decoder: degenerate flag-shaped tokens in real-world + // text must yield no region, not an exception. + final String isoRegion = EmojiFlags.isFlag(symbol) + ? EmojiFlags.isoRegion(symbol).orElseThrow() : null; + if (bundled == null && isoRegion == null) { + return Optional.empty(); // nothing bundled, nothing derived: the join never creates records + } + final String recordSymbol = bundled != null ? bundled.symbol() + : EmojiAnnotations.stripPresentationSelector(symbol); + final Map<String, EmojiAnnotation.Value> attributes = new LinkedHashMap<>(); + if (bundled != null) { + attributes.putAll(bundled.attributes()); + } + if (isoRegion != null) { + final String source = isoRegion.indexOf('-') >= 0 ? TAG_SEQUENCE : FLAG_SEQUENCE; + attributes.put(EmojiAnnotation.ISO_REGION, + new EmojiAnnotation.Value(isoRegion, source, DERIVED_NOTE)); + // A flag is a FLAG in the FLAGS group by the sequence mechanics alone; bundled rows, if any + // ever exist for a flag, would win. + attributes.putIfAbsent(EmojiAnnotation.ENTITY_TYPE, + new EmojiAnnotation.Value(EmojiEntityType.FLAG.name(), source, DERIVED_NOTE)); + attributes.putIfAbsent(EmojiAnnotation.CATEGORY, + new EmojiAnnotation.Value(EmojiCategory.FLAGS.name(), source, DERIVED_NOTE)); + } + if (join != null) { + final Map<String, EmojiAnnotation.Value> joined = join.join(recordSymbol, isoRegion); + if (joined == null) { + throw new IllegalStateException( + "The join returned null for symbol '" + recordSymbol + "'; return an empty map"); + } + for (final Map.Entry<String, EmojiAnnotation.Value> entry : joined.entrySet()) { + if (attributes.putIfAbsent(entry.getKey(), entry.getValue()) != null) { + throw new IllegalStateException("The join returned attribute '" + entry.getKey() + + "' for symbol '" + recordSymbol + "', which collides with an existing attribute"); + } + } + } + // A pure bundled hit needs no new record; return the cached one. + if (bundled != null && attributes.size() == bundled.attributes().size()) { + return Optional.of(bundled); + } + return Optional.of(new EmojiAnnotation(recordSymbol, attributes)); + } +} diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorTest.java new file mode 100644 index 000000000..89d1393f8 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorTest.java @@ -0,0 +1,180 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package opennlp.tools.util.normalizer; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class EmojiAnnotatorTest { + + private static String cp(int... codePoints) { + final StringBuilder sb = new StringBuilder(); + for (final int codePoint : codePoints) { + sb.appendCodePoint(codePoint); + } + return sb.toString(); + } + + private static final String GERMAN_FLAG = cp(0x1F1E9, 0x1F1EA); + + @Test + void annotatesATermFromItsOriginalLayer() { + // The end-to-end path the surface exists for: UAX #29 segmentation keeps the pictograph (with + // its presentation selector) as one token, the fold dimension rewrites it to ASCII for + // matching, and the annotation still resolves because it reads the original layer. + final TermAnalyzer analyzer = TermAnalyzer.builder().emojiFold().build(); + final List<Term> terms = analyzer.analyze("I " + cp(0x2764, 0xFE0F) + " pizza"); + final Term heart = terms.get(1); + assertEquals(cp(0x2764, 0xFE0F), heart.original()); + assertEquals("<3", heart.normalized()); // the fold layer has rewritten the pictograph away + final EmojiAnnotation annotation = new EmojiAnnotator().annotate(heart).orElseThrow(); + assertEquals("red heart", annotation.name().orElseThrow()); + assertEquals(2, annotation.sentiment().orElseThrow()); + assertEquals(EmojiEntityType.HEART, annotation.entityType().orElseThrow()); + assertEquals(EmojiCategory.SMILEYS_AND_EMOTION, annotation.category().orElseThrow()); + assertEquals(Optional.empty(), annotation.isoRegion()); + } + + @Test + void aPureBundledHitReturnsTheCachedRecord() { + // No join, no derived facts: the annotator must not copy the record per token. + final EmojiAnnotation direct = EmojiAnnotations.lookup(cp(0x1F642)).orElseThrow(); + assertSame(direct, new EmojiAnnotator().annotate(cp(0x1F642)).orElseThrow()); + } + + @Test + void annotatesAFlagTermWithDerivedFacts() { + // A flag has no bundled rows; entity type, category, and the region are all derived from the + // sequence itself, provenance-tagged with the UTS #51 mechanism that defines them. + final TermAnalyzer analyzer = TermAnalyzer.builder().build(); + final List<Term> terms = analyzer.analyze("Berlin " + GERMAN_FLAG); + final EmojiAnnotation annotation = + new EmojiAnnotator().annotate(terms.get(1)).orElseThrow(); + assertEquals(GERMAN_FLAG, annotation.symbol()); + assertEquals("DE", annotation.isoRegion().orElseThrow()); + assertEquals(EmojiEntityType.FLAG, annotation.entityType().orElseThrow()); + assertEquals(EmojiCategory.FLAGS, annotation.category().orElseThrow()); + assertEquals(Optional.empty(), annotation.name()); + assertEquals("UTS51:flag-sequence", + annotation.attribute(EmojiAnnotation.ISO_REGION).orElseThrow().source()); + } + + @Test + void annotatesASubdivisionFlag() { + final StringBuilder england = new StringBuilder(cp(0x1F3F4)); + for (final char c : "gbeng".toCharArray()) { + england.appendCodePoint(0xE0000 + c); + } + england.appendCodePoint(0xE007F); + final EmojiAnnotation annotation = + new EmojiAnnotator().annotate(england).orElseThrow(); + assertEquals("GB-ENG", annotation.isoRegion().orElseThrow()); + assertEquals("UTS51:tag-sequence", + annotation.attribute(EmojiAnnotation.ISO_REGION).orElseThrow().source()); + } + + @Test + void annotationIsTotalOverArbitraryTokens() { + final EmojiAnnotator annotator = new EmojiAnnotator(); + assertEquals(Optional.empty(), annotator.annotate("word")); + assertEquals(Optional.empty(), annotator.annotate("")); + assertEquals(Optional.empty(), annotator.annotate(":-)")); + // A lone regional indicator in damaged text yields no annotation, never an exception; the + // fail-loud contract lives on EmojiFlags.isoRegion for direct callers. + assertEquals(Optional.empty(), annotator.annotate(cp(0x1F1E9))); + // An unmapped pictograph is not annotated either. + assertEquals(Optional.empty(), annotator.annotate(cp(0x1F9F8))); + } + + @Test + void joinedFactsMergeIntoTheRecord() { + final List<String> calls = new ArrayList<>(); + final EmojiAnnotator annotator = new EmojiAnnotator((symbol, isoRegion) -> { + calls.add(symbol + "/" + isoRegion); + if ("DE".equals(isoRegion)) { + return Map.of("geonamesId", + new EmojiAnnotation.Value("2921044", "user:gazetteer", "join test")); + } + return Map.of(); + }); + final EmojiAnnotation flag = annotator.annotate(GERMAN_FLAG).orElseThrow(); + assertEquals("2921044", flag.attribute("geonamesId").orElseThrow().value()); + assertEquals("user:gazetteer", flag.attribute("geonamesId").orElseThrow().source()); + assertEquals("DE", flag.isoRegion().orElseThrow()); + // A bundled non-flag symbol is joined with a null region. + final EmojiAnnotation heart = annotator.annotate(cp(0x2764)).orElseThrow(); + assertEquals(Optional.empty(), heart.attribute("geonamesId")); + assertEquals("red heart", heart.name().orElseThrow()); + // The join never runs for unannotated tokens: it augments records, it does not create them. + assertEquals(Optional.empty(), annotator.annotate("word")); + assertEquals(List.of(GERMAN_FLAG + "/DE", cp(0x2764) + "/null"), calls); + } + + @Test + void aContractViolatingJoinFailsLoud() { + final EmojiAnnotator returnsNull = new EmojiAnnotator((symbol, isoRegion) -> null); + assertThrows(IllegalStateException.class, () -> returnsNull.annotate(GERMAN_FLAG)); + // Colliding with a bundled attribute. + final EmojiAnnotator collidesBundled = new EmojiAnnotator((symbol, isoRegion) -> + Map.of(EmojiAnnotation.NAME, new EmojiAnnotation.Value("x", "user:gazetteer", ""))); + assertThrows(IllegalStateException.class, () -> collidesBundled.annotate(cp(0x1F642))); + // Colliding with a derived attribute. + final EmojiAnnotator collidesDerived = new EmojiAnnotator((symbol, isoRegion) -> + Map.of(EmojiAnnotation.ISO_REGION, new EmojiAnnotation.Value("XX", "user:gazetteer", ""))); + assertThrows(IllegalStateException.class, () -> collidesDerived.annotate(GERMAN_FLAG)); + } + + @Test + void argumentsAreValidated() { + assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotator(null)); + final EmojiAnnotator annotator = new EmojiAnnotator(); + assertThrows(IllegalArgumentException.class, () -> annotator.annotate((Term) null)); + assertThrows(IllegalArgumentException.class, () -> annotator.annotate((CharSequence) null)); + } + + @Test + void flagTokensSurviveSegmentation() { + // Adjacent flags segment into their pairs under UAX #29 (WB15/WB16), so per-token annotation + // sees each flag whole; this pins the assumption the bulk surface relies on. + final TermAnalyzer analyzer = TermAnalyzer.builder().build(); + final List<Term> terms = analyzer.analyze(GERMAN_FLAG + cp(0x1F1EB, 0x1F1F7)); + assertEquals(2, terms.size()); + final EmojiAnnotator annotator = new EmojiAnnotator(); + assertEquals("DE", annotator.annotate(terms.get(0)).orElseThrow().isoRegion().orElseThrow()); + assertEquals("FR", annotator.annotate(terms.get(1)).orElseThrow().isoRegion().orElseThrow()); + } + + @Test + void mergedFlagRecordsKeepEveryProvenanceTag() { + // The record store contract: every value, bundled or derived, carries its own source. + final EmojiAnnotation annotation = new EmojiAnnotator().annotate(GERMAN_FLAG).orElseThrow(); + for (final Map.Entry<String, EmojiAnnotation.Value> entry : + annotation.attributes().entrySet()) { + assertTrue(!entry.getValue().source().isEmpty(), "Empty source for " + entry.getKey()); + } + assertEquals(3, annotation.attributes().size()); // isoRegion, entityType, category + } +} diff --git a/opennlp-docs/src/docbkx/normalizer.xml b/opennlp-docs/src/docbkx/normalizer.xml index 7adb613f5..8cbca7b9c 100644 --- a/opennlp-docs/src/docbkx/normalizer.xml +++ b/opennlp-docs/src/docbkx/normalizer.xml @@ -62,10 +62,12 @@ <para> Two engines underpin everything: the <code>CharSequenceNormalizer</code> family offers ready-made, composable normalizers, and the <code>CharClass</code> engine is the low-level, - configurable building block they are made of. Built on these are three higher-level + configurable building block they are made of. Built on these are four higher-level features documented below: a layered term model that projects a token through a configurable stack of transforms while keeping every intermediate form (see - <xref linkend="tools.normalizer.term"/>), per-language profiles that select the transforms + <xref linkend="tools.normalizer.term"/>), an opt-in emoji annotation layer that turns + pictographs into typed signal (see <xref linkend="tools.normalizer.emojiannotations"/>), + per-language profiles that select the transforms appropriate to a language (see <xref linkend="tools.normalizer.language"/>), and confusable folding that reduces lookalike characters for matching (see <xref linkend="tools.normalizer.confusables"/>). @@ -521,6 +523,53 @@ Term term = analyzer.analyze("Running").get(0); </para> </section> + <section xml:id="tools.normalizer.emojiannotations"> + <title>Emoji annotations</title> + <para> + Beyond folding, a pictograph carries signal worth keeping: a name, a coarse sentiment, an + entity type, a document category, and for flags a region. <code>EmojiAnnotator</code> is the + opt-in surface for that signal. It assembles one <code>EmojiAnnotation</code> record per + symbol from three strictly separated layers: <emphasis>bundled facts</emphasis> from the + provenance-tagged, project-authored <code>emoji-annotations.txt</code> table, + <emphasis>derived facts</emphasis> computed from the code point sequence itself + (<code>EmojiFlags</code> decodes a regional indicator pair to its ISO 3166-1 alpha-2 code and + a subdivision tag sequence to its ISO 3166-2 code, with no data file), and optional + <emphasis>joined facts</emphasis> resolved at run time through an + <code>EmojiAnnotationJoin</code> hook against user-installed data such as a gazetteer. + Every attribute value carries its own provenance source, and gazetteer identifiers are never + baked into the bundled table. + </para> + <programlisting language="java"> +<![CDATA[EmojiAnnotator annotator = new EmojiAnnotator(); +TermAnalyzer analyzer = TermAnalyzer.builder().emojiFold().build(); + +// "Berlin <German flag> <red heart>" +String text = "Berlin \uD83C\uDDE9\uD83C\uDDEA \u2764\uFE0F"; +for (Term term : analyzer.analyze(text)) { + annotator.annotate(term).ifPresent(a -> { + a.name(); // "red heart" for the heart (CLDR short name) + a.sentiment(); // 2 for the heart (project-authored coarse score, -2..2) + a.entityType(); // FLAG for the flag, HEART for the heart + a.category(); // FLAGS / SMILEYS_AND_EMOTION + a.isoRegion(); // "DE" for the flag, decoded from the sequence itself + }); +}]]> + </programlisting> + <para> + Annotations are per-symbol <emphasis>metadata</emphasis>, not text transforms, so they are a + parallel accessor surface beside <code>Term</code> rather than additional + <code>Dimension</code>s: <code>annotate(Term)</code> reads the token's original layer, which + is the symbol the author actually wrote even after a fold dimension has rewritten it to + ASCII. Annotation is total over arbitrary tokens (a non-emoji token is simply empty, and + degenerate flag-shaped text yields no region instead of an exception), while the strict + decoder <code>EmojiFlags.isoRegion()</code> fails loudly on malformed sequences for callers + that want the contract. The bundled table's sentiment scores are original project judgments + (tagged <code>UNSPECIFIED</code>); its names and its entity-type/category assignments derive + from the Unicode-licensed CLDR short names and <code>emoji-test.txt</code> group headers, and + no third-party sentiment data set is included. + </para> + </section> + <section xml:id="tools.normalizer.confusables"> <title>Confusable (homoglyph) folding</title> <para>
