This is an automated email from the ASF dual-hosted git repository.
mawiesne pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 9951bfd0a OPENNLP-1870: Add opt-in emoji annotations for NameFinder,
Doccat, and Sentiment (#1177)
9951bfd0a is described below
commit 9951bfd0ae0aee8ca77eb436702e5544fbea2afc
Author: Kristian Rickert <[email protected]>
AuthorDate: Tue Jul 21 08:20:51 2026 -0400
OPENNLP-1870: Add opt-in emoji annotations for NameFinder, Doccat, and
Sentiment (#1177)
Adds the bundled-facts layer of the emoji annotation record store: a
provenance-tagged, project-authored attribute table (emoji-annotations.txt)
loaded by EmojiAnnotations into per-symbol EmojiAnnotation records.
- Attribute-row format (codepoints ; attribute ; value ; source ; notes), so
adding an attribute later is new rows plus loader support instead of a
file
format break, and every value carries its own provenance.
- Attributes: name (CLDR short name, CLDR:annotation), sentiment (project
judgment on a coarse -2..2 scale, UNSPECIFIED), entityType and category
(assigned from the emoji-test.txt subgroup/group headers, UCD:emoji-test).
No third-party sentiment data set is copied; in particular the CC BY-SA
Emoji Sentiment Ranking is not used in any form.
- The loader fails loud on an unknown attribute or malformed row
(IllegalArgumentException naming the resource and line): the data and the
code move together, so an unrecognized attribute is corruption, not
extensibility. Values are validated into their typed domains at load.
- Lookups strip U+FE0F, since presentation selection does not change a
symbol's identity.
- Audit tests pin coverage (every EMOJI-direction source of
emoji-emoticons.txt has a complete four-attribute record), per-attribute
provenance tags, typed domains, and the record/row counts (40/160).
- LICENSE/NOTICE/NOTICE.template attribution for the Unicode-derived values
(CLDR short names, emoji-test.txt group/subgroup classifications) in the
otherwise project-authored table.
Flag emoji intentionally have no bundled rows; their region decodes from the
code point sequence itself in the derived-facts layer (next commit), and
gazetteer identifiers are never baked into bundled data.
---
LICENSE | 8 +-
NOTICE | 9 +
.../tools/doccat/EmojiFeatureGenerator.java | 82 +++++++
.../sentiment/EmojiSentimentContextGenerator.java | 87 +++++++
.../tools/sentiment/EmojiSentimentFactory.java | 46 ++++
.../EmojiAnnotationFeatureGenerator.java | 83 +++++++
.../EmojiAnnotationFeatureGeneratorFactory.java | 51 ++++
.../tools/util/normalizer/EmojiAnnotation.java | 197 ++++++++++++++++
.../tools/util/normalizer/EmojiAnnotationJoin.java | 51 ++++
.../tools/util/normalizer/EmojiAnnotations.java | 256 +++++++++++++++++++++
.../tools/util/normalizer/EmojiAnnotator.java | 197 ++++++++++++++++
.../tools/util/normalizer/EmojiCategory.java | 62 +++++
.../tools/util/normalizer/EmojiEntityType.java | 57 +++++
.../opennlp/tools/util/normalizer/EmojiFlags.java | 227 ++++++++++++++++++
.../tools/util/normalizer/emoji-annotations.txt | 212 +++++++++++++++++
.../tools/doccat/EmojiFeatureGeneratorTest.java | 80 +++++++
.../EmojiSentimentContextGeneratorTest.java | 72 ++++++
.../EmojiAnnotationFeatureGeneratorTest.java | 127 ++++++++++
.../util/normalizer/EmojiAnnotationsTest.java | 256 +++++++++++++++++++++
.../tools/util/normalizer/EmojiAnnotatorTest.java | 176 ++++++++++++++
.../normalizer/EmojiAnnotatorUsageExampleTest.java | 65 ++++++
.../tools/util/normalizer/EmojiFlagsTest.java | 147 ++++++++++++
.../tools/util/normalizer/NormalizerTestUtil.java | 42 ++++
opennlp-distr/src/main/readme/LICENSE | 5 +-
opennlp-docs/src/docbkx/normalizer.xml | 67 +++++-
src/license/NOTICE.template | 9 +
26 files changed, 2666 insertions(+), 5 deletions(-)
diff --git a/LICENSE b/LICENSE
index 1b92a4dc6..e77ddad5f 100644
--- a/LICENSE
+++ b/LICENSE
@@ -375,9 +375,13 @@ The following license applies to the bundled Unicode data
files in
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/tokenize/uax29
(WordBreakProperty.txt, ExtendedPictographic.txt),
opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer
-(confusables.txt, CaseFolding.txt), and
+(confusables.txt, CaseFolding.txt),
opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/tokenize/uax29
-(WordBreakTest.txt):
+(WordBreakTest.txt), and to the Unicode-derived values (the CLDR short names
+and the emoji-test.txt group/subgroup classifications) in the otherwise
+project-authored
+opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
+(see NOTICE):
UNICODE LICENSE V3
diff --git a/NOTICE b/NOTICE
index bca534b1f..c503a5f8f 100644
--- a/NOTICE
+++ b/NOTICE
@@ -123,6 +123,15 @@ top of each bundled file. These files are distributed
under the Unicode License
V3, the full text of which is reproduced in the LICENSE file accompanying this
distribution.
+The project-authored annotation table
+opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
+is licensed under the Apache License 2.0, but its name values (the CLDR short
+names) and its entity-type/category derivations are transcribed from the
+upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51,
+version 17.0), distributed under the Unicode License V3 reproduced in the
+LICENSE file. Its sentiment scores are original project judgments; no
+third-party sentiment data set is included.
+
Copyright (c) 1991-2025 Unicode, Inc. All rights reserved.
============================================================================
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java
new file mode 100644
index 000000000..c06e4fc2d
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.doccat;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.util.normalizer.EmojiAnnotator;
+
+/**
+ * Generates document features from the emoji annotation layer: for every
annotated token the
+ * project-authored coarse sentiment score ({@code emojiSentiment=2}), the
entity type
+ * ({@code emojiType=HEART}), the document category hint
+ * ({@code emojiCategory=SMILEYS_AND_EMOTION}), and for flags the ISO 3166
region
+ * ({@code emojiRegion=DE}). Tokens that are not annotated symbols contribute
nothing.
+ *
+ * <p>Strictly opt-in: pass it alongside the defaults, for example
+ * {@code new DoccatFactory(new FeatureGenerator[] {new
BagOfWordsFeatureGenerator(),
+ * new EmojiFeatureGenerator()})}; the default factory configuration is
unchanged. The no-argument
+ * constructor is required so a trained model can re-instantiate the generator
from its
+ * manifest.</p>
+ *
+ * @see EmojiAnnotator
+ */
+public class EmojiFeatureGenerator implements FeatureGenerator {
+
+ private final EmojiAnnotator annotator;
+
+ /**
+ * Instantiates a generator over the bundled and derived annotation layers.
+ */
+ public EmojiFeatureGenerator() {
+ this(new EmojiAnnotator());
+ }
+
+ /**
+ * Instantiates a generator over a configured annotator, for example one
with a gazetteer join.
+ *
+ * @param annotator The annotator to use. Must not be {@code null}.
+ * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ */
+ public EmojiFeatureGenerator(EmojiAnnotator annotator) {
+ if (annotator == null) {
+ throw new IllegalArgumentException("Annotator must not be null");
+ }
+ this.annotator = annotator;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * @throws IllegalArgumentException if {@code text} is {@code null}.
+ */
+ @Override
+ public Collection<String> extractFeatures(String[] text, Map<String, Object>
extraInformation) {
+ if (text == null) {
+ throw new IllegalArgumentException("Text must not be null");
+ }
+ final List<String> features = new ArrayList<>();
+ for (final String token : text) {
+ annotator.collectFeatures(token, features);
+ }
+ return features;
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java
new file mode 100644
index 000000000..a10aac0d7
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.sentiment;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import opennlp.tools.util.normalizer.EmojiAnnotator;
+
+/**
+ * A {@link SentimentContextGenerator} that adds emoji annotation features to
the default
+ * token context: for every annotated token the project-authored coarse
sentiment score
+ * ({@code emojiSentiment=2}), the entity type ({@code emojiType=HEART}), the
category
+ * ({@code emojiCategory=SMILEYS_AND_EMOTION}), and for flags the region
({@code emojiRegion=DE}).
+ * A crying-face or heart pictograph thereby becomes direct evidence for the
sentiment model
+ * instead of an opaque token.
+ *
+ * <p>Strictly opt-in through the {@link SentimentFactory} seam: train with
+ * {@link EmojiSentimentFactory} (or any factory whose
+ * {@link SentimentFactory#createContextGenerator()} returns this class) and
the same context is
+ * regenerated at prediction time; the default factory and existing models are
unchanged.</p>
+ *
+ * @see EmojiSentimentFactory
+ * @see EmojiAnnotator
+ */
+public class EmojiSentimentContextGenerator extends SentimentContextGenerator {
+
+ private final EmojiAnnotator annotator;
+
+ /**
+ * Instantiates a context generator over the bundled and derived annotation
layers.
+ */
+ public EmojiSentimentContextGenerator() {
+ this(new EmojiAnnotator());
+ }
+
+ /**
+ * Instantiates a context generator over a configured annotator, for example
one with a
+ * gazetteer join.
+ *
+ * @param annotator The annotator to use. Must not be {@code null}.
+ * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ */
+ public EmojiSentimentContextGenerator(EmojiAnnotator annotator) {
+ if (annotator == null) {
+ throw new IllegalArgumentException("Annotator must not be null");
+ }
+ this.annotator = annotator;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Appends the emoji annotation features of every annotated token to the
default token
+ * context.</p>
+ *
+ * @throws IllegalArgumentException if {@code text} is {@code null}.
+ */
+ @Override
+ public String[] getContext(String[] text) {
+ if (text == null) {
+ throw new IllegalArgumentException("Text must not be null");
+ }
+ final List<String> context = new ArrayList<>(text.length);
+ Collections.addAll(context, super.getContext(text));
+ for (final String token : text) {
+ annotator.collectFeatures(token, context);
+ }
+ return context.toArray(new String[0]);
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentFactory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentFactory.java
new file mode 100644
index 000000000..97e03600d
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentFactory.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.sentiment;
+
+/**
+ * A {@link SentimentFactory} whose context generator adds emoji annotation
features (see
+ * {@link EmojiSentimentContextGenerator}). Pass it to
+ * {@link SentimentME#train(String, opennlp.tools.util.ObjectStream,
+ * opennlp.tools.util.TrainingParameters, SentimentFactory) SentimentME.train}
to opt in; the
+ * factory class is recorded in the trained model's manifest, so prediction
re-creates the same
+ * context. The default {@link SentimentFactory} is unchanged.
+ */
+public class EmojiSentimentFactory extends SentimentFactory {
+
+ /**
+ * Instantiates a factory whose context generator adds emoji annotation
features.
+ */
+ public EmojiSentimentFactory() {
+ super();
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Returns an {@link EmojiSentimentContextGenerator}.</p>
+ */
+ @Override
+ public SentimentContextGenerator createContextGenerator() {
+ return new EmojiSentimentContextGenerator();
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java
new file mode 100644
index 000000000..2e6d0d003
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util.featuregen;
+
+import java.util.List;
+
+import opennlp.tools.util.normalizer.EmojiAnnotator;
+
+/**
+ * Generates features from the emoji annotation layer for the token at the
given index: the
+ * project-authored coarse sentiment score ({@code emojiSentiment=2}), the
entity type
+ * ({@code emojiType=HEART}), the document category ({@code
emojiCategory=SMILEYS_AND_EMOTION}),
+ * and for flags the ISO 3166 region ({@code emojiRegion=DE}). For the name
finder the entity
+ * type behaves like gazetteer evidence: a flag token carries {@code
emojiType=FLAG} and its
+ * region without any dictionary.
+ *
+ * <p>Strictly opt-in: this generator only runs when a feature generation
descriptor names
+ * {@link EmojiAnnotationFeatureGeneratorFactory}, so default descriptors and
existing models are
+ * unchanged. A token that is not an annotated symbol contributes no
features.</p>
+ *
+ * <p>Unlike most feature generators this one is stateless and thread-safe
(when its annotator's
+ * join is): it keeps no adaptive data.</p>
+ *
+ * @see EmojiAnnotator
+ */
+public class EmojiAnnotationFeatureGenerator implements
AdaptiveFeatureGenerator {
+
+ private final EmojiAnnotator annotator;
+
+ /**
+ * Instantiates a generator over the bundled and derived annotation layers.
+ */
+ public EmojiAnnotationFeatureGenerator() {
+ this(new EmojiAnnotator());
+ }
+
+ /**
+ * Instantiates a generator over a configured annotator, for example one
with a gazetteer join.
+ *
+ * @param annotator The annotator to use. Must not be {@code null}.
+ * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ */
+ public EmojiAnnotationFeatureGenerator(EmojiAnnotator annotator) {
+ if (annotator == null) {
+ throw new IllegalArgumentException("Annotator must not be null");
+ }
+ this.annotator = annotator;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Adds one feature per present annotation attribute of the token at
{@code index}.</p>
+ *
+ * @throws IllegalArgumentException if {@code features} or {@code tokens} is
{@code null}.
+ */
+ @Override
+ public void createFeatures(List<String> features, String[] tokens, int index,
+ String[] previousOutcomes) {
+ if (features == null) {
+ throw new IllegalArgumentException("Features must not be null");
+ }
+ if (tokens == null) {
+ throw new IllegalArgumentException("Tokens must not be null");
+ }
+ annotator.collectFeatures(tokens[index], features);
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorFactory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorFactory.java
new file mode 100644
index 000000000..1a37185aa
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorFactory.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util.featuregen;
+
+import opennlp.tools.util.InvalidFormatException;
+
+/**
+ * A {@link GeneratorFactory} that produces {@link
EmojiAnnotationFeatureGenerator} instances
+ * when {@link #create()} is called, for use in a feature generation
descriptor:
+ *
+ * <pre>{@code
+ * <generator
class="opennlp.tools.util.featuregen.EmojiAnnotationFeatureGeneratorFactory"/>
+ * }</pre>
+ *
+ * @see EmojiAnnotationFeatureGenerator
+ */
+public class EmojiAnnotationFeatureGeneratorFactory
+ extends GeneratorFactory.AbstractXmlFeatureGeneratorFactory {
+
+ /**
+ * Instantiates a factory that produces {@link
EmojiAnnotationFeatureGenerator} instances.
+ */
+ public EmojiAnnotationFeatureGeneratorFactory() {
+ super();
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Returns a new {@link EmojiAnnotationFeatureGenerator}.</p>
+ */
+ @Override
+ public AdaptiveFeatureGenerator create() throws InvalidFormatException {
+ return new EmojiAnnotationFeatureGenerator();
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
new file mode 100644
index 000000000..393032074
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+
+/**
+ * The annotation record of one emoji: an extensible per-symbol store of
attribute values, each
+ * carrying its own provenance. A record is assembled from up to three
strictly separated layers:
+ * bundled facts from {@code emoji-annotations.txt} (see {@link
EmojiAnnotations}), derived facts
+ * computed from the code point sequence itself, and joined facts resolved at
run time against
+ * user-installed data. The typed accessors ({@link #name()}, {@link
#sentiment()},
+ * {@link #entityType()}, {@link #category()}) are projections of the {@link
#attributes()} map,
+ * so an attribute added later is new rows plus an accessor, not a shape
change.
+ *
+ * <p>Every attribute is optional: a consumer that only wants one dimension
pays nothing for the
+ * rest, and a record never fabricates a value it has no source for. Instances
are immutable and
+ * thread-safe.</p>
+ *
+ * @param symbol The annotated code point sequence, without the U+FE0F
presentation selector.
+ * @param attributes The attribute values keyed by attribute name ({@link
#NAME},
+ * {@link #SENTIMENT}, {@link #ENTITY_TYPE}, {@link
#CATEGORY}).
+ */
+public record EmojiAnnotation(String symbol, Map<String, Value> attributes) {
+
+ /** The attribute key of the human-readable name (the CLDR short name). */
+ public static final String NAME = "name";
+
+ /** The attribute key of the project-authored coarse sentiment score, an
integer in -2..2. */
+ public static final String SENTIMENT = "sentiment";
+
+ /** The attribute key of the coarse {@link EmojiEntityType entity type}. */
+ public static final String ENTITY_TYPE = "entityType";
+
+ /** The attribute key of the {@link EmojiCategory document-category hint}. */
+ public static final String CATEGORY = "category";
+
+ /**
+ * The attribute key of the ISO 3166 region code of a flag emoji. Never a
bundled row: the value
+ * is decoded from the code point sequence by the derived layer (see {@code
EmojiFlags}).
+ */
+ public static final String ISO_REGION = "isoRegion";
+
+ /**
+ * One attribute value with its provenance, mirroring one row of the bundled
data file.
+ *
+ * @param value The attribute value. Must not be {@code null} or empty.
+ * @param source The provenance tag, for example {@code CLDR:annotation},
{@code UCD:emoji-test},
+ * or {@code UNSPECIFIED} (the explicit marker for a project
judgment). Must not be
+ * {@code null} or empty.
+ * @param notes Free-text notes, for example the upstream subgroup a value
was derived from.
+ * Must not be {@code null}; may be empty.
+ */
+ public record Value(String value, String source, String notes) {
+
+ public Value {
+ if (value == null || value.isEmpty()) {
+ throw new IllegalArgumentException("Value must not be null or empty");
+ }
+ if (source == null || source.isEmpty()) {
+ throw new IllegalArgumentException("Source must not be null or empty");
+ }
+ if (notes == null) {
+ throw new IllegalArgumentException("Notes must not be null");
+ }
+ }
+ }
+
+ /**
+ * Creates a record; normally records come from {@link
EmojiAnnotations#lookup(CharSequence)}.
+ *
+ * @param symbol The annotated code point sequence. Must not be {@code
null} or empty.
+ * @param attributes The attribute values keyed by attribute name. Must not
be {@code null} or
+ * contain {@code null} keys or values; it is defensively
copied.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null} or
empty, or if
+ * {@code attributes} is {@code null} or contains a {@code null} key or
value.
+ */
+ public EmojiAnnotation {
+ if (symbol == null || symbol.isEmpty()) {
+ throw new IllegalArgumentException("Symbol must not be null or empty");
+ }
+ if (attributes == null) {
+ throw new IllegalArgumentException("Attributes must not be null");
+ }
+ for (final Map.Entry<String, Value> entry : attributes.entrySet()) {
+ if (entry.getKey() == null || entry.getValue() == null) {
+ throw new IllegalArgumentException(
+ "Attributes must not contain a null key or value, got: " +
attributes);
+ }
+ }
+ attributes = Map.copyOf(attributes);
+ }
+
+ /**
+ * Returns the value of one attribute with its provenance.
+ *
+ * @param name The attribute name, for example {@link #NAME}. Must not be
{@code null}.
+ * @return The value, or empty when this record does not carry the attribute.
+ * @throws IllegalArgumentException if {@code name} is {@code null}.
+ */
+ public Optional<Value> attribute(String name) {
+ if (name == null) {
+ throw new IllegalArgumentException("Name must not be null");
+ }
+ return Optional.ofNullable(attributes.get(name));
+ }
+
+ /**
+ * {@return the human-readable name (the CLDR short name), or empty when not
annotated}
+ */
+ public Optional<String> name() {
+ final Value value = attributes.get(NAME);
+ return value == null ? Optional.empty() : Optional.of(value.value());
+ }
+
+ /**
+ * {@return the ISO 3166 code of a flag emoji ({@code DE}, {@code GB-ENG}),
or empty when this
+ * record is not a flag} Populated by the derived layer through {@code
EmojiAnnotator}.
+ */
+ public Optional<String> isoRegion() {
+ final Value value = attributes.get(ISO_REGION);
+ return value == null ? Optional.empty() : Optional.of(value.value());
+ }
+
+ /**
+ * {@return the project-authored coarse sentiment score in -2..2, or empty
when not annotated}
+ *
+ * @throws IllegalStateException if the stored value is not an integer,
which cannot happen for
+ * records loaded from the bundled data (the loader validates it).
+ */
+ public OptionalInt sentiment() {
+ final Value value = attributes.get(SENTIMENT);
+ if (value == null) {
+ return OptionalInt.empty();
+ }
+ try {
+ return OptionalInt.of(Integer.parseInt(value.value()));
+ } catch (NumberFormatException e) {
+ throw new IllegalStateException("Sentiment value '" + value.value() + "'
of symbol '"
+ + symbol + "' is not an integer", e);
+ }
+ }
+
+ /**
+ * {@return the coarse entity type, or empty when not annotated}
+ *
+ * @throws IllegalStateException if the stored value is not an {@link
EmojiEntityType} constant,
+ * which cannot happen for records loaded from the bundled data (the
loader validates it).
+ */
+ public Optional<EmojiEntityType> entityType() {
+ final Value value = attributes.get(ENTITY_TYPE);
+ if (value == null) {
+ return Optional.empty();
+ }
+ try {
+ return Optional.of(EmojiEntityType.valueOf(value.value()));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalStateException("Entity type value '" + value.value() +
"' of symbol '"
+ + symbol + "' is not an EmojiEntityType constant", e);
+ }
+ }
+
+ /**
+ * {@return the document-category hint, or empty when not annotated}
+ *
+ * @throws IllegalStateException if the stored value is not an {@link
EmojiCategory} constant,
+ * which cannot happen for records loaded from the bundled data (the
loader validates it).
+ */
+ public Optional<EmojiCategory> category() {
+ final Value value = attributes.get(CATEGORY);
+ if (value == null) {
+ return Optional.empty();
+ }
+ try {
+ return Optional.of(EmojiCategory.valueOf(value.value()));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalStateException("Category value '" + value.value() + "'
of symbol '"
+ + symbol + "' is not an EmojiCategory constant", e);
+ }
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotationJoin.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotationJoin.java
new file mode 100644
index 000000000..5cbf474ff
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotationJoin.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.Map;
+
+/**
+ * The joined-facts layer of the emoji annotation record store: a hook through
which facts that
+ * live in user-installed data, typically a gazetteer (coordinates,
containment chain, GeoNames or
+ * Who's On First identifiers), are resolved at run time and merged into an
+ * {@link EmojiAnnotation}. Register an implementation on an
+ * {@link EmojiAnnotator#EmojiAnnotator(EmojiAnnotationJoin) EmojiAnnotator};
no implementation is
+ * shipped.
+ *
+ * <p>The stable join key is the ISO 3166 code decoded by {@link EmojiFlags}:
a flag emoji
+ * resolves against whatever region data the user has installed. Foreign
identifiers (for example
+ * GeoNames or Who's On First) and the check that a decoded code is an
<em>assigned</em> region
+ * live in the join, not in the bundled or derived data.</p>
+ */
+@FunctionalInterface
+public interface EmojiAnnotationJoin {
+
+ /**
+ * Resolves joined attribute values for one symbol. Called by {@link
EmojiAnnotator} only while
+ * an annotation is being assembled, that is when the symbol already has
bundled or derived
+ * facts; the join augments records, it never creates them.
+ *
+ * @param symbol The annotated code point sequence, without the U+FE0F
presentation selector.
+ * @param isoRegion The ISO 3166 code decoded from the symbol, or {@code
null} when the symbol
+ * is not a flag.
+ * @return The joined attribute values keyed by attribute name, empty when
there is nothing to
+ * add; never {@code null}. Keys must not collide with attributes
already on the record
+ * (the annotator fails loud on a collision), and every value carries
its own provenance in
+ * {@link EmojiAnnotation.Value#source()}.
+ */
+ Map<String, EmojiAnnotation.Value> join(String symbol, String isoRegion);
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
new file mode 100644
index 000000000..ff4c5582b
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * The bundled-facts layer of the emoji annotation record store:
license-clean, provenance-tagged
+ * attributes intrinsic to a pictograph (name, coarse sentiment, entity type,
document category),
+ * loaded once from the project-authored {@code emoji-annotations.txt}
resource. Each row of that
+ * file is one attribute of one symbol ({@code codepoints ; attribute ; value
; source ; notes}),
+ * so every value carries its own provenance and adding an attribute later is
new rows plus loader
+ * support instead of a file-format break. The loader fails loud on an unknown
attribute or a
+ * malformed row: the data and the code move together, so an unrecognized
attribute is corruption,
+ * not extensibility.
+ *
+ * <p>Data licensing: the name values are the CLDR short names and the
entity-type/category values
+ * are derived from the group and subgroup headers of the upstream {@code
emoji-test.txt}
+ * (UTS #51, Unicode License V3, see the NOTICE file); the sentiment
scores are original
+ * project judgments tagged {@code UNSPECIFIED}. No third-party sentiment data
set is copied; in
+ * particular the Emoji Sentiment Ranking (CC BY-SA) is not used in any
form.</p>
+ *
+ * <p>{@link EmojiAnnotator} is the accessor. Lookups strip U+FE0F (emoji
presentation selector)
+ * because it does not change identity; bundled rows are keyed without it.
Flag emoji have no
+ * bundled rows: region is derived, and gazetteer ids are never baked in.</p>
+ */
+public final class EmojiAnnotations {
+
+ private static final String RESOURCE = "emoji-annotations.txt";
+
+ /** Starts a comment line in {@code emoji-annotations.txt}. */
+ private static final String COMMENT_PREFIX = "#";
+
+ /**
+ * Field separator in {@code emoji-annotations.txt}
+ * ({@code codepoints ; attribute ; value ; source ; notes}).
+ */
+ private static final String FIELD_SEPARATOR = ";";
+
+ /**
+ * Separates hex code points inside the code point field. The bundled table
format uses ASCII
+ * space ({@code U+0020}), not a general whitespace class.
+ */
+ private static final String MAPPING_CODE_POINT_SEPARATOR = " ";
+
+ // The records keyed by code point sequence, loaded once when this class
initializes.
+ private static final Map<String, EmojiAnnotation> ANNOTATIONS = load();
+
+ private EmojiAnnotations() {
+ }
+
+ /**
+ * Returns the bundled annotation record of one emoji.
+ *
+ * @param symbol The code point sequence of one symbol, for example one
{@link Term#original()}
+ * token. U+FE0F presentation selectors are ignored. Must not
be {@code null}.
+ * @return The record, or empty when the bundled data does not annotate the
symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the bundled data resource is missing.
+ * @throws UncheckedIOException if the bundled data resource cannot be read.
+ */
+ public static Optional<EmojiAnnotation> lookup(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ final String key = stripPresentationSelector(symbol);
+ if (key.isEmpty()) {
+ return Optional.empty();
+ }
+ return Optional.ofNullable(ANNOTATIONS.get(key));
+ }
+
+ /**
+ * Removes every U+FE0F VARIATION SELECTOR-16 from {@code symbol};
allocation-free when none
+ * is present. Package-visible so {@link EmojiAnnotator} keys derived-only
records the same way.
+ *
+ * @param symbol The code point sequence to strip.
+ * @return The sequence without presentation selectors; {@code symbol}'s own
text when it
+ * contains none.
+ */
+ static String stripPresentationSelector(CharSequence symbol) {
+ final int length = symbol.length();
+ int i = 0;
+ while (i < length && symbol.charAt(i) != 0xFE0F) {
+ i++;
+ }
+ if (i == length) {
+ return symbol.toString();
+ }
+ final StringBuilder stripped = new StringBuilder(length - 1);
+ stripped.append(symbol, 0, i);
+ for (int k = i + 1; k < length; k++) {
+ final char c = symbol.charAt(k);
+ if (c != 0xFE0F) {
+ stripped.append(c);
+ }
+ }
+ return stripped.toString();
+ }
+
+ private static Map<String, EmojiAnnotation> load() {
+ try (InputStream in =
EmojiAnnotations.class.getResourceAsStream(RESOURCE)) {
+ if (in == null) {
+ throw new IllegalStateException("Missing emoji annotation data
resource: " + RESOURCE);
+ }
+ return parse(in);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to read emoji annotation data
resource " + RESOURCE, e);
+ }
+ }
+
+ /**
+ * Parses annotation rows of the form {@code codepoints ; attribute ; value
; source ; notes}
+ * with space-separated hexadecimal code points; {@code '#'} starts a
comment line. The notes
+ * column is the fifth and final field, so it may contain {@code ';'}.
Package-private so the
+ * malformed-data handling can be exercised without the bundled resource.
+ *
+ * @param in The stream to read, in UTF-8; consumed and closed by this
method.
+ * @return One immutable record per annotated symbol, keyed by the
selector-stripped symbol.
+ * @throws IOException Thrown if reading fails or a row is malformed.
+ */
+ static Map<String, EmojiAnnotation> parse(InputStream in) throws IOException
{
+ final Map<String, Map<String, EmojiAnnotation.Value>> rows = new
HashMap<>();
+ try (BufferedReader reader =
+ new BufferedReader(new InputStreamReader(in,
StandardCharsets.UTF_8))) {
+ String line;
+ int lineNumber = 0;
+ while ((line = reader.readLine()) != null) {
+ lineNumber++;
+ final String content = line.strip();
+ if (content.isEmpty() || content.startsWith(COMMENT_PREFIX)) {
+ continue;
+ }
+ // Bounded split: only the first four separators are structural.
+ final String[] fields = content.split(FIELD_SEPARATOR, 5);
+ if (fields.length != 5) {
+ throw new IllegalArgumentException("Malformed emoji annotation data
in " + RESOURCE
+ + " at line " + lineNumber + ": expected 5 fields, got " +
fields.length
+ + " in: " + content);
+ }
+ final String symbol = decode(fields[0], lineNumber, content);
+ final String attribute = fields[1].strip();
+ final String value = fields[2].strip();
+ final String source = fields[3].strip();
+ final String notes = fields[4].strip();
+ validate(attribute, value, source, lineNumber, content);
+ final Map<String, EmojiAnnotation.Value> record =
+ rows.computeIfAbsent(symbol, k -> new HashMap<>());
+ if (record.putIfAbsent(attribute, new EmojiAnnotation.Value(value,
source, notes)) != null) {
+ throw new IllegalArgumentException("Malformed emoji annotation data
in " + RESOURCE
+ + " at line " + lineNumber + ": duplicate attribute '" +
attribute
+ + "' in: " + content);
+ }
+ }
+ }
+ final Map<String, EmojiAnnotation> records = new HashMap<>(rows.size());
+ for (final Map.Entry<String, Map<String, EmojiAnnotation.Value>> entry :
rows.entrySet()) {
+ records.put(entry.getKey(), new EmojiAnnotation(entry.getKey(),
entry.getValue()));
+ }
+ return Map.copyOf(records);
+ }
+
+ // Fails loud on an unknown attribute and on a value the attribute's type
does not admit, so a
+ // corrupted or drifted data file cannot load quietly.
+ private static void validate(String attribute, String value, String source,
+ int lineNumber, String content) {
+ if (value.isEmpty()) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in "
+ RESOURCE
+ + " at line " + lineNumber + ": empty value in: " + content);
+ }
+ if (source.isEmpty()) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in "
+ RESOURCE
+ + " at line " + lineNumber + ": empty source in: " + content);
+ }
+ switch (attribute) {
+ case EmojiAnnotation.NAME:
+ break;
+ case EmojiAnnotation.SENTIMENT:
+ final int score;
+ try {
+ score = Integer.parseInt(value);
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("Malformed emoji annotation data
in " + RESOURCE
+ + " at line " + lineNumber + ": sentiment value '" + value
+ + "' is not an integer in: " + content, e);
+ }
+ if (score < -2 || score > 2) {
+ throw new IllegalArgumentException("Malformed emoji annotation data
in " + RESOURCE
+ + " at line " + lineNumber + ": sentiment value " + score
+ + " is outside -2..2 in: " + content);
+ }
+ break;
+ case EmojiAnnotation.ENTITY_TYPE:
+ try {
+ EmojiEntityType.valueOf(value);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Malformed emoji annotation data
in " + RESOURCE
+ + " at line " + lineNumber + ": unrecognized entityType value '"
+ value
+ + "' in: " + content, e);
+ }
+ break;
+ case EmojiAnnotation.CATEGORY:
+ try {
+ EmojiCategory.valueOf(value);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Malformed emoji annotation data
in " + RESOURCE
+ + " at line " + lineNumber + ": unrecognized category value '" +
value
+ + "' in: " + content, e);
+ }
+ break;
+ default:
+ throw new IllegalArgumentException("Malformed emoji annotation data in
" + RESOURCE
+ + " at line " + lineNumber + ": unknown attribute '" + attribute +
"' in: " + content);
+ }
+ }
+
+ private static String decode(String hexCodePoints, int lineNumber, String
content) {
+ final String stripped = hexCodePoints.strip();
+ if (stripped.isEmpty()) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in "
+ RESOURCE
+ + " at line " + lineNumber + ": empty code point sequence in: " +
content);
+ }
+ try {
+ final StringBuilder decoded = new StringBuilder();
+ for (final String hex : stripped.split(MAPPING_CODE_POINT_SEPARATOR)) {
+ decoded.appendCodePoint(Integer.parseInt(hex, 16));
+ }
+ return decoded.toString();
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException("Malformed emoji annotation data in "
+ RESOURCE
+ + " at line " + lineNumber + ": " + content, e);
+ }
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java
new file mode 100644
index 000000000..cf7c590a3
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Assembles one {@link EmojiAnnotation} per symbol from bundled facts
+ * ({@link EmojiAnnotations}), derived facts ({@link EmojiFlags}), and
optional joined facts
+ * ({@link EmojiAnnotationJoin}). {@link #annotate(Term)} keys on {@link
Term#original()};
+ * non-emoji tokens and degenerate flag-shaped text return empty. Instances
are immutable and
+ * thread-safe when the join is.
+ *
+ * <p>Annotations are per-symbol metadata, not text transforms; a parallel
surface beside
+ * {@link Term} rather than {@link Dimension} constants. See OPENNLP-1870.</p>
+ */
+public final class EmojiAnnotator {
+
+ /** Feature-name prefix for the coarse sentiment score, for example {@code
emojiSentiment=2}. */
+ private static final String FEATURE_SENTIMENT_PREFIX = "emojiSentiment=";
+
+ /** Feature-name prefix for the coarse entity type, for example {@code
emojiType=HEART}. */
+ private static final String FEATURE_TYPE_PREFIX = "emojiType=";
+
+ /**
+ * Feature-name prefix for the document-category hint, for example
+ * {@code emojiCategory=SMILEYS_AND_EMOTION}.
+ */
+ private static final String FEATURE_CATEGORY_PREFIX = "emojiCategory=";
+
+ /** Feature-name prefix for a flag's ISO 3166 region, for example {@code
emojiRegion=DE}. */
+ private static final String FEATURE_REGION_PREFIX = "emojiRegion=";
+
+ // Provenance tags of the derived facts; the mechanisms are defined by UTS
#51.
+ private static final String FLAG_SEQUENCE = "UTS51:flag-sequence";
+ private static final String TAG_SEQUENCE = "UTS51:tag-sequence";
+ private static final String DERIVED_NOTE = "decoded from the code point
sequence";
+
+ // The first non-ASCII code unit; every annotatable symbol starts at or
beyond it.
+ private static final char FIRST_NON_ASCII = 0x80;
+
+ private final EmojiAnnotationJoin join;
+
+ /**
+ * Creates an annotator over the bundled and derived layers only.
+ */
+ public EmojiAnnotator() {
+ this.join = null;
+ }
+
+ /**
+ * Creates an annotator that also resolves joined facts through {@code join}.
+ *
+ * @param join The joined-facts hook, called while a record is assembled.
Must not be
+ * {@code null}.
+ * @throws IllegalArgumentException if {@code join} is {@code null}.
+ */
+ public EmojiAnnotator(EmojiAnnotationJoin join) {
+ if (join == null) {
+ throw new IllegalArgumentException("Join must not be null");
+ }
+ this.join = join;
+ }
+
+ /**
+ * Annotates one term, keyed on its {@link Term#original() original} text
(see the class note on
+ * why the original layer is the one annotations describe).
+ *
+ * @param term The term to annotate. Must not be {@code null}.
+ * @return The assembled record, or empty when the term is not an annotated
symbol.
+ * @throws IllegalArgumentException if {@code term} is {@code null}.
+ * @throws IllegalStateException if the configured join violates its
contract (returns
+ * {@code null} or a key colliding with an existing attribute).
+ */
+ public Optional<EmojiAnnotation> annotate(Term term) {
+ if (term == null) {
+ throw new IllegalArgumentException("Term must not be null");
+ }
+ return annotate(term.original());
+ }
+
+ /**
+ * Annotates one symbol.
+ *
+ * @param symbol The code point sequence of one symbol (one token). U+FE0F
presentation
+ * selectors are ignored. Must not be {@code null}.
+ * @return The assembled record, or empty when {@code symbol} is not an
annotated symbol.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalStateException if the configured join violates its
contract (returns
+ * {@code null} or a key colliding with an existing attribute).
+ */
+ public Optional<EmojiAnnotation> annotate(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ if (!isAnnotatableToken(symbol)) {
+ return Optional.empty();
+ }
+ final EmojiAnnotation bundled =
EmojiAnnotations.lookup(symbol).orElse(null);
+ // The lenient decode, not the strict one: degenerate flag-shaped tokens
in real-world text
+ // must yield no region, not an exception. Decoded once on this per-token
hot path.
+ final String isoRegion = EmojiFlags.isoRegionOrNull(symbol);
+ if (bundled == null && isoRegion == null) {
+ return Optional.empty(); // nothing bundled, nothing derived: the join
never creates records
+ }
+ final String recordSymbol = bundled != null ? bundled.symbol()
+ : EmojiAnnotations.stripPresentationSelector(symbol);
+ final Map<String, EmojiAnnotation.Value> attributes = new
LinkedHashMap<>();
+ if (bundled != null) {
+ attributes.putAll(bundled.attributes());
+ }
+ if (isoRegion != null) {
+ final String source = isoRegion.indexOf('-') >= 0 ? TAG_SEQUENCE :
FLAG_SEQUENCE;
+ attributes.put(EmojiAnnotation.ISO_REGION,
+ new EmojiAnnotation.Value(isoRegion, source, DERIVED_NOTE));
+ // A flag is a FLAG in the FLAGS group by the sequence mechanics alone;
bundled rows, if any
+ // ever exist for a flag, would win.
+ attributes.putIfAbsent(EmojiAnnotation.ENTITY_TYPE,
+ new EmojiAnnotation.Value(EmojiEntityType.FLAG.name(), source,
DERIVED_NOTE));
+ attributes.putIfAbsent(EmojiAnnotation.CATEGORY,
+ new EmojiAnnotation.Value(EmojiCategory.FLAGS.name(), source,
DERIVED_NOTE));
+ }
+ if (join != null) {
+ final Map<String, EmojiAnnotation.Value> joined =
join.join(recordSymbol, isoRegion);
+ if (joined == null) {
+ throw new IllegalStateException(
+ "The join returned null for symbol '" + recordSymbol + "'; return
an empty map");
+ }
+ for (final Map.Entry<String, EmojiAnnotation.Value> entry :
joined.entrySet()) {
+ if (attributes.putIfAbsent(entry.getKey(), entry.getValue()) != null) {
+ throw new IllegalStateException("The join returned attribute '" +
entry.getKey()
+ + "' for symbol '" + recordSymbol + "', which collides with an
existing attribute");
+ }
+ }
+ }
+ // A pure bundled hit needs no new record; return the cached one.
+ if (bundled != null && attributes.size() == bundled.attributes().size()) {
+ return Optional.of(bundled);
+ }
+ return Optional.of(new EmojiAnnotation(recordSymbol, attributes));
+ }
+
+ /**
+ * Appends this annotator's features for {@code token} to {@code features}:
one entry per present
+ * annotation attribute (sentiment, entity type, category, region), each
already name-prefixed. A
+ * token that carries no emoji annotation contributes nothing, so callers
may pass every token.
+ *
+ * @param token The token to describe. Must not be {@code null}.
+ * @param features The collection to append feature strings to. Must not be
{@code null}.
+ * @throws IllegalArgumentException if {@code token} or {@code features} is
{@code null}.
+ * @throws IllegalStateException if the configured join violates its
contract (see
+ * {@link #annotate(CharSequence)}).
+ */
+ public void collectFeatures(CharSequence token, Collection<String> features)
{
+ if (token == null) {
+ throw new IllegalArgumentException("Token must not be null");
+ }
+ if (features == null) {
+ throw new IllegalArgumentException("Features must not be null");
+ }
+ final EmojiAnnotation annotation = annotate(token).orElse(null);
+ if (annotation == null) {
+ return;
+ }
+ annotation.sentiment().ifPresent(score ->
features.add(FEATURE_SENTIMENT_PREFIX + score));
+ annotation.entityType().ifPresent(type -> features.add(FEATURE_TYPE_PREFIX
+ type));
+ annotation.category().ifPresent(category ->
features.add(FEATURE_CATEGORY_PREFIX + category));
+ annotation.isoRegion().ifPresent(region ->
features.add(FEATURE_REGION_PREFIX + region));
+ }
+
+ /**
+ * {@return whether {@code token} could carry an emoji annotation and is
worth looking up} Every
+ * annotatable symbol (a pictograph, a regional indicator, or the waving
black flag) starts beyond
+ * ASCII, audited in {@code EmojiAnnotationsTest}; this fast-paths empty and
ordinary tokens
+ * before any lookup or decode runs.
+ */
+ private static boolean isAnnotatableToken(CharSequence token) {
+ return token.length() != 0 && token.charAt(0) >= FIRST_NON_ASCII;
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java
new file mode 100644
index 000000000..cfedf4018
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiCategory.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+/**
+ * The document-category hint of an annotated emoji, one constant per group
header of the upstream
+ * Unicode {@code emoji-test.txt} (Emoji Keyboard/Display Test Data for
+ * <a href="https://www.unicode.org/reports/tr51/";>UTS #51</a>). The
group level of that file
+ * is already a document-category-shaped taxonomy, so this enum mirrors it one
to one rather than
+ * inventing a competing one; the constant a bundled record carries is
provenance-tagged
+ * {@code UCD:emoji-test} in {@code emoji-annotations.txt}.
+ *
+ * <p>All upstream groups are represented, including those the bundled data
does not populate yet,
+ * so growing the data never needs an enum change. See {@link EmojiEntityType}
for the finer,
+ * subgroup-derived projection of the same upstream table.</p>
+ */
+public enum EmojiCategory {
+
+ /** Upstream group {@code Smileys & Emotion}: faces, hearts, and other
emotion symbols. */
+ SMILEYS_AND_EMOTION,
+
+ /** Upstream group {@code People & Body}: people, body parts, hand gestures,
roles. */
+ PEOPLE_AND_BODY,
+
+ /** Upstream group {@code Component}: skin tone and hair components, not
free-standing symbols. */
+ COMPONENT,
+
+ /** Upstream group {@code Animals & Nature}: animals, plants, and nature
symbols. */
+ ANIMALS_AND_NATURE,
+
+ /** Upstream group {@code Food & Drink}: food, drink, and dishware. */
+ FOOD_AND_DRINK,
+
+ /** Upstream group {@code Travel & Places}: places, buildings, transport,
sky and weather. */
+ TRAVEL_AND_PLACES,
+
+ /** Upstream group {@code Activities}: events, sports, games, and arts. */
+ ACTIVITIES,
+
+ /** Upstream group {@code Objects}: clothing, tools, instruments, and other
objects. */
+ OBJECTS,
+
+ /** Upstream group {@code Symbols}: signs, arrows, punctuation-like and
abstract symbols. */
+ SYMBOLS,
+
+ /** Upstream group {@code Flags}: flag emoji; assigned by the derived layer,
not by bundled rows. */
+ FLAGS
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java
new file mode 100644
index 000000000..5c43c7840
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEntityType.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+/**
+ * The coarse entity type of an annotated emoji, a small project-authored
taxonomy assigned from
+ * the subgroup headers of the upstream Unicode {@code emoji-test.txt} (Emoji
Keyboard/Display Test
+ * Data for <a href="https://www.unicode.org/reports/tr51/";>UTS #51</a>):
for example the
+ * {@code face-smiling} and {@code face-concerned} subgroups map to {@link
#FACE} and
+ * {@code transport-ground} maps to {@link #VEHICLE}. The subgroup a bundled
record was derived
+ * from is cited in the notes column of {@code emoji-annotations.txt},
provenance-tagged
+ * {@code UCD:emoji-test}.
+ *
+ * <p>The set is deliberately coarse and grows with the bundled data; a new
constant is a
+ * compatible addition. {@link #FLAG} is assigned by the derived layer
(regional-indicator and tag
+ * sequence decoding), never by a bundled row, so flag coverage does not
depend on data
+ * rows.</p>
+ *
+ * @see EmojiCategory
+ */
+public enum EmojiEntityType {
+
+ /** A face pictograph, from the {@code face-*} subgroups. */
+ FACE,
+
+ /** A heart pictograph, from the {@code heart} subgroup. */
+ HEART,
+
+ /** An animal, from the {@code animal-*} subgroups. */
+ ANIMAL,
+
+ /** Food, from the {@code food-*} subgroups. */
+ FOOD,
+
+ /** A vehicle, from the {@code transport-*} subgroups. */
+ VEHICLE,
+
+ /** A landmark or building, from the {@code place-*} subgroups. */
+ LANDMARK,
+
+ /** A flag; assigned by the derived layer from the code point sequence
itself. */
+ FLAG
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiFlags.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiFlags.java
new file mode 100644
index 000000000..10ef91cb7
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiFlags.java
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.Optional;
+
+/**
+ * The derived-facts layer of the emoji annotation record store: decodes flag
emoji to ISO 3166
+ * codes purely from the code point sequence, with no data file. An emoji flag
sequence is two
+ * regional indicator symbols encoding an ISO 3166-1 alpha-2 code (U+1F1E9
U+1F1EA decodes to
+ * {@code DE}), and an emoji tag sequence over U+1F3F4 WAVING BLACK FLAG
encodes an ISO 3166-2
+ * subdivision code in tag characters (the England flag decodes to {@code
GB-ENG}); both mechanisms
+ * are defined by <a
href="https://www.unicode.org/reports/tr51/";>UTS #51</a>.
+ *
+ * <p>Decoding is mechanical and deliberately does not check
<em>assignment</em>: U+1F1FD U+1F1FD
+ * decodes to {@code XX} even though no such region is assigned, because
validity against the
+ * region registry is a join-time question for whatever gazetteer the user has
installed (see the
+ * joined-facts layer), not a bundled fact. This is also why no per-flag rows
exist in
+ * {@code emoji-annotations.txt}: the region code is fully determined by the
sequence itself.</p>
+ *
+ * <p>{@link #isoRegion(CharSequence)} is the strict decoder: it fails loud on
a sequence that is
+ * flag-shaped but malformed, such as a lone regional indicator or an
unterminated tag sequence.
+ * {@link #isFlag(CharSequence)} is the total predicate for bulk callers that
must never throw on
+ * degenerate real-world text. The expected input is one symbol, for example
one
+ * {@link Term#original()} token; the UAX #29 word tokenizer already
segments a run of
+ * adjacent flags into regional indicator pairs.</p>
+ */
+public final class EmojiFlags {
+
+ private static final int FIRST_REGIONAL_INDICATOR = 0x1F1E6; // REGIONAL
INDICATOR SYMBOL LETTER A
+ private static final int LAST_REGIONAL_INDICATOR = 0x1F1FF; // REGIONAL
INDICATOR SYMBOL LETTER Z
+ private static final int WAVING_BLACK_FLAG = 0x1F3F4;
+ private static final int CANCEL_TAG = 0xE007F;
+ // A tag character maps to the ASCII character at (code point - TAG_OFFSET).
+ private static final int TAG_OFFSET = 0xE0000;
+
+ private EmojiFlags() {
+ }
+
+ /**
+ * {@return whether {@code codePoint} is one of the 26 regional indicator
symbols}
+ *
+ * @param codePoint The code point to test.
+ */
+ private static boolean isRegionalIndicator(int codePoint) {
+ return codePoint >= FIRST_REGIONAL_INDICATOR && codePoint <=
LAST_REGIONAL_INDICATOR;
+ }
+
+ /**
+ * {@return whether {@code symbol} is exactly one well-formed flag emoji}
True only for a
+ * regional indicator pair or a terminated subdivision tag sequence; unlike
+ * {@link #isoRegion(CharSequence)} this never throws on malformed
sequences, so bulk callers
+ * (per-token annotation of arbitrary text) can probe safely.
+ *
+ * @param symbol The code point sequence of one symbol. Must not be {@code
null}.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ */
+ public static boolean isFlag(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ return decode(symbol, true) != null;
+ }
+
+ /**
+ * Decodes one flag emoji to its ISO 3166 code: a regional indicator pair to
the ISO 3166-1
+ * alpha-2 code ({@code DE}) and a subdivision tag sequence to the ISO
3166-2 code
+ * ({@code GB-ENG}).
+ *
+ * @param symbol The code point sequence of one symbol. Must not be {@code
null}.
+ * @return The ISO 3166 code, or empty when {@code symbol} is not
flag-shaped at all (no leading
+ * regional indicator and no tag sequence over U+1F3F4; a lone U+1F3F4
and the ZWJ pirate
+ * flag are not region flags).
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}, or if
it is flag-shaped
+ * but malformed: a lone regional indicator, more or fewer than exactly
two regional
+ * indicators, or a tag sequence that is unterminated, too short for a
subdivision code, not
+ * letter-led, or followed by trailing content.
+ */
+ public static Optional<String> isoRegion(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ return Optional.ofNullable(decode(symbol, false));
+ }
+
+ /**
+ * Leniently decodes one flag emoji, the total decode for bulk per-token
callers: it returns the
+ * ISO 3166 code, or {@code null} when {@code symbol} is not a well-formed
flag (whether not
+ * flag-shaped at all or flag-shaped but malformed), never throwing on
degenerate real-world text.
+ *
+ * @param symbol The code point sequence of one symbol. Must not be {@code
null}.
+ * @return The ISO 3166 code, or {@code null} when {@code symbol} is not a
well-formed flag.
+ * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ */
+ static String isoRegionOrNull(CharSequence symbol) {
+ if (symbol == null) {
+ throw new IllegalArgumentException("Symbol must not be null");
+ }
+ return decode(symbol, true);
+ }
+
+ // Returns the ISO code, or null when symbol is not flag-shaped. A
flag-shaped but malformed
+ // sequence returns null when lenient, otherwise fails loud.
+ private static String decode(CharSequence symbol, boolean lenient) {
+ if (symbol.isEmpty()) {
+ return null;
+ }
+ final int first = Character.codePointAt(symbol, 0);
+ if (isRegionalIndicator(first)) {
+ return decodeRegionalIndicators(symbol, first, lenient);
+ }
+ if (first == WAVING_BLACK_FLAG && symbol.length() > 2
+ && isTagBlock(Character.codePointAt(symbol, 2))) {
+ return decodeTagSequence(symbol, lenient);
+ }
+ return null;
+ }
+
+ // An emoji flag sequence is exactly two regional indicators; anything else
that starts with one
+ // (a lone indicator, an odd run, adjacent flags passed as one symbol) is
malformed.
+ private static String decodeRegionalIndicators(CharSequence symbol, int
first, boolean lenient) {
+ final int firstCount = Character.charCount(first);
+ if (symbol.length() == firstCount) {
+ return malformed(lenient, "Malformed regional indicator sequence: a flag
is a regional"
+ + " indicator pair, got a lone indicator in: " + symbol);
+ }
+ final int second = Character.codePointAt(symbol, firstCount);
+ if (!isRegionalIndicator(second)) {
+ return malformed(lenient, "Malformed regional indicator sequence: a flag
is a regional"
+ + " indicator pair, got a non-indicator after the first indicator
in: " + symbol);
+ }
+ if (symbol.length() > firstCount + Character.charCount(second)) {
+ return malformed(lenient, "Malformed regional indicator sequence:
expected exactly one"
+ + " regional indicator pair in: " + symbol);
+ }
+ return new String(new char[] {
+ (char) ('A' + first - FIRST_REGIONAL_INDICATOR),
+ (char) ('A' + second - FIRST_REGIONAL_INDICATOR)});
+ }
+
+ // An emoji tag sequence: U+1F3F4, tag letters/digits spelling the ISO
3166-2 code (lowercase,
+ // no separator, for example gbeng), then CANCEL TAG. The caller verified
the first tag
+ // character, so the sequence is tag-shaped and any violation below is
malformed.
+ private static String decodeTagSequence(CharSequence symbol, boolean
lenient) {
+ final StringBuilder decoded = new StringBuilder();
+ int i = 2; // past U+1F3F4
+ boolean terminated = false;
+ while (i < symbol.length()) {
+ final int codePoint = Character.codePointAt(symbol, i);
+ i += Character.charCount(codePoint);
+ if (codePoint == CANCEL_TAG) {
+ terminated = true;
+ break;
+ }
+ if (!isTagCharacter(codePoint)) {
+ return malformed(lenient, "Malformed emoji tag sequence: expected a
tag character or"
+ + " CANCEL TAG, got U+" +
Integer.toHexString(codePoint).toUpperCase() + " in: "
+ + symbol);
+ }
+ decoded.append((char) (codePoint - TAG_OFFSET));
+ }
+ if (!terminated) {
+ return malformed(lenient,
+ "Malformed emoji tag sequence: missing the CANCEL TAG terminator in:
" + symbol);
+ }
+ if (i < symbol.length()) {
+ return malformed(lenient,
+ "Malformed emoji tag sequence: content after the CANCEL TAG
terminator in: " + symbol);
+ }
+ if (decoded.length() < 3) {
+ return malformed(lenient, "Malformed emoji tag sequence: an ISO 3166-2
code is a two-letter"
+ + " region and a subdivision suffix, got '" + decoded + "' in: " +
symbol);
+ }
+ if (!isTagLetter(decoded.charAt(0)) || !isTagLetter(decoded.charAt(1))) {
+ return malformed(lenient, "Malformed emoji tag sequence: the region part
of '" + decoded
+ + "' is not two letters in: " + symbol);
+ }
+ // ISO 3166-2 format: uppercase region, hyphen-minus, uppercase
subdivision suffix.
+ final StringBuilder iso = new StringBuilder(decoded.length() + 1);
+ for (int k = 0; k < decoded.length(); k++) {
+ if (k == 2) {
+ iso.append('-');
+ }
+ final char c = decoded.charAt(k);
+ iso.append(isTagLetter(c) ? (char) (c - ('a' - 'A')) : c);
+ }
+ return iso.toString();
+ }
+
+ // Any character of the TAG block; a U+1F3F4 followed by one of these is a
tag sequence attempt,
+ // so its violations are malformed rather than "not a flag" (unlike, say,
the ZWJ pirate flag).
+ private static boolean isTagBlock(int codePoint) {
+ return codePoint >= TAG_OFFSET && codePoint <= CANCEL_TAG;
+ }
+
+ // Tag characters restricted to what an ISO 3166-2 code can contain: tag
digits and tag small
+ // letters. (The full TAG block also carries punctuation tags; those are not
valid here.)
+ private static boolean isTagCharacter(int codePoint) {
+ final int ascii = codePoint - TAG_OFFSET;
+ return (ascii >= '0' && ascii <= '9') || (ascii >= 'a' && ascii <= 'z');
+ }
+
+ private static boolean isTagLetter(char decoded) {
+ return decoded >= 'a' && decoded <= 'z';
+ }
+
+ private static String malformed(boolean lenient, String message) {
+ if (lenient) {
+ return null;
+ }
+ throw new IllegalArgumentException(message);
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
new file mode 100644
index 000000000..ea416c164
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
@@ -0,0 +1,212 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Bundled emoji annotation facts, authored by this project and loaded by
EmojiAnnotations. One row
+# per attribute of one pictograph. Row format:
+#
+# codepoints ; attribute ; value ; source ; notes
+#
+# codepoints is the space-separated hexadecimal code point sequence of the
pictograph, keyed
+# without U+FE0F VARIATION SELECTOR-16 (the emoji presentation selector);
lookups strip it.
+#
+# Attributes and their provenance:
+# name The CLDR short name as printed in the upstream emoji-test.txt
(Emoji
+# Keyboard/Display Test Data for UTS #51, version 17.0); source
CLDR:annotation.
+# sentiment A project-authored coarse polarity score, an integer in -2..2;
source UNSPECIFIED,
+# the marker for a project judgment. No third-party sentiment
data set is copied
+# here; in particular the Emoji Sentiment Ranking (CC BY-SA) is
NOT used in any form.
+# entityType A coarse entity type (an EmojiEntityType constant), assigned
by this project from
+# the subgroup header of the upstream emoji-test.txt; source
UCD:emoji-test, the
+# subgroup is cited in the notes.
+# category A document-category hint (an EmojiCategory constant), the
group header of the
+# upstream emoji-test.txt; source UCD:emoji-test, the group is
cited in the notes.
+#
+# Coverage claim (audited by EmojiAnnotationsTest): every pictograph that
emoji-emoticons.txt
+# folds (every EMOJI-direction source, presentation selector stripped) has a
record here, and
+# every record carries all four attributes. Flag emoji have no rows: their
region is derived from
+# the code point sequence (see EmojiFlags), not bundled.
+#
+# Code points, names, and group/subgroup headers checked against
emoji-test.txt version 17.0.
+#
+# A line starting with '#' is a comment; data rows carry no inline comments.
The notes column is
+# the fifth and final field, so it may contain ';'.
+#
+# Smileys & Emotion
+1F600 ; name ; grinning face ; CLDR:annotation ; emoji-test.txt v17.0 short
name
+1F600 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F600 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F600 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F603 ; name ; grinning face with big eyes ; CLDR:annotation ; emoji-test.txt
v17.0 short name
+1F603 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F603 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F603 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F604 ; name ; grinning face with smiling eyes ; CLDR:annotation ;
emoji-test.txt v17.0 short name
+1F604 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F604 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F604 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F601 ; name ; beaming face with smiling eyes ; CLDR:annotation ;
emoji-test.txt v17.0 short name
+1F601 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F601 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F601 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F642 ; name ; slightly smiling face ; CLDR:annotation ; emoji-test.txt v17.0
short name
+1F642 ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F642 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F642 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F609 ; name ; winking face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F609 ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F609 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F609 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F60A ; name ; smiling face with smiling eyes ; CLDR:annotation ;
emoji-test.txt v17.0 short name
+1F60A ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F60A ; entityType ; FACE ; UCD:emoji-test ; subgroup face-smiling
+1F60A ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+263A ; name ; smiling face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+263A ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+263A ; entityType ; FACE ; UCD:emoji-test ; subgroup face-affection
+263A ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F617 ; name ; kissing face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F617 ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F617 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-affection
+1F617 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F618 ; name ; face blowing a kiss ; CLDR:annotation ; emoji-test.txt v17.0
short name
+1F618 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F618 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-affection
+1F618 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F61B ; name ; face with tongue ; CLDR:annotation ; emoji-test.txt v17.0 short
name
+1F61B ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F61B ; entityType ; FACE ; UCD:emoji-test ; subgroup face-tongue
+1F61B ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F61C ; name ; winking face with tongue ; CLDR:annotation ; emoji-test.txt
v17.0 short name
+1F61C ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F61C ; entityType ; FACE ; UCD:emoji-test ; subgroup face-tongue
+1F61C ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F61D ; name ; squinting face with tongue ; CLDR:annotation ; emoji-test.txt
v17.0 short name
+1F61D ; sentiment ; 1 ; UNSPECIFIED ; project judgment
+1F61D ; entityType ; FACE ; UCD:emoji-test ; subgroup face-tongue
+1F61D ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F610 ; name ; neutral face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F610 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F610 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-neutral-skeptical
+1F610 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F611 ; name ; expressionless face ; CLDR:annotation ; emoji-test.txt v17.0
short name
+1F611 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F611 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-neutral-skeptical
+1F611 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F615 ; name ; confused face ; CLDR:annotation ; emoji-test.txt v17.0 short
name
+1F615 ; sentiment ; -1 ; UNSPECIFIED ; project judgment
+1F615 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F615 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F641 ; name ; slightly frowning face ; CLDR:annotation ; emoji-test.txt v17.0
short name
+1F641 ; sentiment ; -1 ; UNSPECIFIED ; project judgment
+1F641 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F641 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+2639 ; name ; frowning face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+2639 ; sentiment ; -1 ; UNSPECIFIED ; project judgment
+2639 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+2639 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F62E ; name ; face with open mouth ; CLDR:annotation ; emoji-test.txt v17.0
short name
+1F62E ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F62E ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F62E ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F632 ; name ; astonished face ; CLDR:annotation ; emoji-test.txt v17.0 short
name
+1F632 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F632 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F632 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F622 ; name ; crying face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F622 ; sentiment ; -2 ; UNSPECIFIED ; project judgment
+1F622 ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F622 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F62D ; name ; loudly crying face ; CLDR:annotation ; emoji-test.txt v17.0
short name
+1F62D ; sentiment ; -2 ; UNSPECIFIED ; project judgment
+1F62D ; entityType ; FACE ; UCD:emoji-test ; subgroup face-concerned
+1F62D ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+2764 ; name ; red heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+2764 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+2764 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+2764 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F9E1 ; name ; orange heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F9E1 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F9E1 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F9E1 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F49B ; name ; yellow heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F49B ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F49B ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F49B ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F49A ; name ; green heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F49A ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F49A ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F49A ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F499 ; name ; blue heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F499 ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F499 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F499 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F49C ; name ; purple heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F49C ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F49C ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F49C ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F90E ; name ; brown heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F90E ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F90E ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F90E ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F5A4 ; name ; black heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F5A4 ; sentiment ; 0 ; UNSPECIFIED ; project judgment, polarity ambiguous in
running text
+1F5A4 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F5A4 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F90D ; name ; white heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F90D ; sentiment ; 2 ; UNSPECIFIED ; project judgment
+1F90D ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F90D ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+1F494 ; name ; broken heart ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F494 ; sentiment ; -2 ; UNSPECIFIED ; project judgment
+1F494 ; entityType ; HEART ; UCD:emoji-test ; subgroup heart
+1F494 ; category ; SMILEYS_AND_EMOTION ; UCD:emoji-test ; group Smileys &
Emotion
+#
+# Animals & Nature
+1F436 ; name ; dog face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F436 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F436 ; entityType ; ANIMAL ; UCD:emoji-test ; subgroup animal-mammal
+1F436 ; category ; ANIMALS_AND_NATURE ; UCD:emoji-test ; group Animals & Nature
+1F431 ; name ; cat face ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F431 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F431 ; entityType ; ANIMAL ; UCD:emoji-test ; subgroup animal-mammal
+1F431 ; category ; ANIMALS_AND_NATURE ; UCD:emoji-test ; group Animals & Nature
+#
+# Food & Drink
+1F355 ; name ; pizza ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F355 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F355 ; entityType ; FOOD ; UCD:emoji-test ; subgroup food-prepared
+1F355 ; category ; FOOD_AND_DRINK ; UCD:emoji-test ; group Food & Drink
+1F34E ; name ; red apple ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F34E ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F34E ; entityType ; FOOD ; UCD:emoji-test ; subgroup food-fruit
+1F34E ; category ; FOOD_AND_DRINK ; UCD:emoji-test ; group Food & Drink
+#
+# Travel & Places
+1F697 ; name ; automobile ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F697 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F697 ; entityType ; VEHICLE ; UCD:emoji-test ; subgroup transport-ground
+1F697 ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places
+2708 ; name ; airplane ; CLDR:annotation ; emoji-test.txt v17.0 short name
+2708 ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+2708 ; entityType ; VEHICLE ; UCD:emoji-test ; subgroup transport-air
+2708 ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places
+1F5FD ; name ; Statue of Liberty ; CLDR:annotation ; emoji-test.txt v17.0
short name
+1F5FD ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F5FD ; entityType ; LANDMARK ; UCD:emoji-test ; subgroup place-building
+1F5FD ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places
+1F5FC ; name ; Tokyo tower ; CLDR:annotation ; emoji-test.txt v17.0 short name
+1F5FC ; sentiment ; 0 ; UNSPECIFIED ; project judgment
+1F5FC ; entityType ; LANDMARK ; UCD:emoji-test ; subgroup place-building
+1F5FC ; category ; TRAVEL_AND_PLACES ; UCD:emoji-test ; group Travel & Places
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/doccat/EmojiFeatureGeneratorTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/doccat/EmojiFeatureGeneratorTest.java
new file mode 100644
index 000000000..f42abd263
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/doccat/EmojiFeatureGeneratorTest.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.doccat;
+
+import java.util.Collection;
+import java.util.List;
+
+import org.junit.jupiter.api.Test;
+
+import static opennlp.tools.util.normalizer.NormalizerTestUtil.cp;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class EmojiFeatureGeneratorTest {
+
+ @Test
+ void annotatedTokensContributeDocumentFeatures() {
+ final FeatureGenerator generator = new EmojiFeatureGenerator();
+ final Collection<String> features = generator.extractFeatures(
+ new String[] {"great", "pizza", cp(0x1F355), cp(0x1F600)}, null);
+ assertEquals(List.of(
+ "emojiSentiment=0", "emojiType=FOOD", "emojiCategory=FOOD_AND_DRINK",
+ "emojiSentiment=2", "emojiType=FACE",
"emojiCategory=SMILEYS_AND_EMOTION"),
+ List.copyOf(features));
+ }
+
+ @Test
+ void flagsContributeTheirRegion() {
+ final FeatureGenerator generator = new EmojiFeatureGenerator();
+ final Collection<String> features = generator.extractFeatures(
+ new String[] {"match", cp(0x1F1EB, 0x1F1F7)}, null);
+ assertTrue(features.contains("emojiRegion=FR"), "Missing region feature
in: " + features);
+ assertTrue(features.contains("emojiType=FLAG"), "Missing type feature in:
" + features);
+ }
+
+ @Test
+ void plainDocumentsContributeNothing() {
+ final FeatureGenerator generator = new EmojiFeatureGenerator();
+ // Includes a lone regional indicator: bulk extraction must survive
damaged text.
+ final Collection<String> features = generator.extractFeatures(
+ new String[] {"no", "emoji", "here", "", cp(0x1F1E9)}, null);
+ assertTrue(features.isEmpty(), "Unexpected features: " + features);
+ }
+
+ @Test
+ void argumentsAreValidated() {
+ final FeatureGenerator generator = new EmojiFeatureGenerator();
+ assertThrows(IllegalArgumentException.class, () ->
generator.extractFeatures(null, null));
+ assertThrows(IllegalArgumentException.class, () -> new
EmojiFeatureGenerator(null));
+ }
+
+ @Test
+ void optInThroughTheDoccatFactorySeam() {
+ // The wiring is the existing DoccatFactory feature generator seam; the
default factory keeps
+ // only BagOfWords, so default behavior is unchanged and this factory is
the opt-in.
+ final DoccatFactory factory = new DoccatFactory(new FeatureGenerator[] {
+ new BagOfWordsFeatureGenerator(), new EmojiFeatureGenerator()});
+ assertEquals(2, factory.getFeatureGenerators().length);
+ final DoccatFactory defaults = new DoccatFactory();
+ assertEquals(1, defaults.getFeatureGenerators().length);
+ assertEquals(BagOfWordsFeatureGenerator.class,
+ defaults.getFeatureGenerators()[0].getClass());
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentiment/EmojiSentimentContextGeneratorTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentiment/EmojiSentimentContextGeneratorTest.java
new file mode 100644
index 000000000..f09629920
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentiment/EmojiSentimentContextGeneratorTest.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.sentiment;
+
+import java.util.List;
+
+import org.junit.jupiter.api.Test;
+
+import static opennlp.tools.util.normalizer.NormalizerTestUtil.cp;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class EmojiSentimentContextGeneratorTest {
+
+ @Test
+ void emojiFeaturesAreAppendedToTheTokenContext() {
+ final String[] tokens = {"so", "sad", cp(0x1F62D)};
+ final String[] context = new
EmojiSentimentContextGenerator().getContext(tokens);
+ final List<String> features = List.of(context);
+ // The default token context is preserved in full...
+ assertEquals(List.of(tokens), features.subList(0, tokens.length));
+ // ...and the pictograph additionally contributes its typed evidence.
+ assertTrue(features.contains("emojiSentiment=-2"), "Missing feature in: "
+ features);
+ assertTrue(features.contains("emojiType=FACE"), "Missing feature in: " +
features);
+ assertTrue(features.contains("emojiCategory=SMILEYS_AND_EMOTION"),
+ "Missing feature in: " + features);
+ }
+
+ @Test
+ void plainTextContextEqualsTheDefaultGenerators() {
+ // Without emoji the opt-in generator produces exactly the default
context, so opting in
+ // cannot disturb what a model learns from ordinary text.
+ final String[] tokens = {"no", "emoji", "here", "", cp(0x1F1E9)};
+ assertArrayEquals(new SentimentContextGenerator().getContext(tokens),
+ new EmojiSentimentContextGenerator().getContext(tokens));
+ }
+
+ @Test
+ void theFactorySeamCreatesTheOptInGenerator() {
+ // Opt-in path: EmojiSentimentFactory is passed to SentimentME.train and
recorded in the
+ // model manifest, so prediction re-creates the same context generator.
The default factory
+ // is unchanged.
+ assertEquals(EmojiSentimentContextGenerator.class,
+ new EmojiSentimentFactory().createContextGenerator().getClass());
+ assertEquals(SentimentContextGenerator.class,
+ new SentimentFactory().createContextGenerator().getClass());
+ }
+ @Test
+ void nullTextFailsLoud() {
+ final EmojiSentimentContextGenerator generator = new
EmojiSentimentContextGenerator();
+ final IllegalArgumentException e =
+ assertThrows(IllegalArgumentException.class, () ->
generator.getContext(null));
+ assertEquals("Text must not be null", e.getMessage());
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorTest.java
new file mode 100644
index 000000000..6cedd8909
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorTest.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util.featuregen;
+
+import java.io.ByteArrayInputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import org.junit.jupiter.api.Test;
+
+import opennlp.tools.util.normalizer.EmojiAnnotation;
+import opennlp.tools.util.normalizer.EmojiAnnotator;
+
+import static opennlp.tools.util.normalizer.NormalizerTestUtil.cp;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class EmojiAnnotationFeatureGeneratorTest {
+
+ @Test
+ void annotatedTokensContributeTypedFeatures() {
+ final AdaptiveFeatureGenerator generator = new
EmojiAnnotationFeatureGenerator();
+ final String[] tokens = {"I", cp(0x2764, 0xFE0F), "Berlin", cp(0x1F1E9,
0x1F1EA)};
+ final List<String> features = new ArrayList<>();
+ generator.createFeatures(features, tokens, 1, null);
+ assertEquals(List.of("emojiSentiment=2", "emojiType=HEART",
+ "emojiCategory=SMILEYS_AND_EMOTION"), features);
+ features.clear();
+ generator.createFeatures(features, tokens, 3, null);
+ // The flag's entity type acts as gazetteer-like evidence for the name
finder, and the region
+ // is decoded from the sequence with no dictionary. There is no sentiment
feature: a record
+ // never fabricates a value it has no source for.
+ assertEquals(List.of("emojiType=FLAG", "emojiCategory=FLAGS",
"emojiRegion=DE"), features);
+ }
+
+ @Test
+ void ordinaryAndDegenerateTokensContributeNothing() {
+ final AdaptiveFeatureGenerator generator = new
EmojiAnnotationFeatureGenerator();
+ final List<String> features = new ArrayList<>();
+ final String[] tokens = {"word", "", ":-)", cp(0x1F1E9), cp(0x1F9F8)};
+ for (int i = 0; i < tokens.length; i++) {
+ generator.createFeatures(features, tokens, i, null);
+ }
+ // Includes the lone regional indicator: bulk feature generation must
survive damaged text.
+ assertTrue(features.isEmpty(), "Unexpected features: " + features);
+ }
+
+ @Test
+ void joinedFactsFlowIntoFeaturesThroughAConfiguredAnnotator() {
+ final EmojiAnnotator annotator = new EmojiAnnotator((symbol, isoRegion) ->
Map.of());
+ final AdaptiveFeatureGenerator generator = new
EmojiAnnotationFeatureGenerator(annotator);
+ final List<String> features = new ArrayList<>();
+ generator.createFeatures(features, new String[] {cp(0x1F642)}, 0, null);
+ assertEquals(List.of("emojiSentiment=1", "emojiType=FACE",
+ "emojiCategory=SMILEYS_AND_EMOTION"), features);
+ assertThrows(IllegalArgumentException.class,
+ () -> new EmojiAnnotationFeatureGenerator(null));
+ }
+
+ @Test
+ void wiredThroughAFeatureGenerationDescriptor() throws Exception {
+ // The opt-in seam: only a descriptor that names the factory activates the
generator; no
+ // default descriptor is touched.
+ final String descriptor = "<featureGenerators name=\"test\">\n"
+ + " <generator
class=\"opennlp.tools.util.featuregen.AggregatedFeatureGeneratorFactory\">\n"
+ + " <generator
class=\"opennlp.tools.util.featuregen.TokenFeatureGeneratorFactory\"/>\n"
+ + " <generator class=\""
+ +
"opennlp.tools.util.featuregen.EmojiAnnotationFeatureGeneratorFactory\"/>\n"
+ + " </generator>\n"
+ + "</featureGenerators>\n";
+ final AdaptiveFeatureGenerator generator = GeneratorFactory.create(
+ new ByteArrayInputStream(descriptor.getBytes(StandardCharsets.UTF_8)),
null);
+ final List<String> features = new ArrayList<>();
+ generator.createFeatures(features, new String[] {cp(0x1F622)}, 0, null);
+ assertTrue(features.contains("w=" + cp(0x1F622)), "Missing token feature
in: " + features);
+ assertTrue(features.contains("emojiSentiment=-2"), "Missing emoji feature
in: " + features);
+ assertTrue(features.contains("emojiType=FACE"), "Missing emoji feature in:
" + features);
+ }
+
+ @Test
+ void isStatelessUnderAdaptiveDataCalls() {
+ final AdaptiveFeatureGenerator generator = new
EmojiAnnotationFeatureGenerator();
+ generator.updateAdaptiveData(new String[] {"a"}, new String[] {"o"});
+ generator.clearAdaptiveData();
+ final List<String> features = new ArrayList<>();
+ generator.createFeatures(features, new String[] {cp(0x1F642)}, 0, null);
+ assertEquals(3, features.size());
+ }
+
+ @Test
+ void featurePrefixesMatchTheAnnotationAttributes() {
+ // Guards the feature vocabulary: a rename here silently orphans trained
models, so the names
+ // are pinned against the annotation attribute keys they project.
+ assertEquals("sentiment", EmojiAnnotation.SENTIMENT);
+ assertEquals("entityType", EmojiAnnotation.ENTITY_TYPE);
+ assertEquals("category", EmojiAnnotation.CATEGORY);
+ assertEquals("isoRegion", EmojiAnnotation.ISO_REGION);
+ }
+ @Test
+ void nullArgumentsFailLoud() {
+ final AdaptiveFeatureGenerator generator = new
EmojiAnnotationFeatureGenerator();
+ final IllegalArgumentException nullFeatures =
assertThrows(IllegalArgumentException.class,
+ () -> generator.createFeatures(null, new String[] {"x"}, 0, null));
+ assertEquals("Features must not be null", nullFeatures.getMessage());
+ final IllegalArgumentException nullTokens =
assertThrows(IllegalArgumentException.class,
+ () -> generator.createFeatures(new ArrayList<>(), null, 0, null));
+ assertEquals("Tokens must not be null", nullTokens.getMessage());
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java
new file mode 100644
index 000000000..4cf2080bf
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotationsTest.java
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import static opennlp.tools.util.normalizer.NormalizerTestUtil.cp;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class EmojiAnnotationsTest {
+
+ private static final String[] ATTRIBUTES = {EmojiAnnotation.NAME,
EmojiAnnotation.SENTIMENT,
+ EmojiAnnotation.ENTITY_TYPE, EmojiAnnotation.CATEGORY};
+
+ private static Map<String, EmojiAnnotation> bundled() throws IOException {
+ try (InputStream in =
EmojiAnnotations.class.getResourceAsStream("emoji-annotations.txt")) {
+ return EmojiAnnotations.parse(in);
+ }
+ }
+
+ private static Map<String, EmojiAnnotation> parse(String data) throws
IOException {
+ return EmojiAnnotations.parse(new
ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8)));
+ }
+
+ // --- audits of the bundled data ---
+
+ @Test
+ void everyBundledRecordCarriesAllFourAttributes() throws IOException {
+ // The coverage claim in the data file header: a record is complete, never
a partial row set,
+ // so a consumer can rely on any of the four dimensions once a symbol is
annotated at all.
+ for (final EmojiAnnotation annotation : bundled().values()) {
+ for (final String attribute : ATTRIBUTES) {
+ assertTrue(annotation.attribute(attribute).isPresent(),
+ "Missing attribute '" + attribute + "' for symbol: " +
annotation.symbol());
+ }
+ }
+ }
+
+ @Test
+ void everyBundledValueCarriesItsPinnedProvenance() throws IOException {
+ // The licensing audit: names come from the CLDR short names, entity type
and category from
+ // the emoji-test.txt group/subgroup headers, and sentiment is ALWAYS a
project judgment
+ // (UNSPECIFIED). A sentiment row with any other source would mean
third-party sentiment data
+ // (for example the CC BY-SA Emoji Sentiment Ranking) leaked into the
bundled file.
+ final Map<String, String> expectedSource = Map.of(
+ EmojiAnnotation.NAME, "CLDR:annotation",
+ EmojiAnnotation.SENTIMENT, "UNSPECIFIED",
+ EmojiAnnotation.ENTITY_TYPE, "UCD:emoji-test",
+ EmojiAnnotation.CATEGORY, "UCD:emoji-test");
+ for (final EmojiAnnotation annotation : bundled().values()) {
+ for (final Map.Entry<String, EmojiAnnotation.Value> entry :
+ annotation.attributes().entrySet()) {
+ final EmojiAnnotation.Value value = entry.getValue();
+ assertEquals(expectedSource.get(entry.getKey()), value.source(),
+ "Unexpected source for " + annotation.symbol() + " " +
entry.getKey());
+ assertFalse(value.notes().isEmpty(),
+ "Empty notes for " + annotation.symbol() + " " + entry.getKey());
+ }
+ }
+ }
+
+ @Test
+ void everyEmojiFoldSourceIsAnnotated() throws IOException {
+ // The cross-file coverage claim: every pictograph emoji-emoticons.txt can
fold has a record
+ // here (presentation selector stripped), so the fold layer and the
annotation layer never
+ // disagree about which symbols the project understands.
+ final Map<String, EmojiAnnotation> annotations = bundled();
+ final EmojiEmoticons.Tables tables;
+ try (InputStream in =
EmojiEmoticons.class.getResourceAsStream("emoji-emoticons.txt")) {
+ tables = EmojiEmoticons.parse(in);
+ }
+ for (final List<EmojiEmoticons.Mapping> candidates :
tables.emojiToEmoticon().table().values()) {
+ for (final EmojiEmoticons.Mapping mapping : candidates) {
+ final String key = mapping.source().replace(cp(0xFE0F), "");
+ assertTrue(annotations.containsKey(key),
+ "Emoji fold source has no annotation record: " + mapping);
+ }
+ }
+ }
+
+ @Test
+ void bundledRecordAndRowCountsAreAudited() throws IOException {
+ // Locks the bundled counts so a data edit trips this test for a conscious
bump, the same
+ // discipline as the emoji-emoticons.txt and CaseFolding.txt audits.
+ final Map<String, EmojiAnnotation> annotations = bundled();
+ assertEquals(40, annotations.size());
+ int rowCount = 0;
+ for (final EmojiAnnotation annotation : annotations.values()) {
+ rowCount += annotation.attributes().size();
+ }
+ assertEquals(160, rowCount);
+ }
+
+ @Test
+ void bundledValuesAreWithinTheirTypedDomains() throws IOException {
+ // The typed accessors must never throw for bundled data: sentiment parses
into -2..2 and the
+ // enum-backed attributes resolve to constants. Also asserts values are
printable ASCII, the
+ // same discipline as the data file itself.
+ for (final EmojiAnnotation annotation : bundled().values()) {
+ final int sentiment = annotation.sentiment().orElseThrow();
+ assertTrue(sentiment >= -2 && sentiment <= 2,
+ "Sentiment outside -2..2 for " + annotation.symbol());
+ annotation.entityType().orElseThrow();
+ annotation.category().orElseThrow();
+ final String name = annotation.name().orElseThrow();
+ name.chars().forEach(c ->
+ assertTrue(c >= 0x20 && c < 0x7F, "Non-ASCII-printable name: " +
name));
+ }
+ }
+
+ @Test
+ void everyBundledSymbolStartsBeyondAscii() throws IOException {
+ // The feature generators fast-path tokens whose first char is ASCII
without touching the
+ // annotation layer; this audit keeps that guard sound. It holds
structurally: annotatable
+ // symbols are pictographs, and flags (regional indicators, the waving
black flag) are
+ // supplementary-plane sequences handled by the derived layer.
+ for (final String symbol : bundled().keySet()) {
+ assertTrue(symbol.charAt(0) > 0x7F, "ASCII-leading annotated symbol: " +
symbol);
+ }
+ }
+
+ // --- lookup behavior ---
+
+ @Test
+ void lookupResolvesATypedRecord() {
+ final EmojiAnnotation annotation =
EmojiAnnotations.lookup(cp(0x1F642)).orElseThrow();
+ assertEquals(cp(0x1F642), annotation.symbol());
+ assertEquals("slightly smiling face", annotation.name().orElseThrow());
+ assertEquals(1, annotation.sentiment().orElseThrow());
+ assertEquals(EmojiEntityType.FACE, annotation.entityType().orElseThrow());
+ assertEquals(EmojiCategory.SMILEYS_AND_EMOTION,
annotation.category().orElseThrow());
+ }
+
+ @Test
+ void lookupStripsThePresentationSelector() {
+ // U+2764 U+FE0F (emoji presentation) and bare U+2764 are the same symbol;
the bundled rows
+ // are keyed without the selector.
+ final EmojiAnnotation withSelector =
+ EmojiAnnotations.lookup(cp(0x2764) + cp(0xFE0F)).orElseThrow();
+ final EmojiAnnotation bare =
EmojiAnnotations.lookup(cp(0x2764)).orElseThrow();
+ assertEquals(bare, withSelector);
+ assertEquals("red heart", withSelector.name().orElseThrow());
+ }
+
+ @Test
+ void lookupOfUnannotatedTextIsEmpty() {
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup("word"));
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup(":-)"));
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup(""));
+ // A bare presentation selector strips to an empty key.
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup(cp(0xFE0F)));
+ // Flags are handled by the derived layer, never by bundled rows.
+ assertEquals(Optional.empty(), EmojiAnnotations.lookup(cp(0x1F1E9) +
cp(0x1F1EA)));
+ }
+
+ @Test
+ void lookupFailsLoudOnNull() {
+ assertThrows(IllegalArgumentException.class, () ->
EmojiAnnotations.lookup(null));
+ }
+
+ // --- fail-loud parsing ---
+
+ static Stream<String> malformedData() {
+ // The data and the code move together, so every malformed row fails loud
rather than loading
+ // quietly: wrong field count, an unknown attribute, malformed hex, an
empty structural field,
+ // a sentiment outside the coarse scale, an unrecognized enum value, and a
duplicate row.
+ return Stream.of(
+ "1F642 ; name ; slightly smiling face ; CLDR:annotation\n",
+ "1F642 ; shortName ; slightly smiling face ; CLDR:annotation ; n\n",
+ "1F64X ; name ; slightly smiling face ; CLDR:annotation ; n\n",
+ " ; name ; slightly smiling face ; CLDR:annotation ; n\n",
+ "1F642 ; name ; ; CLDR:annotation ; n\n",
+ "1F642 ; name ; slightly smiling face ; ; n\n",
+ "1F642 ; sentiment ; 3 ; UNSPECIFIED ; n\n",
+ "1F642 ; sentiment ; -3 ; UNSPECIFIED ; n\n",
+ "1F642 ; sentiment ; positive ; UNSPECIFIED ; n\n",
+ "1F642 ; entityType ; SMILEY ; UCD:emoji-test ; n\n",
+ "1F642 ; category ; EMOTION ; UCD:emoji-test ; n\n",
+ "1F642 ; sentiment ; 1 ; UNSPECIFIED ; first\n"
+ + "1F642 ; sentiment ; 2 ; UNSPECIFIED ; duplicate\n");
+ }
+
+ @ParameterizedTest
+ @MethodSource("malformedData")
+ void parseFailsLoud(String data) {
+ assertThrows(IllegalArgumentException.class, () -> parse(data));
+ }
+
+ @Test
+ void parseSkipsCommentAndBlankLinesAndKeepsSemicolonsInNotes() throws
IOException {
+ final Map<String, EmojiAnnotation> parsed = parse("# comment\n\n \n"
+ + "1F642 ; name ; slightly smiling face ; CLDR:annotation ; note; with
semicolon\n");
+ assertEquals(1, parsed.size());
+ assertEquals("note; with semicolon",
+
parsed.get(cp(0x1F642)).attribute(EmojiAnnotation.NAME).orElseThrow().notes());
+ }
+
+ // --- record validation (records are also user-constructable, for tests and
custom joins) ---
+
+ @Test
+ void recordConstructionFailsLoudOnBadArguments() {
+ final Map<String, EmojiAnnotation.Value> attributes =
Map.of(EmojiAnnotation.NAME,
+ new EmojiAnnotation.Value("slightly smiling face", "CLDR:annotation",
""));
+ assertThrows(IllegalArgumentException.class, () -> new
EmojiAnnotation(null, attributes));
+ assertThrows(IllegalArgumentException.class, () -> new EmojiAnnotation("",
attributes));
+ assertThrows(IllegalArgumentException.class, () -> new
EmojiAnnotation(cp(0x1F642), null));
+ // A Value rejects a null or empty value and a null or empty source, and a
null notes.
+ assertThrows(IllegalArgumentException.class, () -> new
EmojiAnnotation.Value(null, "s", ""));
+ assertThrows(IllegalArgumentException.class, () -> new
EmojiAnnotation.Value("", "s", ""));
+ assertThrows(IllegalArgumentException.class, () -> new
EmojiAnnotation.Value("v", null, ""));
+ assertThrows(IllegalArgumentException.class, () -> new
EmojiAnnotation.Value("v", "", ""));
+ assertThrows(IllegalArgumentException.class, () -> new
EmojiAnnotation.Value("v", "s", null));
+ }
+
+ @Test
+ void typedAccessorsFailLoudOnValuesOutsideTheirDomain() {
+ // Only reachable through hand-built records; the loader validates bundled
data. A quietly
+ // swallowed conversion here would surface as a silently missing feature
downstream.
+ final EmojiAnnotation annotation = new EmojiAnnotation(cp(0x1F642), Map.of(
+ EmojiAnnotation.SENTIMENT, new EmojiAnnotation.Value("happy",
"UNSPECIFIED", ""),
+ EmojiAnnotation.ENTITY_TYPE, new EmojiAnnotation.Value("SMILEY",
"UNSPECIFIED", ""),
+ EmojiAnnotation.CATEGORY, new EmojiAnnotation.Value("EMOTION",
"UNSPECIFIED", "")));
+ assertThrows(IllegalStateException.class, annotation::sentiment);
+ assertThrows(IllegalStateException.class, annotation::entityType);
+ assertThrows(IllegalStateException.class, annotation::category);
+ assertThrows(IllegalArgumentException.class, () ->
annotation.attribute(null));
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorTest.java
new file mode 100644
index 000000000..97e1fcc14
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorTest.java
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import static opennlp.tools.util.normalizer.NormalizerTestUtil.cp;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class EmojiAnnotatorTest {
+
+ private static final String GERMAN_FLAG = cp(0x1F1E9, 0x1F1EA);
+
+ @Test
+ void annotatesATermFromItsOriginalLayer() {
+ // The end-to-end path the surface exists for: UAX #29 segmentation keeps
the pictograph (with
+ // its presentation selector) as one token, the fold dimension rewrites it
to ASCII for
+ // matching, and the annotation still resolves because it reads the
original layer.
+ final TermAnalyzer analyzer = TermAnalyzer.builder().emojiFold().build();
+ final List<Term> terms = analyzer.analyze("I " + cp(0x2764, 0xFE0F) + "
pizza");
+ final Term heart = terms.get(1);
+ assertEquals(cp(0x2764, 0xFE0F), heart.original());
+ assertEquals("<3", heart.normalized()); // the fold layer has rewritten
the pictograph away
+ final EmojiAnnotation annotation = new
EmojiAnnotator().annotate(heart).orElseThrow();
+ assertEquals("red heart", annotation.name().orElseThrow());
+ assertEquals(2, annotation.sentiment().orElseThrow());
+ assertEquals(EmojiEntityType.HEART, annotation.entityType().orElseThrow());
+ assertEquals(EmojiCategory.SMILEYS_AND_EMOTION,
annotation.category().orElseThrow());
+ assertEquals(Optional.empty(), annotation.isoRegion());
+ }
+
+ @Test
+ void aPureBundledHitReturnsTheCachedRecord() {
+ // No join, no derived facts: the annotator must not copy the record per
token.
+ final EmojiAnnotation direct =
EmojiAnnotations.lookup(cp(0x1F642)).orElseThrow();
+ assertSame(direct, new
EmojiAnnotator().annotate(cp(0x1F642)).orElseThrow());
+ }
+
+ @Test
+ void annotatesAFlagTermWithDerivedFacts() {
+ // A flag has no bundled rows; entity type, category, and the region are
all derived from the
+ // sequence itself, provenance-tagged with the UTS #51 mechanism that
defines them.
+ final TermAnalyzer analyzer = TermAnalyzer.builder().build();
+ final List<Term> terms = analyzer.analyze("Berlin " + GERMAN_FLAG);
+ final EmojiAnnotation annotation =
+ new EmojiAnnotator().annotate(terms.get(1)).orElseThrow();
+ assertEquals(GERMAN_FLAG, annotation.symbol());
+ assertEquals("DE", annotation.isoRegion().orElseThrow());
+ assertEquals(EmojiEntityType.FLAG, annotation.entityType().orElseThrow());
+ assertEquals(EmojiCategory.FLAGS, annotation.category().orElseThrow());
+ assertEquals(Optional.empty(), annotation.name());
+ assertEquals("UTS51:flag-sequence",
+
annotation.attribute(EmojiAnnotation.ISO_REGION).orElseThrow().source());
+ }
+
+ @Test
+ void annotatesASubdivisionFlag() {
+ final StringBuilder england = new StringBuilder(cp(0x1F3F4));
+ for (final char c : "gbeng".toCharArray()) {
+ england.appendCodePoint(0xE0000 + c);
+ }
+ england.appendCodePoint(0xE007F);
+ final EmojiAnnotation annotation =
+ new EmojiAnnotator().annotate(england).orElseThrow();
+ assertEquals("GB-ENG", annotation.isoRegion().orElseThrow());
+ assertEquals("UTS51:tag-sequence",
+
annotation.attribute(EmojiAnnotation.ISO_REGION).orElseThrow().source());
+ }
+
+ static Stream<String> nonAnnotatedTokens() {
+ // A lone regional indicator in damaged text yields no annotation, never
an exception; the
+ // fail-loud contract lives on EmojiFlags.isoRegion for direct callers. An
unmapped pictograph
+ // is not annotated either.
+ return Stream.of("word", "", ":-)", cp(0x1F1E9), cp(0x1F9F8));
+ }
+
+ @ParameterizedTest
+ @MethodSource("nonAnnotatedTokens")
+ void annotationIsTotalOverArbitraryTokens(String token) {
+ assertEquals(Optional.empty(), new EmojiAnnotator().annotate(token));
+ }
+
+ @Test
+ void joinedFactsMergeIntoTheRecord() {
+ final List<String> calls = new ArrayList<>();
+ final EmojiAnnotator annotator = new EmojiAnnotator((symbol, isoRegion) ->
{
+ calls.add(symbol + "/" + isoRegion);
+ if ("DE".equals(isoRegion)) {
+ return Map.of("geonamesId",
+ new EmojiAnnotation.Value("2921044", "user:gazetteer", "join
test"));
+ }
+ return Map.of();
+ });
+ final EmojiAnnotation flag = annotator.annotate(GERMAN_FLAG).orElseThrow();
+ assertEquals("2921044",
flag.attribute("geonamesId").orElseThrow().value());
+ assertEquals("user:gazetteer",
flag.attribute("geonamesId").orElseThrow().source());
+ assertEquals("DE", flag.isoRegion().orElseThrow());
+ // A bundled non-flag symbol is joined with a null region.
+ final EmojiAnnotation heart = annotator.annotate(cp(0x2764)).orElseThrow();
+ assertEquals(Optional.empty(), heart.attribute("geonamesId"));
+ assertEquals("red heart", heart.name().orElseThrow());
+ // The join never runs for unannotated tokens: it augments records, it
does not create them.
+ assertEquals(Optional.empty(), annotator.annotate("word"));
+ assertEquals(List.of(GERMAN_FLAG + "/DE", cp(0x2764) + "/null"), calls);
+ }
+
+ @Test
+ void aContractViolatingJoinFailsLoud() {
+ final EmojiAnnotator returnsNull = new EmojiAnnotator((symbol, isoRegion)
-> null);
+ assertThrows(IllegalStateException.class, () ->
returnsNull.annotate(GERMAN_FLAG));
+ // Colliding with a bundled attribute.
+ final EmojiAnnotator collidesBundled = new EmojiAnnotator((symbol,
isoRegion) ->
+ Map.of(EmojiAnnotation.NAME, new EmojiAnnotation.Value("x",
"user:gazetteer", "")));
+ assertThrows(IllegalStateException.class, () ->
collidesBundled.annotate(cp(0x1F642)));
+ // Colliding with a derived attribute.
+ final EmojiAnnotator collidesDerived = new EmojiAnnotator((symbol,
isoRegion) ->
+ Map.of(EmojiAnnotation.ISO_REGION, new EmojiAnnotation.Value("XX",
"user:gazetteer", "")));
+ assertThrows(IllegalStateException.class, () ->
collidesDerived.annotate(GERMAN_FLAG));
+ }
+
+ @Test
+ void argumentsAreValidated() {
+ assertThrows(IllegalArgumentException.class, () -> new
EmojiAnnotator(null));
+ final EmojiAnnotator annotator = new EmojiAnnotator();
+ assertThrows(IllegalArgumentException.class, () ->
annotator.annotate((Term) null));
+ assertThrows(IllegalArgumentException.class, () ->
annotator.annotate((CharSequence) null));
+ }
+
+ @Test
+ void flagTokensSurviveSegmentation() {
+ // Adjacent flags segment into their pairs under UAX #29 (WB15/WB16), so
per-token annotation
+ // sees each flag whole; this pins the assumption the bulk surface relies
on.
+ final TermAnalyzer analyzer = TermAnalyzer.builder().build();
+ final List<Term> terms = analyzer.analyze(GERMAN_FLAG + cp(0x1F1EB,
0x1F1F7));
+ assertEquals(2, terms.size());
+ final EmojiAnnotator annotator = new EmojiAnnotator();
+ assertEquals("DE",
annotator.annotate(terms.get(0)).orElseThrow().isoRegion().orElseThrow());
+ assertEquals("FR",
annotator.annotate(terms.get(1)).orElseThrow().isoRegion().orElseThrow());
+ }
+
+ @Test
+ void mergedFlagRecordsKeepEveryProvenanceTag() {
+ // The record store contract: every value, bundled or derived, carries its
own source.
+ final EmojiAnnotation annotation = new
EmojiAnnotator().annotate(GERMAN_FLAG).orElseThrow();
+ for (final Map.Entry<String, EmojiAnnotation.Value> entry :
+ annotation.attributes().entrySet()) {
+ assertFalse(entry.getValue().source().isEmpty(), "Empty source for " +
entry.getKey());
+ }
+ assertEquals(3, annotation.attributes().size()); // isoRegion, entityType,
category
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorUsageExampleTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorUsageExampleTest.java
new file mode 100644
index 000000000..b9446c824
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiAnnotatorUsageExampleTest.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+import org.junit.jupiter.api.Test;
+
+import static opennlp.tools.util.normalizer.NormalizerTestUtil.cp;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Pins the cookbook path documented in {@code normalizer.xml}: segment with
+ * {@link TermAnalyzer}, annotate each term with {@link EmojiAnnotator}, and
read the
+ * typed fields of the resulting {@link EmojiAnnotation}.
+ */
+public class EmojiAnnotatorUsageExampleTest {
+
+ @Test
+ void testAnnotatesFlagAndHeartFromDocumentedExample() {
+ final EmojiAnnotator annotator = new EmojiAnnotator();
+ final TermAnalyzer analyzer = TermAnalyzer.builder().emojiFold().build();
+
+ // "Berlin <German flag> <red heart>"
+ final String text = "Berlin " + cp(0x1F1E9, 0x1F1EA) + " " + cp(0x2764,
0xFE0F);
+ final List<String> regions = new ArrayList<>();
+ final List<String> names = new ArrayList<>();
+ final List<EmojiEntityType> entityTypes = new ArrayList<>();
+ final List<EmojiCategory> categories = new ArrayList<>();
+ final List<Integer> sentiments = new ArrayList<>();
+ for (final Term term : analyzer.analyze(text)) {
+ annotator.annotate(term).ifPresent(a -> {
+ a.isoRegion().ifPresent(regions::add);
+ a.name().ifPresent(names::add);
+ a.entityType().ifPresent(entityTypes::add);
+ a.category().ifPresent(categories::add);
+ a.sentiment().ifPresent(s -> sentiments.add(s));
+ });
+ }
+
+ // Exactly the values the manual prints, in term order: the flag is
derived, the heart bundled.
+ assertEquals(List.of("DE"), regions);
+ assertEquals(List.of("red heart"), names);
+ assertEquals(List.of(EmojiEntityType.FLAG, EmojiEntityType.HEART),
entityTypes);
+ assertEquals(List.of(EmojiCategory.FLAGS,
EmojiCategory.SMILEYS_AND_EMOTION), categories);
+ assertEquals(List.of(2), sentiments);
+ assertEquals(Optional.empty(), annotator.annotate("Berlin"));
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiFlagsTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiFlagsTest.java
new file mode 100644
index 000000000..644323ac3
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/EmojiFlagsTest.java
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util.normalizer;
+
+import java.util.Optional;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import static opennlp.tools.util.normalizer.NormalizerTestUtil.cp;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class EmojiFlagsTest {
+
+ // Regional indicator symbols A..Z.
+ private static int ri(char letter) {
+ return 0x1F1E6 + (letter - 'A');
+ }
+
+ // Tag characters spelling an ASCII string, without the terminator.
+ private static String tags(String ascii) {
+ final StringBuilder sb = new StringBuilder();
+ for (int i = 0; i < ascii.length(); i++) {
+ sb.appendCodePoint(0xE0000 + ascii.charAt(i));
+ }
+ return sb.toString();
+ }
+
+ private static final String BLACK_FLAG = cp(0x1F3F4);
+ private static final String CANCEL = cp(0xE007F);
+
+ @Test
+ void regionalIndicatorPairsDecodeToIso3166Alpha2() {
+ assertEquals("DE", EmojiFlags.isoRegion(cp(ri('D'),
ri('E'))).orElseThrow());
+ assertEquals("US", EmojiFlags.isoRegion(cp(ri('U'),
ri('S'))).orElseThrow());
+ assertEquals("JP", EmojiFlags.isoRegion(cp(ri('J'),
ri('P'))).orElseThrow());
+ // Exceptionally reserved and macro-region codes decode the same way (UN,
EU).
+ assertEquals("UN", EmojiFlags.isoRegion(cp(ri('U'),
ri('N'))).orElseThrow());
+ assertEquals("EU", EmojiFlags.isoRegion(cp(ri('E'),
ri('U'))).orElseThrow());
+ }
+
+ @Test
+ void decodingDoesNotCheckAssignment() {
+ // Whether XX is an assigned region is a join-time question against the
user's gazetteer, not
+ // a bundled or derived fact; the pair decodes mechanically.
+ assertEquals("XX", EmojiFlags.isoRegion(cp(ri('X'),
ri('X'))).orElseThrow());
+ }
+
+ @Test
+ void subdivisionTagSequencesDecodeToIso3166Dash2() {
+ assertEquals("GB-ENG",
+ EmojiFlags.isoRegion(BLACK_FLAG + tags("gbeng") +
CANCEL).orElseThrow());
+ assertEquals("GB-SCT",
+ EmojiFlags.isoRegion(BLACK_FLAG + tags("gbsct") +
CANCEL).orElseThrow());
+ assertEquals("GB-WLS",
+ EmojiFlags.isoRegion(BLACK_FLAG + tags("gbwls") +
CANCEL).orElseThrow());
+ // Digits are valid in a subdivision suffix (ISO 3166-2 allows
alphanumeric suffixes).
+ assertEquals("FR-75", EmojiFlags.isoRegion(BLACK_FLAG + tags("fr75") +
CANCEL).orElseThrow());
+ }
+
+ @Test
+ void nonFlagsAreEmptyNotErrors() {
+ assertEquals(Optional.empty(), EmojiFlags.isoRegion(""));
+ assertEquals(Optional.empty(), EmojiFlags.isoRegion("word"));
+ assertEquals(Optional.empty(), EmojiFlags.isoRegion(cp(0x1F642)));
+ // A lone waving black flag is an ordinary pictograph, not a region flag.
+ assertEquals(Optional.empty(), EmojiFlags.isoRegion(BLACK_FLAG));
+ // The ZWJ pirate flag is a different mechanism (U+1F3F4 U+200D U+2620
U+FE0F), not a tag
+ // sequence; it is not flag-shaped for region decoding.
+ assertEquals(Optional.empty(),
+ EmojiFlags.isoRegion(cp(0x1F3F4, 0x200D, 0x2620, 0xFE0F)));
+ // Leading ordinary letter: not flag-shaped even if regional indicators
follow.
+ assertEquals(Optional.empty(), EmojiFlags.isoRegion("A" + cp(ri('D'),
ri('E'))));
+ }
+
+ static Stream<String> malformedRegionalIndicatorSequences() {
+ // A lone indicator, an odd run, adjacent flags passed as one symbol, and
an indicator followed
+ // by other content are all malformed for the strict decoder.
+ return Stream.of(
+ cp(ri('D')),
+ cp(ri('D'), ri('E'), ri('F')),
+ cp(ri('D'), ri('E'), ri('F'), ri('R')),
+ cp(ri('D')) + "E",
+ cp(ri('D'), ri('E')) + "!");
+ }
+
+ @ParameterizedTest
+ @MethodSource("malformedRegionalIndicatorSequences")
+ void malformedRegionalIndicatorSequencesFailLoud(String input) {
+ assertThrows(IllegalArgumentException.class, () ->
EmojiFlags.isoRegion(input));
+ }
+
+ static Stream<String> malformedTagSequences() {
+ return Stream.of(
+ BLACK_FLAG + tags("gbeng"),
+ BLACK_FLAG + CANCEL,
+ BLACK_FLAG + tags("gb") + CANCEL,
+ BLACK_FLAG + tags("1beng") + CANCEL,
+ BLACK_FLAG + tags("gb") + "e" + tags("ng") + CANCEL,
+ BLACK_FLAG + tags("gbeng") + CANCEL + "x");
+ }
+
+ @ParameterizedTest
+ @MethodSource("malformedTagSequences")
+ void malformedTagSequencesFailLoud(String input) {
+ assertThrows(IllegalArgumentException.class, () ->
EmojiFlags.isoRegion(input));
+ }
+
+ @Test
+ void isFlagIsTotalOverMalformedInput() {
+ // The bulk predicate never throws on degenerate text (per-token
annotation must survive a
+ // stray lone indicator in real-world input); it is true exactly for
well-formed flags.
+ assertTrue(EmojiFlags.isFlag(cp(ri('D'), ri('E'))));
+ assertTrue(EmojiFlags.isFlag(BLACK_FLAG + tags("gbeng") + CANCEL));
+ assertFalse(EmojiFlags.isFlag(cp(ri('D'))));
+ assertFalse(EmojiFlags.isFlag(cp(ri('D'), ri('E'), ri('F'))));
+ assertFalse(EmojiFlags.isFlag(BLACK_FLAG + tags("gbeng")));
+ assertFalse(EmojiFlags.isFlag(BLACK_FLAG));
+ assertFalse(EmojiFlags.isFlag("word"));
+ assertFalse(EmojiFlags.isFlag(""));
+ }
+
+ @Test
+ void nullFailsLoud() {
+ assertThrows(IllegalArgumentException.class, () ->
EmojiFlags.isoRegion(null));
+ assertThrows(IllegalArgumentException.class, () ->
EmojiFlags.isFlag(null));
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/NormalizerTestUtil.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/NormalizerTestUtil.java
new file mode 100644
index 000000000..d14df0c11
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/NormalizerTestUtil.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util.normalizer;
+
+/**
+ * Test helpers shared across the normalizer test package.
+ */
+public final class NormalizerTestUtil {
+
+ private NormalizerTestUtil() {
+ }
+
+ /**
+ * Builds a string from the given Unicode code points, so
supplementary-plane symbols can
+ * be written by code point instead of as surrogate-pair literals.
+ *
+ * @param codePoints The code points to append, in order.
+ * @return The string holding those code points. Never {@code null}.
+ */
+ public static String cp(int... codePoints) {
+ final StringBuilder sb = new StringBuilder();
+ for (final int codePoint : codePoints) {
+ sb.appendCodePoint(codePoint);
+ }
+ return sb.toString();
+ }
+}
diff --git a/opennlp-distr/src/main/readme/LICENSE
b/opennlp-distr/src/main/readme/LICENSE
index 473434955..478cb9b9e 100644
--- a/opennlp-distr/src/main/readme/LICENSE
+++ b/opennlp-distr/src/main/readme/LICENSE
@@ -615,7 +615,10 @@ The following license applies to the Unicode data files
bundled inside the
opennlp-runtime jar (opennlp/tools/util/normalizer/confusables.txt,
opennlp/tools/util/normalizer/CaseFolding.txt,
opennlp/tools/tokenize/uax29/WordBreakProperty.txt, and
-opennlp/tools/tokenize/uax29/ExtendedPictographic.txt):
+opennlp/tools/tokenize/uax29/ExtendedPictographic.txt), and to the
+Unicode-derived values (the CLDR short names and the emoji-test.txt
+group/subgroup classifications) in the otherwise project-authored
+opennlp/tools/util/normalizer/emoji-annotations.txt:
UNICODE LICENSE V3
diff --git a/opennlp-docs/src/docbkx/normalizer.xml
b/opennlp-docs/src/docbkx/normalizer.xml
index 7adb613f5..d955e0366 100644
--- a/opennlp-docs/src/docbkx/normalizer.xml
+++ b/opennlp-docs/src/docbkx/normalizer.xml
@@ -62,10 +62,12 @@
<para>
Two engines underpin everything: the
<code>CharSequenceNormalizer</code> family offers
ready-made, composable normalizers, and the
<code>CharClass</code> engine is the low-level,
- configurable building block they are made of. Built on
these are three higher-level
+ configurable building block they are made of. Built on
these are four higher-level
features documented below: a layered term model that
projects a token through a
configurable stack of transforms while keeping every
intermediate form (see
- <xref linkend="tools.normalizer.term"/>), per-language
profiles that select the transforms
+ <xref linkend="tools.normalizer.term"/>), an opt-in
emoji annotation layer that turns
+ pictographs into typed signal (see <xref
linkend="tools.normalizer.emojiannotations"/>),
+ per-language profiles that select the transforms
appropriate to a language (see <xref
linkend="tools.normalizer.language"/>), and confusable
folding that reduces lookalike characters for matching
(see
<xref linkend="tools.normalizer.confusables"/>).
@@ -521,6 +523,67 @@ Term term = analyzer.analyze("Running").get(0);
</para>
</section>
+ <section xml:id="tools.normalizer.emojiannotations">
+ <title>Emoji annotations</title>
+ <para>
+ Beyond folding, a pictograph carries signal worth
keeping: a name, a coarse sentiment, an
+ entity type, a document category, and for flags a
region. <code>EmojiAnnotator</code> is the
+ opt-in surface for that signal. It assembles one
<code>EmojiAnnotation</code> record per
+ symbol from three strictly separated layers:
<emphasis>bundled facts</emphasis> from the
+ provenance-tagged, project-authored
<code>emoji-annotations.txt</code> table,
+ <emphasis>derived facts</emphasis> computed from the
code point sequence itself
+ (<code>EmojiFlags</code> decodes a regional indicator
pair to its ISO 3166-1 alpha-2 code and
+ a subdivision tag sequence to its ISO 3166-2 code, with
no data file), and optional
+ <emphasis>joined facts</emphasis> resolved at run time
through an
+ <code>EmojiAnnotationJoin</code> hook against
user-installed data such as a gazetteer.
+ Every attribute value carries its own provenance
source, and gazetteer identifiers are never
+ baked into the bundled table.
+ </para>
+ <programlisting language="java">
+<![CDATA[EmojiAnnotator annotator = new EmojiAnnotator();
+TermAnalyzer analyzer = TermAnalyzer.builder().emojiFold().build();
+
+// "Berlin <German flag> <red heart>"
+String text = "Berlin \uD83C\uDDE9\uD83C\uDDEA \u2764\uFE0F";
+for (Term term : analyzer.analyze(text)) {
+ annotator.annotate(term).ifPresent(a -> {
+ a.name(); // "red heart" for the heart (CLDR short name)
+ a.sentiment(); // 2 for the heart (project-authored coarse score, -2..2)
+ a.entityType(); // FLAG for the flag, HEART for the heart
+ a.category(); // FLAGS / SMILEYS_AND_EMOTION
+ a.isoRegion(); // "DE" for the flag, decoded from the sequence itself
+ });
+}]]>
+ </programlisting>
+ <para>
+ <code>EmojiAnnotatorUsageExampleTest</code> asserts the
behavior shown here.
+ Annotations are per-symbol
<emphasis>metadata</emphasis>, not text transforms, so they are a
+ parallel accessor surface beside <code>Term</code>
rather than additional
+ <code>Dimension</code>s: <code>annotate(Term)</code>
reads the token's original layer, which
+ is the symbol the author actually wrote even after a
fold dimension has rewritten it to
+ ASCII. Annotation is total over arbitrary tokens (a
non-emoji token is simply empty, and
+ degenerate flag-shaped text yields no region instead of
an exception), while the strict
+ decoder <code>EmojiFlags.isoRegion()</code> fails
loudly on malformed sequences for callers
+ that want the contract. The bundled table's sentiment
scores are original project judgments
+ (tagged <code>UNSPECIFIED</code>); its names and its
entity-type/category assignments derive
+ from the Unicode-licensed CLDR short names and
<code>emoji-test.txt</code> group headers, and
+ no third-party sentiment data set is included.
+ </para>
+ <para>
+ The annotations are wired as strictly opt-in feature
inputs to three components, each
+ through its existing extension seam so that default
behavior and existing models are
+ unchanged: the name finder (and any consumer of feature
generation descriptors) activates
+ <code>EmojiAnnotationFeatureGenerator</code> by naming
+ <code>EmojiAnnotationFeatureGeneratorFactory</code> in
a descriptor, the document
+ categorizer takes <code>EmojiFeatureGenerator</code>
alongside its defaults through the
+ <code>DoccatFactory</code> feature generator array, and
sentiment analysis trains with
+ <code>EmojiSentimentFactory</code>, whose context
generator adds the emoji evidence and is
+ re-created from the model manifest at prediction time.
A flag token then carries
+ gazetteer-like evidence (<code>emojiType=FLAG</code>,
<code>emojiRegion=DE</code>) with no
+ dictionary, and a heart or crying face contributes its
coarse polarity directly.
+ </para>
+ </section>
+
<section xml:id="tools.normalizer.confusables">
<title>Confusable (homoglyph) folding</title>
<para>
diff --git a/src/license/NOTICE.template b/src/license/NOTICE.template
index 593720c09..da75e2a6a 100644
--- a/src/license/NOTICE.template
+++ b/src/license/NOTICE.template
@@ -123,6 +123,15 @@ top of each bundled file. These files are distributed
under the Unicode License
V3, the full text of which is reproduced in the LICENSE file accompanying this
distribution.
+The project-authored annotation table
+opennlp-core/opennlp-runtime/src/main/resources/opennlp/tools/util/normalizer/emoji-annotations.txt
+is licensed under the Apache License 2.0, but its name values (the CLDR short
+names) and its entity-type/category derivations are transcribed from the
+upstream emoji-test.txt (Emoji Keyboard/Display Test Data for UTS #51,
+version 17.0), distributed under the Unicode License V3 reproduced in the
+LICENSE file. Its sentiment scores are original project judgments; no
+third-party sentiment data set is included.
+
Copyright (c) 1991-2025 Unicode, Inc. All rights reserved.
============================================================================
\ No newline at end of file
