Kristian Rickert created OPENNLP-1870:
-----------------------------------------
Summary: Emoji annotation layer wired to Sentiment, Name Finder,
and Doccat
Key: OPENNLP-1870
URL: https://issues.apache.org/jira/browse/OPENNLP-1870
Project: OpenNLP
Issue Type: Task
Components: dl, Sentiment Detection
Affects Versions: 3.0.0-M4
Reporter: Kristian Rickert
Assignee: Kristian Rickert
*Stacks on:* OPENNLP-1869.
*Summary.* Extend the emoji lookup with a small, provenance-tagged annotation
layer per symbol: human-readable name (CLDR short-name), sentiment (with its
own {{sentiment_source}}), coarse entity type (VEHICLE, LANDMARK, ANIMAL, FLAG,
FOOD, ...), and a document-category hint. Surface these as parallel layers on
the {{Term}} model (original emoji, folded emoticon, name text, sentiment tag)
so each consumer picks the layer it needs. Wire the annotations as optional
input to Sentiment Detection, the Name Finder (name text plus entity-type as a
gazetteer-like signal), and the Document Categorizer. Each annotation dimension
is opt-in, so a consumer that only wants the folded text pays nothing for the
rest.
*Acceptance criteria.*
* Optional annotation dimensions (name, sentiment, entity type, category), each
with its own provenance/source field, surfaced through {{Term}}.
* Wired as optional features into Sentiment Detection, Name Finder, and
Document Categorizer.
* Each dimension independently opt-in; default behavior unchanged.
* Licensing cleared and documented for every bundled annotation source (see
epic-level LEGAL item).
h2. Out of scope (for the whole epic)
* DL whitespace/dash offset-safety (done in 1850:
{{NameFinderDL.findInOriginal}}, {{AbstractDL.normalizeInputAligned}}).
* The {{Alignment}} model and {{andThen}} themselves (delivered in 1850; this
epic consumes them).
* Changing any default normalization behavior. This epic adds offset-aware
capability and the provenance/annotation model; it does not turn expanding
folds on by default.
h2. Open questions to close in design review
# The gate (NFC/NFKC/case-fold edits): confirm option (c) for the MVP, or take
ICU4J. Gates OPENNLP-1867.
# CSV vs annotated {{.txt}} for the lookups; one combined file or one per fold
type.
# Offset-aware method on {{CharSequenceNormalizer}} directly vs a parallel
capability interface.
# Default aggressiveness: full case folding and NFKC change tokens visibly, so
likely opt-in, not default.
# Whether {{UNSPECIFIED}} entries are allowed in a release build at all, or
each must be reviewed and signed off.
# Which dataset anchors each annotation dimension (emoji-sentiment ranking,
CLDR for names, and what for entity type and category); same file or a separate
keyed file.
# Emoji/emoticon as punctuation vs a dedicated {{WordType}} for counting and
feature generation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
