[
https://issues.apache.org/jira/browse/OPENNLP-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1852:
--------------------------------------
Description:
h2. Summary
Normalization can change string length in both directions, and today only one
corner of the library accounts for it. Canonical and compatibility
decompositions, full case folding, ligature and fraction expansion, and
run-collapsing all change the number of characters between the input and the
normalized output. The {{CharClass}} engine already produces an {{OffsetMap}}
for its whitespace/dash folds (used by the DL components to map entity spans
back to the original text), but the broader {{CharSequenceNormalizer}} family
returns a plain {{CharSequence}} with no offset information. The goal of this
issue is to make the whole normalization pipeline offset-aware so that any
chain of expanding or contracting rungs can map a result position (an entity
span, a search hit, a highlight) back to original-document coordinates by
construction, not case by case.
This is a follow-up to OPENNLP-1850. The in-flight PRs already make the only
offset-sensitive path that applies length-changing folds today (the DL
whitespace/dash chunking in {{NameFinderDL}}) offset-safe. This issue
generalizes that so the rest of the normalizer family can be used safely in
offset-sensitive contexts.
h2. Background: there is a standard, and it classifies every character for us
We do not need to hand-enumerate "special characters." Unicode already catalogs
which characters expand or contract and to what, so the work is to consume that
data, not invent it.
|| Source standard || What it covers || Direction || Notes ||
| UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD) | Canonical and compatibility
(de)composition | both | Implemented by {{java.text.Normalizer}}; no lookup
table of our own required |
| UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX #44)
| Per-character expansions, tagged by type: {{<fraction>}} (1/2),
{{<super>}}/{{<sub>}} (squared to 2), {{<compat>}} (ligature fi, ellipsis to
three dots, Roman numerals), {{<square>}}, {{<wide>}}/{{<narrow>}} | expand |
This is the authoritative expansion catalog and the basis for NFKC |
| CaseFolding.txt (UTS #21 section 5.18) | Case folding, including the full
(expanding) folds with status F: eszett to ss, ligature ff to ff | expand and
1:1 | Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding,
so full folds genuinely need this data |
| Project folds (dash, quote, ellipsis, digit, confusable) | Our own target
choices, some derived from Unicode properties (Dash, White_Space) and some
hand-authored | both | These are the ones with no single governing standard;
see provenance below |
Contractions are the mirror image of the same data: NFC composition (combining
marks fold into one precomposed character), our whitespace and dash
run-collapses, and the UTF-16 case where a supplementary code point (two UTF-16
units) collapses to one unit.
h2. The core idea: one offset model, not a per-character table
Expansion and contraction are the same problem stated twice. They both mean
"the output length differs from the input length," and {{OffsetMap}} already
solves it generically: its builder records one original offset per output
character, so 1-to-3 (ellipsis), 3-to-1 (run collapse), and 2-to-1
(supplementary dash) are all just "map each output position back to its
source." Nothing in the runtime path is per-character. The character data only
decides _what_ each fold produces; {{OffsetMap}} handles _where everything
went_.
h2. Proposed architecture
# *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}}
family an offset-aware mode that returns a {{NormalizedText}} (normalized text
plus {{OffsetMap}}), in addition to the existing {{CharSequence}} method. A
length-preserving rung returns an identity map; a length-changing rung builds a
real one. Most rungs can derive the map cheaply during the pass they already
make.
# *Map composition.* Add {{OffsetMap.andThen(OffsetMap inner)}} (or an
equivalent composer) so a pipeline of N rungs collapses to a single
original-to-final map. Composition is associative and is what lets an arbitrary
chain stay offset-safe.
# *Offset-aware pipeline.* Add an offset-aware build/apply mode to
{{TextNormalizer}} that runs the configured rungs in order, composes their
maps, and returns one {{NormalizedText}} for the whole chain.
# *Reuse the standard engines where possible.* NFC/NFD/NFKC/NFKD go through
{{java.text.Normalizer}} with an offset map derived from the decomposition
boundaries; we do not maintain a decomposition table. Case folding uses the
bundled CaseFolding data. Only project-specific folds carry their own small
tables.
The existing {{CharClass.normalizeMapped}}/{{collapseMapped}} and the DL
{{normalizeInputMapped}} are the first instances of this pattern; this issue
lifts the pattern up to the whole family.
h2. Data files and provenance metadata
We will bundle a CaseFolding lookup (sourced from the Unicode
{{CaseFolding.txt}}) and aim for completeness against the upstream file. The
key addition is *provenance metadata*: every mapping records which standard it
follows, so we can audit coverage, distinguish standard-mandated folds from
project choices, and re-derive the table when Unicode revises.
Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a
documented header schema) with columns roughly:
{noformat}
source ; target ; fold_type ; standard ;
sentiment ; unicode_version ; notes
00DF ; 0073 0073 ; CASE_FOLD ; UTS21:CaseFolding:F ;
; 17.0.0 ; sharp s -> ss
2026 ; 002E 002E 002E ; NFKC ; UAX44:Decomposition:compat ;
; 17.0.0 ; horizontal ellipsis
2014 2013 ; 002D ; DASH ; UNSPECIFIED ;
; - ; dash-to-hyphen target choice
1F600 ; 003A 0029 ; EMOJI ; CLDR:annotation ;
positive ; 17.0.0 ; grinning face -> :) (expansion)
003A 0029 ; 1F642 ; EMOTICON ; UNSPECIFIED ;
positive ; - ; :) -> slightly smiling face (contraction)
{noformat}
The {{standard}} field is the important one. Suggested values:
* {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form folds
* {{UAX44:Decomposition:<type>}} for a specific decomposition type
* {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S
(simple), T (Turkic)
* {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a
published standard." This is the explicit "we just did that" marker the team
wants, so a reviewer or auditor immediately knows the entry is a project
decision (for example our choice of ASCII hyphen-minus as the dash target, or
expanding the ellipsis ourselves rather than relying on NFKC).
h2. Emoji and emoticons, both directions, sentiment, and token classification
Emoji-to-emoticon and emoticon-to-emoji are exactly the bidirectional,
length-changing case this issue is built for: an emoji rendered as {{:)}} is an
expansion, the reverse a contraction, and a supplementary-plane emoji is also
the two-UTF-16-units-collapsing case. Both directions are in scope, riding the
same offset-mapped rung and lookup table. There is no governing standard for
the mapping itself (emoticons are informal and the relation is many-to-many and
culturally variable), so most rows are tagged {{UNSPECIFIED}}; where a
direction can be anchored on published data we do (for example CLDR emoji
annotations for the emoji short-name) and tag it accordingly.
The bigger payoff is turning an opaque pictograph into *lexical signal*. Beyond
the emoticon form, the same lookup can carry the emoji's actual text, its CLDR
short-name (for example "grinning face"), so the rung can surface a real word
sequence instead of an out-of-vocabulary symbol that most models simply drop.
That text becomes the input that sentiment analysis, semantic search, embedding
pipelines, and LLM-style consumers actually read and reason about, which is
where this helps downstream understanding the most. The {{Term}} model is the
natural carrier: one token can expose {{original}} (the emoji), a folded
{{emoticon}}, a human-readable {{name}}, and a {{sentiment}} tag as parallel
layers, and each consumer picks the layer it needs (a sentiment lexicon match,
the name text for an embedding, or the original for display). Sentiment carries
its own provenance via a {{sentiment_source}} field (an emoji sentiment ranking
dataset, a project lexicon, or {{UNSPECIFIED}}), kept separate from the
mapping's {{standard}} tag so each is independently auditable and replaceable.
The same annotation mechanism generalizes well beyond sentiment. An emoji's
name is frequently an entity or concept in its own right (an airplane, a tower,
the Statue of Liberty, a national flag), so surfacing that name plus an
optional coarse entity-type tag (for example VEHICLE, LANDMARK, ANIMAL, FLAG,
FOOD) gives the Name Finder real lexical content and a gazetteer-like signal
where it would otherwise see an unknown glyph. The same row can also carry a
document-category hint (a pizza or an airplane is a strong cue for a food or a
travel document), feeding Document Categorizer features. So the lookup is less
a single sentiment column and more a small, provenance-tagged annotation layer
per symbol: name, sentiment, entity type, and category, each optional and each
with its own source, surfaced through the {{Term}} model and consumed by the
Sentiment Detection, Name Finder, and Doccat components respectively. Each
annotation dimension is opt-in, so a consumer that only wants the folded text
pays nothing for the rest.
Token classification ties in too. The UAX #29 tokenizer already gives emoji
their own {{WordType}} category, so the same data lets downstream counting and
feature generation treat emoji and emoticons as a consistent class rather than
stray symbols, for example folding them into a punctuation-like bucket or a
dedicated emoji feature for corpus statistics and the ML feature generators.
That keeps a document's token counts and features stable whether sentiment is
expressed as {{:)}}, as an emoji, or as the spelled-out name.
h2. Reconciliation
The runtime fold tables will be reconciled from several sources, so we need
explicit rules:
# *Prefer standard engines over tables.* For anything {{java.text.Normalizer}}
already does (NFC/NFKC/NFD/NFKD), use it and do not duplicate the mapping in a
file. The file is for what Java does not give us: full case folding and our
project folds.
# *Provenance on every entry.* Each bundled mapping carries its {{standard}}
tag (above). A build/test step audits the table against the authoritative
upstream file for that standard (for example: every CaseFolding.txt status C
and F row is present and identical) and fails on drift or gaps, which is how we
keep "try to make it complete" honest.
# *Conflict precedence.* If two sources map the same code point differently for
the same fold type, the more specific published standard wins over a general
one, and any deviation we deliberately keep must be re-tagged {{UNSPECIFIED}}
with a note, never left looking standard-backed.
# *Do we need to roll our own?* Mostly no. The open question to settle in this
issue is the exact boundary: confirm that NFC/NFKC and case folding are fully
delegated to standard data, and that {{UNSPECIFIED}} is reserved for the
genuinely project-specific folds (dash target, quote folding,
ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable
skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
h2. Open questions for the design review
* CSV vs annotated {{.txt}} for the lookup, and whether one combined file or
one file per fold type.
* Whether the offset-aware method lives on {{CharSequenceNormalizer}} directly
(every rung implements it) or on a parallel interface that the pipeline adapts.
* How aggressively to expand by default: full case folding (eszett to ss) and
NFKC change tokens visibly, so they should likely stay opt-in rungs, not
defaults.
* Whether {{UNSPECIFIED}} entries should be allowed in a release build at all,
or must each be reviewed and signed off.
* Which dataset anchors each annotation dimension (an emoji sentiment ranking
for sentiment, CLDR for names, and what for entity type and category), and
whether the annotations live in the same lookup or a separate file keyed by
code point.
* Whether emoji and emoticons classify as punctuation or as a dedicated emoji
{{WordType}} for counting and feature-generation purposes.
h2. Scope and acceptance criteria
* {{OffsetMap.andThen}} composition with tests, including expand-then-contract
and contract-then-expand chains.
* An offset-aware mode on the normalizer family and on {{TextNormalizer}},
returning a composed {{NormalizedText}} for a multi-rung chain.
* A bundled, provenance-tagged CaseFolding lookup with a completeness audit
test against upstream {{CaseFolding.txt}}.
* Round-trip tests: for representative expanding (ellipsis, eszett, ligature,
fraction) and contracting (run collapse, supplementary dash, NFC compose)
inputs, a position in the normalized text maps back to the correct original
characters.
* A bidirectional, opt-in emoji/emoticon rung backed by the same
provenance-tagged lookup, offset-mapped like every other rung, with round-trip
tests in both directions.
* A small, provenance-tagged annotation layer on the lookup (name, sentiment,
entity type, category), surfaced through the {{Term}} model and wired as
optional input to the Sentiment Detection, Name Finder, and Document
Categorizer components.
* Consistent {{WordType}}-level classification of emoji and emoticons so token
counts and ML feature generators treat them as one class.
h2. Out of scope
* The DL whitespace/dash offset-safety, already handled under OPENNLP-1850
({{NameFinderDL.findInOriginal}} plus {{AbstractDL.normalizeInputMapped}}).
* Changing any default normalization behavior; this issue adds the offset-aware
capability and the provenance model, it does not turn expanding folds on by
default.
was:
h2. Summary
Normalization can change string length in both directions, and today only one
corner of the library accounts for it. Canonical and compatibility
decompositions, full case folding, ligature and fraction expansion, and
run-collapsing all change the number of characters between the input and the
normalized output. The {{CharClass}} engine already produces an {{OffsetMap}}
for its whitespace/dash folds (used by the DL components to map entity spans
back to the original text), but the broader {{CharSequenceNormalizer}} family
returns a plain {{CharSequence}} with no offset information. The goal of this
issue is to make the whole normalization pipeline offset-aware so that any
chain of expanding or contracting rungs can map a result position (an entity
span, a search hit, a highlight) back to original-document coordinates by
construction, not case by case.
This is a follow-up to OPENNLP-1850. The in-flight PRs already make the only
offset-sensitive path that applies length-changing folds today (the DL
whitespace/dash chunking in {{{}NameFinderDL{}}}) offset-safe. This issue
generalizes that so the rest of the normalizer family can be used safely in
offset-sensitive contexts.
h2. Background: there is a standard, and it classifies every character for us
We do not need to hand-enumerate "special characters." Unicode already catalogs
which characters expand or contract and to what, so the work is to consume that
data, not invent it.
||Source standard||What it covers||Direction||Notes||
|UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD)|Canonical and compatibility
(de)composition|both|Implemented by {{{}java.text.Normalizer{}}}; no lookup
table of our own required|
|UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX
#44)|Per-character expansions, tagged by type: {{<fraction>}} (1/2),
{{{}<super>{}}}/{{{}<sub>{}}} (squared to 2), {{<compat>}} (ligature fi,
ellipsis to three dots, Roman numerals), {{{}<square>{}}},
{{{}<wide>{}}}/{{{}<narrow>{}}}|expand|This is the authoritative expansion
catalog and the basis for NFKC|
|CaseFolding.txt (UTS #21 section 5.18)|Case folding, including the full
(expanding) folds with status F: eszett to ss, ligature ff to ff|expand and
1:1|Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding, so
full folds genuinely need this data|
|Project folds (dash, quote, ellipsis, digit, confusable)|Our own target
choices, some derived from Unicode properties (Dash, White_Space) and some
hand-authored|both|These are the ones with no single governing standard; see
provenance below|
Contractions are the mirror image of the same data: NFC composition (combining
marks fold into one precomposed character), our whitespace and dash
run-collapses, and the UTF-16 case where a supplementary code point (two UTF-16
units) collapses to one unit.
h2. The core idea: one offset model, not a per-character table
Expansion and contraction are the same problem stated twice. They both mean
"the output length differs from the input length," and {{OffsetMap}} already
solves it generically: its builder records one original offset per output
character, so 1-to-3 (ellipsis), 3-to-1 (run collapse), and 2-to-1
(supplementary dash) are all just "map each output position back to its
source." Nothing in the runtime path is per-character. The character data only
decides _what_ each fold produces; {{OffsetMap}} handles {_}where everything
went{_}.
h2. Proposed architecture
# *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}}
family an offset-aware mode that returns a {{NormalizedText}} (normalized text
plus {{{}OffsetMap{}}}), in addition to the existing {{CharSequence}} method. A
length-preserving rung returns an identity map; a length-changing rung builds a
real one. Most rungs can derive the map cheaply during the pass they already
make.
# *Map composition.* Add {{OffsetMap.andThen(OffsetMap inner)}} (or an
equivalent composer) so a pipeline of N rungs collapses to a single
original-to-final map. Composition is associative and is what lets an arbitrary
chain stay offset-safe.
# *Offset-aware pipeline.* Add an offset-aware build/apply mode to
{{TextNormalizer}} that runs the configured rungs in order, composes their
maps, and returns one {{NormalizedText}} for the whole chain.
# *Reuse the standard engines where possible.* NFC/NFD/NFKC/NFKD go through
{{java.text.Normalizer}} with an offset map derived from the decomposition
boundaries; we do not maintain a decomposition table. Case folding uses the
bundled CaseFolding data. Only project-specific folds carry their own small
tables.
The existing {{{}CharClass.normalizeMapped{}}}/{{{}collapseMapped{}}} and the
DL {{normalizeInputMapped}} are the first instances of this pattern; this issue
lifts the pattern up to the whole family.
h2. Data files and provenance metadata
We will bundle a CaseFolding lookup (sourced from the Unicode
{{{}CaseFolding.txt{}}}) and aim for completeness against the upstream file.
The key addition is {*}provenance metadata{*}: every mapping records which
standard it follows, so we can audit coverage, distinguish standard-mandated
folds from project choices, and re-derive the table when Unicode revises.
Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a
documented header schema) with columns roughly:
{noformat}
source ; target ; fold_type ; standard ;
unicode_version ; notes
00DF ; 0073 0073 ; CASE_FOLD ; UTS21:CaseFolding:F ;
17.0.0 ; LATIN SMALL LETTER SHARP S -> ss
2026 ; 002E 002E 002E ; NFKC ; UAX44:Decomposition:compat ;
17.0.0 ; HORIZONTAL ELLIPSIS
2014 2013 ; 002D ; DASH ; UNSPECIFIED ; -
; project dash-to-hyphen target choice
{noformat}
The {{standard}} field is the important one. Suggested values:
* {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form folds
* {{UAX44:Decomposition:<type>}} for a specific decomposition type
* {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S
(simple), T (Turkic)
* {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a
published standard." This is the explicit "we just did that" marker the team
wants, so a reviewer or auditor immediately knows the entry is a project
decision (for example our choice of ASCII hyphen-minus as the dash target, or
expanding the ellipsis ourselves rather than relying on NFKC).
h2. Reconciliation
The runtime fold tables will be reconciled from several sources, so we need
explicit rules:
# *Prefer standard engines over tables.* For anything {{java.text.Normalizer}}
already does (NFC/NFKC/NFD/NFKD), use it and do not duplicate the mapping in a
file. The file is for what Java does not give us: full case folding and our
project folds.
# *Provenance on every entry.* Each bundled mapping carries its {{standard}}
tag (above). A build/test step audits the table against the authoritative
upstream file for that standard (for example: every CaseFolding.txt status C
and F row is present and identical) and fails on drift or gaps, which is how we
keep "try to make it complete" honest.
# *Conflict precedence.* If two sources map the same code point differently
for the same fold type, the more specific published standard wins over a
general one, and any deviation we deliberately keep must be re-tagged
{{UNSPECIFIED}} with a note, never left looking standard-backed.
# *Do we need to roll our own?* Mostly no. The open question to settle in this
issue is the exact boundary: confirm that NFC/NFKC and case folding are fully
delegated to standard data, and that {{UNSPECIFIED}} is reserved for the
genuinely project-specific folds (dash target, quote folding,
ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable
skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
h2. Open questions for the design review
* CSV vs annotated {{.txt}} for the lookup, and whether one combined file or
one file per fold type.
* Whether the offset-aware method lives on {{CharSequenceNormalizer}} directly
(every rung implements it) or on a parallel interface that the pipeline adapts.
* How aggressively to expand by default: full case folding (eszett to ss) and
NFKC change tokens visibly, so they should likely stay opt-in rungs, not
defaults.
* Whether {{UNSPECIFIED}} entries should be allowed in a release build at all,
or must each be reviewed and signed off.
h2. Scope and acceptance criteria
* {{OffsetMap.andThen}} composition with tests, including expand-then-contract
and contract-then-expand chains.
* An offset-aware mode on the normalizer family and on {{{}TextNormalizer{}}},
returning a composed {{NormalizedText}} for a multi-rung chain.
* A bundled, provenance-tagged CaseFolding lookup with a completeness audit
test against upstream {{{}CaseFolding.txt{}}}.
* Round-trip tests: for representative expanding (ellipsis, eszett, ligature,
fraction) and contracting (run collapse, supplementary dash, NFC compose)
inputs, a position in the normalized text maps back to the correct original
characters.
h2. Out of scope
* The DL whitespace/dash offset-safety, already handled under OPENNLP-1850
({{{}NameFinderDL.findInOriginal{}}} plus
{{{}AbstractDL.normalizeInputMapped{}}}).
* Changing any default normalization behavior; this issue adds the
offset-aware capability and the provenance model, it does not turn expanding
folds on by default.
> Offset-aware normalization pipeline: handle length-changing folds (expansion
> and contraction) with a single offset model
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1852
> URL: https://issues.apache.org/jira/browse/OPENNLP-1852
> Project: OpenNLP
> Issue Type: New Feature
> Components: Tokenizer
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
>
> h2. Summary
> Normalization can change string length in both directions, and today only one
> corner of the library accounts for it. Canonical and compatibility
> decompositions, full case folding, ligature and fraction expansion, and
> run-collapsing all change the number of characters between the input and the
> normalized output. The {{CharClass}} engine already produces an {{OffsetMap}}
> for its whitespace/dash folds (used by the DL components to map entity spans
> back to the original text), but the broader {{CharSequenceNormalizer}} family
> returns a plain {{CharSequence}} with no offset information. The goal of this
> issue is to make the whole normalization pipeline offset-aware so that any
> chain of expanding or contracting rungs can map a result position (an entity
> span, a search hit, a highlight) back to original-document coordinates by
> construction, not case by case.
> This is a follow-up to OPENNLP-1850. The in-flight PRs already make the only
> offset-sensitive path that applies length-changing folds today (the DL
> whitespace/dash chunking in {{NameFinderDL}}) offset-safe. This issue
> generalizes that so the rest of the normalizer family can be used safely in
> offset-sensitive contexts.
> h2. Background: there is a standard, and it classifies every character for us
> We do not need to hand-enumerate "special characters." Unicode already
> catalogs which characters expand or contract and to what, so the work is to
> consume that data, not invent it.
> || Source standard || What it covers || Direction || Notes ||
> | UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD) | Canonical and
> compatibility (de)composition | both | Implemented by
> {{java.text.Normalizer}}; no lookup table of our own required |
> | UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX
> #44) | Per-character expansions, tagged by type: {{<fraction>}} (1/2),
> {{<super>}}/{{<sub>}} (squared to 2), {{<compat>}} (ligature fi, ellipsis to
> three dots, Roman numerals), {{<square>}}, {{<wide>}}/{{<narrow>}} | expand |
> This is the authoritative expansion catalog and the basis for NFKC |
> | CaseFolding.txt (UTS #21 section 5.18) | Case folding, including the full
> (expanding) folds with status F: eszett to ss, ligature ff to ff | expand and
> 1:1 | Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding,
> so full folds genuinely need this data |
> | Project folds (dash, quote, ellipsis, digit, confusable) | Our own target
> choices, some derived from Unicode properties (Dash, White_Space) and some
> hand-authored | both | These are the ones with no single governing standard;
> see provenance below |
> Contractions are the mirror image of the same data: NFC composition
> (combining marks fold into one precomposed character), our whitespace and
> dash run-collapses, and the UTF-16 case where a supplementary code point (two
> UTF-16 units) collapses to one unit.
> h2. The core idea: one offset model, not a per-character table
> Expansion and contraction are the same problem stated twice. They both mean
> "the output length differs from the input length," and {{OffsetMap}} already
> solves it generically: its builder records one original offset per output
> character, so 1-to-3 (ellipsis), 3-to-1 (run collapse), and 2-to-1
> (supplementary dash) are all just "map each output position back to its
> source." Nothing in the runtime path is per-character. The character data
> only decides _what_ each fold produces; {{OffsetMap}} handles _where
> everything went_.
> h2. Proposed architecture
> # *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}}
> family an offset-aware mode that returns a {{NormalizedText}} (normalized
> text plus {{OffsetMap}}), in addition to the existing {{CharSequence}}
> method. A length-preserving rung returns an identity map; a length-changing
> rung builds a real one. Most rungs can derive the map cheaply during the pass
> they already make.
> # *Map composition.* Add {{OffsetMap.andThen(OffsetMap inner)}} (or an
> equivalent composer) so a pipeline of N rungs collapses to a single
> original-to-final map. Composition is associative and is what lets an
> arbitrary chain stay offset-safe.
> # *Offset-aware pipeline.* Add an offset-aware build/apply mode to
> {{TextNormalizer}} that runs the configured rungs in order, composes their
> maps, and returns one {{NormalizedText}} for the whole chain.
> # *Reuse the standard engines where possible.* NFC/NFD/NFKC/NFKD go through
> {{java.text.Normalizer}} with an offset map derived from the decomposition
> boundaries; we do not maintain a decomposition table. Case folding uses the
> bundled CaseFolding data. Only project-specific folds carry their own small
> tables.
> The existing {{CharClass.normalizeMapped}}/{{collapseMapped}} and the DL
> {{normalizeInputMapped}} are the first instances of this pattern; this issue
> lifts the pattern up to the whole family.
> h2. Data files and provenance metadata
> We will bundle a CaseFolding lookup (sourced from the Unicode
> {{CaseFolding.txt}}) and aim for completeness against the upstream file. The
> key addition is *provenance metadata*: every mapping records which standard
> it follows, so we can audit coverage, distinguish standard-mandated folds
> from project choices, and re-derive the table when Unicode revises.
> Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a
> documented header schema) with columns roughly:
> {noformat}
> source ; target ; fold_type ; standard ;
> sentiment ; unicode_version ; notes
> 00DF ; 0073 0073 ; CASE_FOLD ; UTS21:CaseFolding:F ;
> ; 17.0.0 ; sharp s -> ss
> 2026 ; 002E 002E 002E ; NFKC ; UAX44:Decomposition:compat ;
> ; 17.0.0 ; horizontal ellipsis
> 2014 2013 ; 002D ; DASH ; UNSPECIFIED ;
> ; - ; dash-to-hyphen target choice
> 1F600 ; 003A 0029 ; EMOJI ; CLDR:annotation ;
> positive ; 17.0.0 ; grinning face -> :) (expansion)
> 003A 0029 ; 1F642 ; EMOTICON ; UNSPECIFIED ;
> positive ; - ; :) -> slightly smiling face (contraction)
> {noformat}
> The {{standard}} field is the important one. Suggested values:
> * {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form
> folds
> * {{UAX44:Decomposition:<type>}} for a specific decomposition type
> * {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S
> (simple), T (Turkic)
> * {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a
> published standard." This is the explicit "we just did that" marker the team
> wants, so a reviewer or auditor immediately knows the entry is a project
> decision (for example our choice of ASCII hyphen-minus as the dash target, or
> expanding the ellipsis ourselves rather than relying on NFKC).
> h2. Emoji and emoticons, both directions, sentiment, and token classification
> Emoji-to-emoticon and emoticon-to-emoji are exactly the bidirectional,
> length-changing case this issue is built for: an emoji rendered as {{:)}} is
> an expansion, the reverse a contraction, and a supplementary-plane emoji is
> also the two-UTF-16-units-collapsing case. Both directions are in scope,
> riding the same offset-mapped rung and lookup table. There is no governing
> standard for the mapping itself (emoticons are informal and the relation is
> many-to-many and culturally variable), so most rows are tagged
> {{UNSPECIFIED}}; where a direction can be anchored on published data we do
> (for example CLDR emoji annotations for the emoji short-name) and tag it
> accordingly.
> The bigger payoff is turning an opaque pictograph into *lexical signal*.
> Beyond the emoticon form, the same lookup can carry the emoji's actual text,
> its CLDR short-name (for example "grinning face"), so the rung can surface a
> real word sequence instead of an out-of-vocabulary symbol that most models
> simply drop. That text becomes the input that sentiment analysis, semantic
> search, embedding pipelines, and LLM-style consumers actually read and reason
> about, which is where this helps downstream understanding the most. The
> {{Term}} model is the natural carrier: one token can expose {{original}} (the
> emoji), a folded {{emoticon}}, a human-readable {{name}}, and a {{sentiment}}
> tag as parallel layers, and each consumer picks the layer it needs (a
> sentiment lexicon match, the name text for an embedding, or the original for
> display). Sentiment carries its own provenance via a {{sentiment_source}}
> field (an emoji sentiment ranking dataset, a project lexicon, or
> {{UNSPECIFIED}}), kept separate from the mapping's {{standard}} tag so each
> is independently auditable and replaceable.
> The same annotation mechanism generalizes well beyond sentiment. An emoji's
> name is frequently an entity or concept in its own right (an airplane, a
> tower, the Statue of Liberty, a national flag), so surfacing that name plus
> an optional coarse entity-type tag (for example VEHICLE, LANDMARK, ANIMAL,
> FLAG, FOOD) gives the Name Finder real lexical content and a gazetteer-like
> signal where it would otherwise see an unknown glyph. The same row can also
> carry a document-category hint (a pizza or an airplane is a strong cue for a
> food or a travel document), feeding Document Categorizer features. So the
> lookup is less a single sentiment column and more a small, provenance-tagged
> annotation layer per symbol: name, sentiment, entity type, and category, each
> optional and each with its own source, surfaced through the {{Term}} model
> and consumed by the Sentiment Detection, Name Finder, and Doccat components
> respectively. Each annotation dimension is opt-in, so a consumer that only
> wants the folded text pays nothing for the rest.
> Token classification ties in too. The UAX #29 tokenizer already gives emoji
> their own {{WordType}} category, so the same data lets downstream counting
> and feature generation treat emoji and emoticons as a consistent class rather
> than stray symbols, for example folding them into a punctuation-like bucket
> or a dedicated emoji feature for corpus statistics and the ML feature
> generators. That keeps a document's token counts and features stable whether
> sentiment is expressed as {{:)}}, as an emoji, or as the spelled-out name.
> h2. Reconciliation
> The runtime fold tables will be reconciled from several sources, so we need
> explicit rules:
> # *Prefer standard engines over tables.* For anything
> {{java.text.Normalizer}} already does (NFC/NFKC/NFD/NFKD), use it and do not
> duplicate the mapping in a file. The file is for what Java does not give us:
> full case folding and our project folds.
> # *Provenance on every entry.* Each bundled mapping carries its {{standard}}
> tag (above). A build/test step audits the table against the authoritative
> upstream file for that standard (for example: every CaseFolding.txt status C
> and F row is present and identical) and fails on drift or gaps, which is how
> we keep "try to make it complete" honest.
> # *Conflict precedence.* If two sources map the same code point differently
> for the same fold type, the more specific published standard wins over a
> general one, and any deviation we deliberately keep must be re-tagged
> {{UNSPECIFIED}} with a note, never left looking standard-backed.
> # *Do we need to roll our own?* Mostly no. The open question to settle in
> this issue is the exact boundary: confirm that NFC/NFKC and case folding are
> fully delegated to standard data, and that {{UNSPECIFIED}} is reserved for
> the genuinely project-specific folds (dash target, quote folding,
> ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable
> skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
> h2. Open questions for the design review
> * CSV vs annotated {{.txt}} for the lookup, and whether one combined file or
> one file per fold type.
> * Whether the offset-aware method lives on {{CharSequenceNormalizer}}
> directly (every rung implements it) or on a parallel interface that the
> pipeline adapts.
> * How aggressively to expand by default: full case folding (eszett to ss) and
> NFKC change tokens visibly, so they should likely stay opt-in rungs, not
> defaults.
> * Whether {{UNSPECIFIED}} entries should be allowed in a release build at
> all, or must each be reviewed and signed off.
> * Which dataset anchors each annotation dimension (an emoji sentiment ranking
> for sentiment, CLDR for names, and what for entity type and category), and
> whether the annotations live in the same lookup or a separate file keyed by
> code point.
> * Whether emoji and emoticons classify as punctuation or as a dedicated emoji
> {{WordType}} for counting and feature-generation purposes.
> h2. Scope and acceptance criteria
> * {{OffsetMap.andThen}} composition with tests, including
> expand-then-contract and contract-then-expand chains.
> * An offset-aware mode on the normalizer family and on {{TextNormalizer}},
> returning a composed {{NormalizedText}} for a multi-rung chain.
> * A bundled, provenance-tagged CaseFolding lookup with a completeness audit
> test against upstream {{CaseFolding.txt}}.
> * Round-trip tests: for representative expanding (ellipsis, eszett, ligature,
> fraction) and contracting (run collapse, supplementary dash, NFC compose)
> inputs, a position in the normalized text maps back to the correct original
> characters.
> * A bidirectional, opt-in emoji/emoticon rung backed by the same
> provenance-tagged lookup, offset-mapped like every other rung, with
> round-trip tests in both directions.
> * A small, provenance-tagged annotation layer on the lookup (name, sentiment,
> entity type, category), surfaced through the {{Term}} model and wired as
> optional input to the Sentiment Detection, Name Finder, and Document
> Categorizer components.
> * Consistent {{WordType}}-level classification of emoji and emoticons so
> token counts and ML feature generators treat them as one class.
> h2. Out of scope
> * The DL whitespace/dash offset-safety, already handled under OPENNLP-1850
> ({{NameFinderDL.findInOriginal}} plus {{AbstractDL.normalizeInputMapped}}).
> * Changing any default normalization behavior; this issue adds the
> offset-aware capability and the provenance model, it does not turn expanding
> folds on by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
