[
https://issues.apache.org/jira/browse/OPENNLP-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1852:
--------------------------------------
Description:
h2. Summary
Normalization can change string length in both directions, and today only one
corner of the library accounts for it. Canonical and compatibility
decompositions, full case folding, ligature and fraction expansion, and
run-collapsing all change the number of characters between the input and the
normalized output. The {{CharClass}} engine already produces an {{Alignment}}
(carried in an {{AlignedText}}) for its whitespace/dash folds, used by the DL
components to map entity spans back to the original text, but the broader
{{CharSequenceNormalizer}} family returns a plain {{CharSequence}} with no
offset information. The goal of this issue is to make the whole normalization
pipeline offset-aware so that any chain of expanding or contracting rungs can
map a result position (an entity span, a search hit, a highlight) back to
original-document coordinates by construction, not case by case.
This is a follow-up to OPENNLP-1850. That work delivered the offset *model*
this issue builds on (see "Already delivered" below) and made the only
offset-sensitive path that applies length-changing folds today (the DL
whitespace/dash chunking in {{NameFinderDL}}) offset-safe. This issue
generalizes the model across the rest of the normalizer family so it can be
used safely in offset-sensitive contexts.
h2. Already delivered in OPENNLP-1850 (do not re-scope here)
The keystone is in place; this issue should build on it rather than reinvent it.
* {{Alignment}} -- a bidirectional edit-sequence offset model
({{equal}}/{{replace}} runs) that represents expansion, contraction, deletion
(as gaps), and 1:1 changes uniformly, and maps span-to-span with
{{toOriginalSpan}}/{{toNormalizedSpan}}. It replaced the earlier
per-output-character {{OffsetMap}}, which assumed the normalized text
contiguously covered the original and over-covered deletions. ({{OffsetMap}}
and {{NormalizedText}} were never released and are removed.)
* {{Alignment.andThen(Alignment)}} -- associative composition of two stages
into one original-to-final alignment. Built and tested, including
expand-then-contract and contract-then-expand chains.
* The offset-aware *contract* on {{CharClass}}: {{normalizeAligned}},
{{collapseAligned}}, {{collapsePreservingAligned}}, {{trimAligned}},
{{removeAllAligned}}, each returning an {{AlignedText}} (normalized text plus
{{Alignment}}).
* The DL path consuming it: {{AbstractDL.normalizeInputAligned}} and
{{NameFinderDL.findInOriginal}} map decoded spans back via
{{Alignment.toOriginalSpan}}.
So the remaining work is to lift the offset-aware contract from {{CharClass}}
up to the whole {{CharSequenceNormalizer}} family and the {{TextNormalizer}}
pipeline, add the provenance-tagged data, and add the emoji/emoticon rung and
annotation layer.
h2. Background: there is a standard, and it classifies every character for us
We do not need to hand-enumerate "special characters." Unicode already catalogs
which characters expand or contract and to what, so the work is to consume that
data, not invent it.
|| Source standard || What it covers || Direction || Notes ||
| UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD) | Canonical and compatibility
(de)composition | both | Implemented by {{java.text.Normalizer}} -- but see the
edit-tracking caveat under Proposed architecture: it returns a plain string
with no edit information |
| UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX #44)
| Per-character expansions, tagged by type: {{<fraction>}} (1/2),
{{<super>}}/{{<sub>}} (squared to 2), {{<compat>}} (ligature fi, ellipsis to
three dots, Roman numerals), {{<square>}}, {{<wide>}}/{{<narrow>}} | expand |
This is the authoritative expansion catalog and the basis for NFKC |
| CaseFolding.txt (UTS #21 section 5.18) | Case folding, including the full
(expanding) folds with status F: eszett to ss, ligature ff to ff | expand and
1:1 | Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding,
so full folds genuinely need this data |
| Project folds (dash, quote, ellipsis, digit, confusable) | Our own target
choices, some derived from Unicode properties (Dash, White_Space) and some
hand-authored | both | These are the ones with no single governing standard;
see provenance below |
Contractions are the mirror image of the same data: NFC composition (combining
marks fold into one precomposed character), our whitespace and dash
run-collapses, and the UTF-16 case where a supplementary code point (two UTF-16
units) collapses to one unit.
h2. The core idea: one offset model, not a per-character table
Expansion and contraction are the same problem stated twice: the output length
differs from the input length. {{Alignment}} solves it generically. It records
the edits between input and output as a sequence of _equal_ runs (text copied
through, length unchanged) and _replace_ runs (M original characters that
produced P normalized characters), so 1-to-3 (ellipsis), 3-to-1 (run collapse),
2-to-1 (supplementary dash), and N-to-0 (deletion) are all just runs. It maps
span-to-span, so a match that ends next to deleted text reports a tight
original span instead of over-covering the deletion, and it composes with
{{andThen}}. The character data only decides _what_ each fold produces;
{{Alignment}} handles _where everything went_.
h2. Proposed architecture
# *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}}
family an offset-aware mode that returns an {{AlignedText}} (normalized text
plus {{Alignment}}), in addition to the existing {{CharSequence}} method. A
length-preserving rung returns an identity alignment; a length-changing rung
builds a real one. The {{CharClass}}-backed rungs derive it cheaply during the
pass they already make; the standard-engine rungs are the hard case (next
point).
# *Map composition.* {{Alignment.andThen}} -- *already delivered*. A pipeline
of N rungs collapses to a single original-to-final alignment by folding
{{andThen}} over the stages.
# *Offset-aware pipeline.* Add an offset-aware build/apply mode to
{{TextNormalizer}} that runs the configured rungs in order, composes their
alignments, and returns one {{AlignedText}} for the whole chain.
# *Reuse the standard engines where possible -- with a known edit-tracking
gap.* NFC/NFD/NFKC/NFKD and full case folding are the value of delegating to
standards, but {{java.text.Normalizer.normalize(s, form)}} returns only the
result string with *no edit information*, so an {{Alignment}} cannot be read
out of it directly. The realistic options, to be decided in design review: (a)
depend on ICU4J's {{Normalizer2}}, which can emit an {{Edits}} object that maps
directly onto our {{equal}}/{{replace}} runs; (b) compute the alignment by
diffing input against output at decomposition boundaries; or (c) restrict the
offset-aware path to folds we generate ourselves (the {{CharClass}} family plus
a bundled-table case fold) and treat {{java.text.Normalizer}}-only NFC/NFKC as
offset-opaque. This is the single biggest design decision in the issue and
should be settled first.
The {{CharClass}} {{*Aligned}} family and the DL {{normalizeInputAligned}} are
the first instances of the offset-aware contract; this issue lifts the pattern
up to the whole family.
h2. Data files and provenance metadata
We will bundle a CaseFolding lookup (sourced from the Unicode
{{CaseFolding.txt}}) and aim for completeness against the upstream file. The
key addition is *provenance metadata*: every mapping records which standard it
follows, so we can audit coverage, distinguish standard-mandated folds from
project choices, and re-derive the table when Unicode revises.
Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a
documented header schema) with columns roughly:
{noformat}
source ; target ; fold_type ; standard ;
sentiment ; unicode_version ; notes
00DF ; 0073 0073 ; CASE_FOLD ; UTS21:CaseFolding:F ;
; 17.0.0 ; sharp s -> ss
2026 ; 002E 002E 002E ; NFKC ; UAX44:Decomposition:compat ;
; 17.0.0 ; horizontal ellipsis
2014 2013 ; 002D ; DASH ; UNSPECIFIED ;
; - ; dash-to-hyphen target choice
1F600 ; 003A 0029 ; EMOJI ; CLDR:annotation ;
positive ; 17.0.0 ; grinning face -> :) (expansion)
003A 0029 ; 1F642 ; EMOTICON ; UNSPECIFIED ;
positive ; - ; :) -> slightly smiling face (contraction)
{noformat}
The {{standard}} field is the important one. Suggested values:
* {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form folds
* {{UAX44:Decomposition:<type>}} for a specific decomposition type
* {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S
(simple), T (Turkic)
* {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a
published standard." This is the explicit "we just did that" marker the team
wants, so a reviewer or auditor immediately knows the entry is a project
decision (for example our choice of ASCII hyphen-minus as the dash target, or
expanding the ellipsis ourselves rather than relying on NFKC).
h2. Emoji and emoticons, both directions, sentiment, and token classification
Emoji-to-emoticon and emoticon-to-emoji are exactly the bidirectional,
length-changing case this issue is built for: an emoji rendered as {{:)}} is an
expansion, the reverse a contraction, and a supplementary-plane emoji is also
the two-UTF-16-units-collapsing case. Both directions are in scope, riding the
same offset-aware rung and lookup table. There is no governing standard for the
mapping itself (emoticons are informal and the relation is many-to-many and
culturally variable), so most rows are tagged {{UNSPECIFIED}}; where a
direction can be anchored on published data we do (for example CLDR emoji
annotations for the emoji short-name) and tag it accordingly.
The bigger payoff is turning an opaque pictograph into *lexical signal*. Beyond
the emoticon form, the same lookup can carry the emoji's actual text, its CLDR
short-name (for example "grinning face"), so the rung can surface a real word
sequence instead of an out-of-vocabulary symbol that most models simply drop.
That text becomes the input that sentiment analysis, semantic search, embedding
pipelines, and LLM-style consumers actually read and reason about, which is
where this helps downstream understanding the most. The {{Term}} model is the
natural carrier: one token can expose {{original}} (the emoji), a folded
{{emoticon}}, a human-readable {{name}}, and a {{sentiment}} tag as parallel
layers, and each consumer picks the layer it needs (a sentiment lexicon match,
the name text for an embedding, or the original for display). Sentiment carries
its own provenance via a {{sentiment_source}} field (an emoji sentiment ranking
dataset, a project lexicon, or {{UNSPECIFIED}}), kept separate from the
mapping's {{standard}} tag so each is independently auditable and replaceable.
The same annotation mechanism generalizes well beyond sentiment. An emoji's
name is frequently an entity or concept in its own right (an airplane, a tower,
the Statue of Liberty, a national flag), so surfacing that name plus an
optional coarse entity-type tag (for example VEHICLE, LANDMARK, ANIMAL, FLAG,
FOOD) gives the Name Finder real lexical content and a gazetteer-like signal
where it would otherwise see an unknown glyph. The same row can also carry a
document-category hint (a pizza or an airplane is a strong cue for a food or a
travel document), feeding Document Categorizer features. So the lookup is less
a single sentiment column and more a small, provenance-tagged annotation layer
per symbol: name, sentiment, entity type, and category, each optional and each
with its own source, surfaced through the {{Term}} model and consumed by the
Sentiment Detection, Name Finder, and Doccat components respectively. Each
annotation dimension is opt-in, so a consumer that only wants the folded text
pays nothing for the rest.
Token classification ties in too. The UAX #29 tokenizer already gives emoji
their own {{WordType}} category, so the same data lets downstream counting and
feature generation treat emoji and emoticons as a consistent class rather than
stray symbols, for example folding them into a punctuation-like bucket or a
dedicated emoji feature for corpus statistics and the ML feature generators.
That keeps a document's token counts and features stable whether sentiment is
expressed as {{:)}}, as an emoji, or as the spelled-out name.
h2. Reconciliation
The runtime fold tables will be reconciled from several sources, so we need
explicit rules:
# *Prefer standard engines over tables.* For anything {{java.text.Normalizer}}
already does (NFC/NFKC/NFD/NFKD), use it for the *text* and do not duplicate
the mapping in a file (the offset-aware question for these is the edit-tracking
gap above, not the mapping itself). The file is for what Java does not give us:
full case folding and our project folds.
# *Provenance on every entry.* Each bundled mapping carries its {{standard}}
tag (above). A build/test step audits the table against the authoritative
upstream file for that standard (for example: every CaseFolding.txt status C
and F row is present and identical) and fails on drift or gaps, which is how we
keep "try to make it complete" honest.
# *Conflict precedence.* If two sources map the same code point differently for
the same fold type, the more specific published standard wins over a general
one, and any deviation we deliberately keep must be re-tagged {{UNSPECIFIED}}
with a note, never left looking standard-backed.
# *Do we need to roll our own?* Mostly no. The open question to settle in this
issue is the exact boundary: confirm that NFC/NFKC and case folding are fully
delegated to standard data, and that {{UNSPECIFIED}} is reserved for the
genuinely project-specific folds (dash target, quote folding,
ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable
skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
h2. Design decisions (from OPENNLP-1850; assumed baseline on main)
The following were settled during OPENNLP-1850 and are already shipped. This
issue builds on them rather than reopening them.
* *Offset-aware contract.* A parallel {{OffsetAwareNormalizer}} interface
(extends {{CharSequenceNormalizer}}, adds {{normalizeAligned()}}).
{{TextNormalizer.Builder.buildAligned()}} composes only rungs that implement it
and throws {{IllegalStateException}} naming the offending rung otherwise.
Callers probe with {{instanceof OffsetAwareNormalizer}} (same pattern as
{{OffsetMappingNameFinder}} in the DL layer).
* *Standard-engine folds in aligned pipelines.* NFC, NFKC, accent fold,
confusable skeleton, and case fold delegate to {{java.text.Normalizer}} or JDK
case mapping and *cannot* report per-character edits. They are excluded from
{{buildAligned()}} in 1850; making them offset-aware is the core work of *this*
issue.
* *Default vs opt-in behavior.* {{TextNormalizer.defaultChain()}} is the
conservative matching chain: strip invisible, NFC, whitespace, quotes, dashes,
case fold, accent fold. DL input folding (whitespace and dashes) is opt-in via
{{InferenceOptions}}. NFKC is not in {{defaultChain()}}. This issue does not
change default normalization behavior; offset-aware capability and
provenance-tagged lookups are additive.
* *Emoji classification.* {{WordType.EMOJI}} is a dedicated category (not
punctuation). Classification precedence is emoji > script >
alphanumeric/numeric. {{Extended_Pictographic.txt}} is a filtered UCD subset
bundled and cursor-parsed like the other 1850 data files.
* *Bundled data format (1850 precedent).* Line-oriented {{.txt}} resources,
cursor-parsed at load time, fail-loud on malformed input. No regex in the parse
path. Provenance columns are not present in 1850 data files; they are
introduced in this issue for CaseFolding and project-specific mappings.
h2. Remaining design work
* * *How NFC/NFKC/full case fold produce an {{Alignment}}.*
{{java.text.Normalizer}} returns only a string. Choose and implement one of:
ICU4J {{Normalizer2}} + {{Edits}}, input/output diff at decomposition
boundaries, or a bundled CaseFolding lookup that generates the alignment during
our own pass. Evaluate ICU dependency vs diff complexity in the first
implementation PR.
* *Provenance-tagged lookup format.* Extend the 1850 {{.txt}} + cursor-parse
pattern with a documented header/schema for {{standard}}, {{fold_type}}, and
{{UNSPECIFIED}} project choices; or introduce a small CSV if columnar
provenance is clearer. One file per fold type for bundled tables (consistent
with {{WordBreakProperty.txt}}, {{confusables.txt}}, etc.).
* *{{UNSPECIFIED}} entries.* Allowed for genuine project choices (dash target,
quote targets, emoji/emoticon mappings with no governing standard). Each must
carry an inline note. Standard-backed rows must audit clean against the
upstream file. No separate sign-off gate in the build for 1850-style project
folds.
* *Emoji/emoticon annotation layer.* Bidirectional emoji↔emoticon rung plus
optional {{Term}} layers (name, sentiment, entity type, category). CLDR
annotations for names where available; sentiment/entity/category sources and
file layout decided in the first annotation PR. Annotations in a separate
lookup file keyed by code point, not mixed into the fold table.
h2. Scope and acceptance criteria
* (DONE in OPENNLP-1850) {{Alignment.andThen}} composition with tests,
including expand-then-contract and contract-then-expand chains.
* (DONE in OPENNLP-1850) The offset-aware contract on {{CharClass}} returning
{{AlignedText}}, with round-trip tests for collapse, supplementary-dash
contraction, supplementary-replacement expansion, and edge deletions.
* An offset-aware mode on the rest of the {{CharSequenceNormalizer}} family and
on {{TextNormalizer}}, returning a composed {{AlignedText}} for a multi-rung
chain (requires resolving the NFC/NFKC edit-tracking question).
* A bundled, provenance-tagged CaseFolding lookup with a completeness audit
test against upstream {{CaseFolding.txt}}.
* Round-trip tests: for representative expanding (ellipsis, eszett, ligature,
fraction) and contracting (run collapse, supplementary dash, NFC compose)
inputs, a position in the normalized text maps back to the correct original
characters through the full pipeline.
* A bidirectional, opt-in emoji/emoticon rung backed by the same
provenance-tagged lookup, offset-aware like every other rung, with round-trip
tests in both directions.
* A small, provenance-tagged annotation layer on the lookup (name, sentiment,
entity type, category), surfaced through the {{Term}} model and wired as
optional input to the Sentiment Detection, Name Finder, and Document
Categorizer components.
* Consistent {{WordType}}-level classification of emoji and emoticons so token
counts and ML feature generators treat them as one class.
h2. Design guidelines and codebase consistency
OPENNLP-1850 merged the normalization engine the rest of this work builds on.
The following are already in place on {{main}} and are assumed by this issue:
* {{CharClass}} / {{CodePointSet}} cursor engine — UCD-sourced {{White_Space}}
and {{Dash}} sets, no regex on text paths
* {{Alignment}} / {{AlignedText}}, {{OffsetAwareNormalizer}},
{{TextNormalizer.buildAligned()}}, and the {{CharClass}} {{*Aligned}} variants
* UAX #29 word tokenizer ({{WordSegmenter}}, {{WordTokenizer}}, {{WordType}})
* {{Term}} / {{TermAnalyzer}} and per-language {{NormalizationProfile(s)}}
* DL integration: Unicode {{White_Space}} chunking, optional input folding,
{{NameFinderDL.findInOriginal()}} via {{Alignment}}
h3. Design principals
Three design goals govern that engine and should govern new normalization work
going forward:
# *Standards-sourced character classes.* UCD properties ({{White_Space}},
{{Dash}}, …), not {{Character.isWhitespace}} or regex character classes like
{{\s}}.
# *Cursor-based text transforms.* Single forward code-point scans for
normalization and token-boundary logic on user text. Regex is fine for config
parsing and other non-user-text utilities.
# *Offset preservation where spans matter.* Length-changing folds compose
through {{Alignment}} so consumers can map normalized positions back to the
original input. That is this issue's job for the standard-engine folds
{{buildAligned()}} rejects today; the {{CharClass}} rungs and DL path already
do it.
The manual (OPENNLP-1850 docs) should describe these as *design goals* the
engine follows and that new code should follow. Do not claim every legacy class
in {{opennlp.tools.util.normalizer}} or all of OpenNLP already meets them —
several pre-existing normalizers still use regex and {{StringUtil}} still uses
JVM whitespace predicates. That gap is tracked below, not a defect in the
merged engine.
Target manual wording ({{normalizer.xml}} intro):
{quote}
The normalization engine follows three design goals: standards-sourced
character classes, cursor-based text transforms, and offset preservation where
spans matter. New normalization code should align with these goals; older
components are migrated on a best-effort basis (tracked in OPENNLP-1852).
{quote}
State the no-regex / UCD rationale once in the normalizer chapter; DL and
tokenizer chapters should xref rather than restate.
h3. Consistency scoreboard
Sub-tasks to file under this epic when ready; until then this table is the
scoreboard:
|| # || Summary || Component || Priority || Status ||
| 1 | Audit {{split("\\s+")}} and {{Character.isWhitespace}} call sites outside
the 1850 engine | codebase-wide | medium | todo |
| 2 | Migrate {{StringUtil}} whitespace to UCD-backed API or document
intentional exception | {{opennlp-api}} | medium | todo |
| 3 | Replace regex in legacy {{TwitterCharSequenceNormalizer}} |
{{opennlp-runtime}} | low | todo |
| 4 | Replace regex in legacy {{UrlCharSequenceNormalizer}} |
{{opennlp-runtime}} | low | todo |
| 5 | Replace regex in legacy {{ShrinkCharSequenceNormalizer}} |
{{opennlp-runtime}} | low | todo |
| 6 | Replace regex in legacy {{NumberCharSequenceNormalizer}} |
{{opennlp-runtime}} | low | todo |
| 7 | Replace regex in legacy {{EmojiCharSequenceNormalizer}} |
{{opennlp-runtime}} | low | todo |
| 8 | Audit spellcheck / extension normalizers for regex-on-text paths |
extensions | low | todo |
| 9 | Soften manual intro to design-goals wording; trim duplicated
no-regex/ReDoS prose across chapters | {{opennlp-docs}} | medium | todo |
| 10 | Add design-guidelines xref in {{normalizer.xml}} pointing here |
{{opennlp-docs}} | low | todo |
When the scoreboard is substantially green, tighten the manual from "design
goals" to stronger statements. Scoreboard items do not block the offset-aware
fold work in this issue.
h2. Out of scope
* * Changing default normalization behavior; this issue adds offset-aware
capability and the provenance model, it does not turn expanding folds on by
default.
* Legacy regex normalizer migration, {{StringUtil}} whitespace standardization,
and manual wording — tracked in the scoreboard above, not acceptance criteria
for this issue.
was:
h2. Summary
Normalization can change string length in both directions, and today only one
corner of the library accounts for it. Canonical and compatibility
decompositions, full case folding, ligature and fraction expansion, and
run-collapsing all change the number of characters between the input and the
normalized output. The {{CharClass}} engine already produces an {{Alignment}}
(carried in an {{AlignedText}}) for its whitespace/dash folds, used by the DL
components to map entity spans back to the original text, but the broader
{{CharSequenceNormalizer}} family returns a plain {{CharSequence}} with no
offset information. The goal of this issue is to make the whole normalization
pipeline offset-aware so that any chain of expanding or contracting rungs can
map a result position (an entity span, a search hit, a highlight) back to
original-document coordinates by construction, not case by case.
This is a follow-up to OPENNLP-1850. That work delivered the offset *model*
this issue builds on (see "Already delivered" below) and made the only
offset-sensitive path that applies length-changing folds today (the DL
whitespace/dash chunking in {{NameFinderDL}}) offset-safe. This issue
generalizes the model across the rest of the normalizer family so it can be
used safely in offset-sensitive contexts.
h2. Already delivered in OPENNLP-1850 (do not re-scope here)
The keystone is in place; this issue should build on it rather than reinvent it.
* {{Alignment}} -- a bidirectional edit-sequence offset model
({{equal}}/{{replace}} runs) that represents expansion, contraction, deletion
(as gaps), and 1:1 changes uniformly, and maps span-to-span with
{{toOriginalSpan}}/{{toNormalizedSpan}}. It replaced the earlier
per-output-character {{OffsetMap}}, which assumed the normalized text
contiguously covered the original and over-covered deletions. ({{OffsetMap}}
and {{NormalizedText}} were never released and are removed.)
* {{Alignment.andThen(Alignment)}} -- associative composition of two stages
into one original-to-final alignment. Built and tested, including
expand-then-contract and contract-then-expand chains.
* The offset-aware *contract* on {{CharClass}}: {{normalizeAligned}},
{{collapseAligned}}, {{collapsePreservingAligned}}, {{trimAligned}},
{{removeAllAligned}}, each returning an {{AlignedText}} (normalized text plus
{{Alignment}}).
* The DL path consuming it: {{AbstractDL.normalizeInputAligned}} and
{{NameFinderDL.findInOriginal}} map decoded spans back via
{{Alignment.toOriginalSpan}}.
So the remaining work is to lift the offset-aware contract from {{CharClass}}
up to the whole {{CharSequenceNormalizer}} family and the {{TextNormalizer}}
pipeline, add the provenance-tagged data, and add the emoji/emoticon rung and
annotation layer.
h2. Background: there is a standard, and it classifies every character for us
We do not need to hand-enumerate "special characters." Unicode already catalogs
which characters expand or contract and to what, so the work is to consume that
data, not invent it.
|| Source standard || What it covers || Direction || Notes ||
| UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD) | Canonical and compatibility
(de)composition | both | Implemented by {{java.text.Normalizer}} -- but see the
edit-tracking caveat under Proposed architecture: it returns a plain string
with no edit information |
| UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX #44)
| Per-character expansions, tagged by type: {{<fraction>}} (1/2),
{{<super>}}/{{<sub>}} (squared to 2), {{<compat>}} (ligature fi, ellipsis to
three dots, Roman numerals), {{<square>}}, {{<wide>}}/{{<narrow>}} | expand |
This is the authoritative expansion catalog and the basis for NFKC |
| CaseFolding.txt (UTS #21 section 5.18) | Case folding, including the full
(expanding) folds with status F: eszett to ss, ligature ff to ff | expand and
1:1 | Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding,
so full folds genuinely need this data |
| Project folds (dash, quote, ellipsis, digit, confusable) | Our own target
choices, some derived from Unicode properties (Dash, White_Space) and some
hand-authored | both | These are the ones with no single governing standard;
see provenance below |
Contractions are the mirror image of the same data: NFC composition (combining
marks fold into one precomposed character), our whitespace and dash
run-collapses, and the UTF-16 case where a supplementary code point (two UTF-16
units) collapses to one unit.
h2. The core idea: one offset model, not a per-character table
Expansion and contraction are the same problem stated twice: the output length
differs from the input length. {{Alignment}} solves it generically. It records
the edits between input and output as a sequence of _equal_ runs (text copied
through, length unchanged) and _replace_ runs (M original characters that
produced P normalized characters), so 1-to-3 (ellipsis), 3-to-1 (run collapse),
2-to-1 (supplementary dash), and N-to-0 (deletion) are all just runs. It maps
span-to-span, so a match that ends next to deleted text reports a tight
original span instead of over-covering the deletion, and it composes with
{{andThen}}. The character data only decides _what_ each fold produces;
{{Alignment}} handles _where everything went_.
h2. Proposed architecture
# *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}}
family an offset-aware mode that returns an {{AlignedText}} (normalized text
plus {{Alignment}}), in addition to the existing {{CharSequence}} method. A
length-preserving rung returns an identity alignment; a length-changing rung
builds a real one. The {{CharClass}}-backed rungs derive it cheaply during the
pass they already make; the standard-engine rungs are the hard case (next
point).
# *Map composition.* {{Alignment.andThen}} -- *already delivered*. A pipeline
of N rungs collapses to a single original-to-final alignment by folding
{{andThen}} over the stages.
# *Offset-aware pipeline.* Add an offset-aware build/apply mode to
{{TextNormalizer}} that runs the configured rungs in order, composes their
alignments, and returns one {{AlignedText}} for the whole chain.
# *Reuse the standard engines where possible -- with a known edit-tracking
gap.* NFC/NFD/NFKC/NFKD and full case folding are the value of delegating to
standards, but {{java.text.Normalizer.normalize(s, form)}} returns only the
result string with *no edit information*, so an {{Alignment}} cannot be read
out of it directly. The realistic options, to be decided in design review: (a)
depend on ICU4J's {{Normalizer2}}, which can emit an {{Edits}} object that maps
directly onto our {{equal}}/{{replace}} runs; (b) compute the alignment by
diffing input against output at decomposition boundaries; or (c) restrict the
offset-aware path to folds we generate ourselves (the {{CharClass}} family plus
a bundled-table case fold) and treat {{java.text.Normalizer}}-only NFC/NFKC as
offset-opaque. This is the single biggest design decision in the issue and
should be settled first.
The {{CharClass}} {{*Aligned}} family and the DL {{normalizeInputAligned}} are
the first instances of the offset-aware contract; this issue lifts the pattern
up to the whole family.
h2. Data files and provenance metadata
We will bundle a CaseFolding lookup (sourced from the Unicode
{{CaseFolding.txt}}) and aim for completeness against the upstream file. The
key addition is *provenance metadata*: every mapping records which standard it
follows, so we can audit coverage, distinguish standard-mandated folds from
project choices, and re-derive the table when Unicode revises.
Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a
documented header schema) with columns roughly:
{noformat}
source ; target ; fold_type ; standard ;
sentiment ; unicode_version ; notes
00DF ; 0073 0073 ; CASE_FOLD ; UTS21:CaseFolding:F ;
; 17.0.0 ; sharp s -> ss
2026 ; 002E 002E 002E ; NFKC ; UAX44:Decomposition:compat ;
; 17.0.0 ; horizontal ellipsis
2014 2013 ; 002D ; DASH ; UNSPECIFIED ;
; - ; dash-to-hyphen target choice
1F600 ; 003A 0029 ; EMOJI ; CLDR:annotation ;
positive ; 17.0.0 ; grinning face -> :) (expansion)
003A 0029 ; 1F642 ; EMOTICON ; UNSPECIFIED ;
positive ; - ; :) -> slightly smiling face (contraction)
{noformat}
The {{standard}} field is the important one. Suggested values:
* {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form folds
* {{UAX44:Decomposition:<type>}} for a specific decomposition type
* {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S
(simple), T (Turkic)
* {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a
published standard." This is the explicit "we just did that" marker the team
wants, so a reviewer or auditor immediately knows the entry is a project
decision (for example our choice of ASCII hyphen-minus as the dash target, or
expanding the ellipsis ourselves rather than relying on NFKC).
h2. Emoji and emoticons, both directions, sentiment, and token classification
Emoji-to-emoticon and emoticon-to-emoji are exactly the bidirectional,
length-changing case this issue is built for: an emoji rendered as {{:)}} is an
expansion, the reverse a contraction, and a supplementary-plane emoji is also
the two-UTF-16-units-collapsing case. Both directions are in scope, riding the
same offset-aware rung and lookup table. There is no governing standard for the
mapping itself (emoticons are informal and the relation is many-to-many and
culturally variable), so most rows are tagged {{UNSPECIFIED}}; where a
direction can be anchored on published data we do (for example CLDR emoji
annotations for the emoji short-name) and tag it accordingly.
The bigger payoff is turning an opaque pictograph into *lexical signal*. Beyond
the emoticon form, the same lookup can carry the emoji's actual text, its CLDR
short-name (for example "grinning face"), so the rung can surface a real word
sequence instead of an out-of-vocabulary symbol that most models simply drop.
That text becomes the input that sentiment analysis, semantic search, embedding
pipelines, and LLM-style consumers actually read and reason about, which is
where this helps downstream understanding the most. The {{Term}} model is the
natural carrier: one token can expose {{original}} (the emoji), a folded
{{emoticon}}, a human-readable {{name}}, and a {{sentiment}} tag as parallel
layers, and each consumer picks the layer it needs (a sentiment lexicon match,
the name text for an embedding, or the original for display). Sentiment carries
its own provenance via a {{sentiment_source}} field (an emoji sentiment ranking
dataset, a project lexicon, or {{UNSPECIFIED}}), kept separate from the
mapping's {{standard}} tag so each is independently auditable and replaceable.
The same annotation mechanism generalizes well beyond sentiment. An emoji's
name is frequently an entity or concept in its own right (an airplane, a tower,
the Statue of Liberty, a national flag), so surfacing that name plus an
optional coarse entity-type tag (for example VEHICLE, LANDMARK, ANIMAL, FLAG,
FOOD) gives the Name Finder real lexical content and a gazetteer-like signal
where it would otherwise see an unknown glyph. The same row can also carry a
document-category hint (a pizza or an airplane is a strong cue for a food or a
travel document), feeding Document Categorizer features. So the lookup is less
a single sentiment column and more a small, provenance-tagged annotation layer
per symbol: name, sentiment, entity type, and category, each optional and each
with its own source, surfaced through the {{Term}} model and consumed by the
Sentiment Detection, Name Finder, and Doccat components respectively. Each
annotation dimension is opt-in, so a consumer that only wants the folded text
pays nothing for the rest.
Token classification ties in too. The UAX #29 tokenizer already gives emoji
their own {{WordType}} category, so the same data lets downstream counting and
feature generation treat emoji and emoticons as a consistent class rather than
stray symbols, for example folding them into a punctuation-like bucket or a
dedicated emoji feature for corpus statistics and the ML feature generators.
That keeps a document's token counts and features stable whether sentiment is
expressed as {{:)}}, as an emoji, or as the spelled-out name.
h2. Reconciliation
The runtime fold tables will be reconciled from several sources, so we need
explicit rules:
# *Prefer standard engines over tables.* For anything {{java.text.Normalizer}}
already does (NFC/NFKC/NFD/NFKD), use it for the *text* and do not duplicate
the mapping in a file (the offset-aware question for these is the edit-tracking
gap above, not the mapping itself). The file is for what Java does not give us:
full case folding and our project folds.
# *Provenance on every entry.* Each bundled mapping carries its {{standard}}
tag (above). A build/test step audits the table against the authoritative
upstream file for that standard (for example: every CaseFolding.txt status C
and F row is present and identical) and fails on drift or gaps, which is how we
keep "try to make it complete" honest.
# *Conflict precedence.* If two sources map the same code point differently for
the same fold type, the more specific published standard wins over a general
one, and any deviation we deliberately keep must be re-tagged {{UNSPECIFIED}}
with a note, never left looking standard-backed.
# *Do we need to roll our own?* Mostly no. The open question to settle in this
issue is the exact boundary: confirm that NFC/NFKC and case folding are fully
delegated to standard data, and that {{UNSPECIFIED}} is reserved for the
genuinely project-specific folds (dash target, quote folding,
ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable
skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
h2. Open questions for the design review
* *How NFC/NFKC/case-fold rungs produce an {{Alignment}}* given
{{java.text.Normalizer}} exposes no edits: ICU4J {{Normalizer2}}/{{Edits}}, an
input/output diff, or leaving those folds offset-opaque. (The decision that
gates everything else.)
* CSV vs annotated {{.txt}} for the lookup, and whether one combined file or
one file per fold type.
* Whether the offset-aware method lives on {{CharSequenceNormalizer}} directly
(every rung implements it) or on a parallel interface that the pipeline adapts.
* How aggressively to expand by default: full case folding (eszett to ss) and
NFKC change tokens visibly, so they should likely stay opt-in rungs, not
defaults.
* Whether {{UNSPECIFIED}} entries should be allowed in a release build at all,
or must each be reviewed and signed off.
* Which dataset anchors each annotation dimension (an emoji sentiment ranking
for sentiment, CLDR for names, and what for entity type and category), and
whether the annotations live in the same lookup or a separate file keyed by
code point.
* Whether emoji and emoticons classify as punctuation or as a dedicated emoji
{{WordType}} for counting and feature-generation purposes.
h2. Scope and acceptance criteria
* (DONE in OPENNLP-1850) {{Alignment.andThen}} composition with tests,
including expand-then-contract and contract-then-expand chains.
* (DONE in OPENNLP-1850) The offset-aware contract on {{CharClass}} returning
{{AlignedText}}, with round-trip tests for collapse, supplementary-dash
contraction, supplementary-replacement expansion, and edge deletions.
* An offset-aware mode on the rest of the {{CharSequenceNormalizer}} family and
on {{TextNormalizer}}, returning a composed {{AlignedText}} for a multi-rung
chain (requires resolving the NFC/NFKC edit-tracking question).
* A bundled, provenance-tagged CaseFolding lookup with a completeness audit
test against upstream {{CaseFolding.txt}}.
* Round-trip tests: for representative expanding (ellipsis, eszett, ligature,
fraction) and contracting (run collapse, supplementary dash, NFC compose)
inputs, a position in the normalized text maps back to the correct original
characters through the full pipeline.
* A bidirectional, opt-in emoji/emoticon rung backed by the same
provenance-tagged lookup, offset-aware like every other rung, with round-trip
tests in both directions.
* A small, provenance-tagged annotation layer on the lookup (name, sentiment,
entity type, category), surfaced through the {{Term}} model and wired as
optional input to the Sentiment Detection, Name Finder, and Document
Categorizer components.
* Consistent {{WordType}}-level classification of emoji and emoticons so token
counts and ML feature generators treat them as one class.
h2. Out of scope
* The DL whitespace/dash offset-safety, already handled under OPENNLP-1850
({{NameFinderDL.findInOriginal}} plus {{AbstractDL.normalizeInputAligned}}).
* The {{Alignment}} model and {{andThen}} composition themselves, delivered
under OPENNLP-1850; this issue consumes them.
* Changing any default normalization behavior; this issue adds the offset-aware
capability and the provenance model, it does not turn expanding folds on by
default.
> Offset-aware normalization pipeline: handle length-changing folds (expansion
> and contraction) with a single offset model
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1852
> URL: https://issues.apache.org/jira/browse/OPENNLP-1852
> Project: OpenNLP
> Issue Type: Epic
> Components: Tokenizer
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
>
> h2. Summary
> Normalization can change string length in both directions, and today only one
> corner of the library accounts for it. Canonical and compatibility
> decompositions, full case folding, ligature and fraction expansion, and
> run-collapsing all change the number of characters between the input and the
> normalized output. The {{CharClass}} engine already produces an {{Alignment}}
> (carried in an {{AlignedText}}) for its whitespace/dash folds, used by the DL
> components to map entity spans back to the original text, but the broader
> {{CharSequenceNormalizer}} family returns a plain {{CharSequence}} with no
> offset information. The goal of this issue is to make the whole normalization
> pipeline offset-aware so that any chain of expanding or contracting rungs can
> map a result position (an entity span, a search hit, a highlight) back to
> original-document coordinates by construction, not case by case.
> This is a follow-up to OPENNLP-1850. That work delivered the offset *model*
> this issue builds on (see "Already delivered" below) and made the only
> offset-sensitive path that applies length-changing folds today (the DL
> whitespace/dash chunking in {{NameFinderDL}}) offset-safe. This issue
> generalizes the model across the rest of the normalizer family so it can be
> used safely in offset-sensitive contexts.
> h2. Already delivered in OPENNLP-1850 (do not re-scope here)
> The keystone is in place; this issue should build on it rather than reinvent
> it.
> * {{Alignment}} -- a bidirectional edit-sequence offset model
> ({{equal}}/{{replace}} runs) that represents expansion, contraction, deletion
> (as gaps), and 1:1 changes uniformly, and maps span-to-span with
> {{toOriginalSpan}}/{{toNormalizedSpan}}. It replaced the earlier
> per-output-character {{OffsetMap}}, which assumed the normalized text
> contiguously covered the original and over-covered deletions. ({{OffsetMap}}
> and {{NormalizedText}} were never released and are removed.)
> * {{Alignment.andThen(Alignment)}} -- associative composition of two stages
> into one original-to-final alignment. Built and tested, including
> expand-then-contract and contract-then-expand chains.
> * The offset-aware *contract* on {{CharClass}}: {{normalizeAligned}},
> {{collapseAligned}}, {{collapsePreservingAligned}}, {{trimAligned}},
> {{removeAllAligned}}, each returning an {{AlignedText}} (normalized text plus
> {{Alignment}}).
> * The DL path consuming it: {{AbstractDL.normalizeInputAligned}} and
> {{NameFinderDL.findInOriginal}} map decoded spans back via
> {{Alignment.toOriginalSpan}}.
> So the remaining work is to lift the offset-aware contract from {{CharClass}}
> up to the whole {{CharSequenceNormalizer}} family and the {{TextNormalizer}}
> pipeline, add the provenance-tagged data, and add the emoji/emoticon rung and
> annotation layer.
> h2. Background: there is a standard, and it classifies every character for us
> We do not need to hand-enumerate "special characters." Unicode already
> catalogs which characters expand or contract and to what, so the work is to
> consume that data, not invent it.
> || Source standard || What it covers || Direction || Notes ||
> | UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD) | Canonical and
> compatibility (de)composition | both | Implemented by
> {{java.text.Normalizer}} -- but see the edit-tracking caveat under Proposed
> architecture: it returns a plain string with no edit information |
> | UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX
> #44) | Per-character expansions, tagged by type: {{<fraction>}} (1/2),
> {{<super>}}/{{<sub>}} (squared to 2), {{<compat>}} (ligature fi, ellipsis to
> three dots, Roman numerals), {{<square>}}, {{<wide>}}/{{<narrow>}} | expand |
> This is the authoritative expansion catalog and the basis for NFKC |
> | CaseFolding.txt (UTS #21 section 5.18) | Case folding, including the full
> (expanding) folds with status F: eszett to ss, ligature ff to ff | expand and
> 1:1 | Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding,
> so full folds genuinely need this data |
> | Project folds (dash, quote, ellipsis, digit, confusable) | Our own target
> choices, some derived from Unicode properties (Dash, White_Space) and some
> hand-authored | both | These are the ones with no single governing standard;
> see provenance below |
> Contractions are the mirror image of the same data: NFC composition
> (combining marks fold into one precomposed character), our whitespace and
> dash run-collapses, and the UTF-16 case where a supplementary code point (two
> UTF-16 units) collapses to one unit.
> h2. The core idea: one offset model, not a per-character table
> Expansion and contraction are the same problem stated twice: the output
> length differs from the input length. {{Alignment}} solves it generically. It
> records the edits between input and output as a sequence of _equal_ runs
> (text copied through, length unchanged) and _replace_ runs (M original
> characters that produced P normalized characters), so 1-to-3 (ellipsis),
> 3-to-1 (run collapse), 2-to-1 (supplementary dash), and N-to-0 (deletion) are
> all just runs. It maps span-to-span, so a match that ends next to deleted
> text reports a tight original span instead of over-covering the deletion, and
> it composes with {{andThen}}. The character data only decides _what_ each
> fold produces; {{Alignment}} handles _where everything went_.
> h2. Proposed architecture
> # *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}}
> family an offset-aware mode that returns an {{AlignedText}} (normalized text
> plus {{Alignment}}), in addition to the existing {{CharSequence}} method. A
> length-preserving rung returns an identity alignment; a length-changing rung
> builds a real one. The {{CharClass}}-backed rungs derive it cheaply during
> the pass they already make; the standard-engine rungs are the hard case (next
> point).
> # *Map composition.* {{Alignment.andThen}} -- *already delivered*. A pipeline
> of N rungs collapses to a single original-to-final alignment by folding
> {{andThen}} over the stages.
> # *Offset-aware pipeline.* Add an offset-aware build/apply mode to
> {{TextNormalizer}} that runs the configured rungs in order, composes their
> alignments, and returns one {{AlignedText}} for the whole chain.
> # *Reuse the standard engines where possible -- with a known edit-tracking
> gap.* NFC/NFD/NFKC/NFKD and full case folding are the value of delegating to
> standards, but {{java.text.Normalizer.normalize(s, form)}} returns only the
> result string with *no edit information*, so an {{Alignment}} cannot be read
> out of it directly. The realistic options, to be decided in design review:
> (a) depend on ICU4J's {{Normalizer2}}, which can emit an {{Edits}} object
> that maps directly onto our {{equal}}/{{replace}} runs; (b) compute the
> alignment by diffing input against output at decomposition boundaries; or (c)
> restrict the offset-aware path to folds we generate ourselves (the
> {{CharClass}} family plus a bundled-table case fold) and treat
> {{java.text.Normalizer}}-only NFC/NFKC as offset-opaque. This is the single
> biggest design decision in the issue and should be settled first.
> The {{CharClass}} {{*Aligned}} family and the DL {{normalizeInputAligned}}
> are the first instances of the offset-aware contract; this issue lifts the
> pattern up to the whole family.
> h2. Data files and provenance metadata
> We will bundle a CaseFolding lookup (sourced from the Unicode
> {{CaseFolding.txt}}) and aim for completeness against the upstream file. The
> key addition is *provenance metadata*: every mapping records which standard
> it follows, so we can audit coverage, distinguish standard-mandated folds
> from project choices, and re-derive the table when Unicode revises.
> Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a
> documented header schema) with columns roughly:
> {noformat}
> source ; target ; fold_type ; standard ;
> sentiment ; unicode_version ; notes
> 00DF ; 0073 0073 ; CASE_FOLD ; UTS21:CaseFolding:F ;
> ; 17.0.0 ; sharp s -> ss
> 2026 ; 002E 002E 002E ; NFKC ; UAX44:Decomposition:compat ;
> ; 17.0.0 ; horizontal ellipsis
> 2014 2013 ; 002D ; DASH ; UNSPECIFIED ;
> ; - ; dash-to-hyphen target choice
> 1F600 ; 003A 0029 ; EMOJI ; CLDR:annotation ;
> positive ; 17.0.0 ; grinning face -> :) (expansion)
> 003A 0029 ; 1F642 ; EMOTICON ; UNSPECIFIED ;
> positive ; - ; :) -> slightly smiling face (contraction)
> {noformat}
> The {{standard}} field is the important one. Suggested values:
> * {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form
> folds
> * {{UAX44:Decomposition:<type>}} for a specific decomposition type
> * {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S
> (simple), T (Turkic)
> * {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a
> published standard." This is the explicit "we just did that" marker the team
> wants, so a reviewer or auditor immediately knows the entry is a project
> decision (for example our choice of ASCII hyphen-minus as the dash target, or
> expanding the ellipsis ourselves rather than relying on NFKC).
> h2. Emoji and emoticons, both directions, sentiment, and token classification
> Emoji-to-emoticon and emoticon-to-emoji are exactly the bidirectional,
> length-changing case this issue is built for: an emoji rendered as {{:)}} is
> an expansion, the reverse a contraction, and a supplementary-plane emoji is
> also the two-UTF-16-units-collapsing case. Both directions are in scope,
> riding the same offset-aware rung and lookup table. There is no governing
> standard for the mapping itself (emoticons are informal and the relation is
> many-to-many and culturally variable), so most rows are tagged
> {{UNSPECIFIED}}; where a direction can be anchored on published data we do
> (for example CLDR emoji annotations for the emoji short-name) and tag it
> accordingly.
> The bigger payoff is turning an opaque pictograph into *lexical signal*.
> Beyond the emoticon form, the same lookup can carry the emoji's actual text,
> its CLDR short-name (for example "grinning face"), so the rung can surface a
> real word sequence instead of an out-of-vocabulary symbol that most models
> simply drop. That text becomes the input that sentiment analysis, semantic
> search, embedding pipelines, and LLM-style consumers actually read and reason
> about, which is where this helps downstream understanding the most. The
> {{Term}} model is the natural carrier: one token can expose {{original}} (the
> emoji), a folded {{emoticon}}, a human-readable {{name}}, and a {{sentiment}}
> tag as parallel layers, and each consumer picks the layer it needs (a
> sentiment lexicon match, the name text for an embedding, or the original for
> display). Sentiment carries its own provenance via a {{sentiment_source}}
> field (an emoji sentiment ranking dataset, a project lexicon, or
> {{UNSPECIFIED}}), kept separate from the mapping's {{standard}} tag so each
> is independently auditable and replaceable.
> The same annotation mechanism generalizes well beyond sentiment. An emoji's
> name is frequently an entity or concept in its own right (an airplane, a
> tower, the Statue of Liberty, a national flag), so surfacing that name plus
> an optional coarse entity-type tag (for example VEHICLE, LANDMARK, ANIMAL,
> FLAG, FOOD) gives the Name Finder real lexical content and a gazetteer-like
> signal where it would otherwise see an unknown glyph. The same row can also
> carry a document-category hint (a pizza or an airplane is a strong cue for a
> food or a travel document), feeding Document Categorizer features. So the
> lookup is less a single sentiment column and more a small, provenance-tagged
> annotation layer per symbol: name, sentiment, entity type, and category, each
> optional and each with its own source, surfaced through the {{Term}} model
> and consumed by the Sentiment Detection, Name Finder, and Doccat components
> respectively. Each annotation dimension is opt-in, so a consumer that only
> wants the folded text pays nothing for the rest.
> Token classification ties in too. The UAX #29 tokenizer already gives emoji
> their own {{WordType}} category, so the same data lets downstream counting
> and feature generation treat emoji and emoticons as a consistent class rather
> than stray symbols, for example folding them into a punctuation-like bucket
> or a dedicated emoji feature for corpus statistics and the ML feature
> generators. That keeps a document's token counts and features stable whether
> sentiment is expressed as {{:)}}, as an emoji, or as the spelled-out name.
> h2. Reconciliation
> The runtime fold tables will be reconciled from several sources, so we need
> explicit rules:
> # *Prefer standard engines over tables.* For anything
> {{java.text.Normalizer}} already does (NFC/NFKC/NFD/NFKD), use it for the
> *text* and do not duplicate the mapping in a file (the offset-aware question
> for these is the edit-tracking gap above, not the mapping itself). The file
> is for what Java does not give us: full case folding and our project folds.
> # *Provenance on every entry.* Each bundled mapping carries its {{standard}}
> tag (above). A build/test step audits the table against the authoritative
> upstream file for that standard (for example: every CaseFolding.txt status C
> and F row is present and identical) and fails on drift or gaps, which is how
> we keep "try to make it complete" honest.
> # *Conflict precedence.* If two sources map the same code point differently
> for the same fold type, the more specific published standard wins over a
> general one, and any deviation we deliberately keep must be re-tagged
> {{UNSPECIFIED}} with a note, never left looking standard-backed.
> # *Do we need to roll our own?* Mostly no. The open question to settle in
> this issue is the exact boundary: confirm that NFC/NFKC and case folding are
> fully delegated to standard data, and that {{UNSPECIFIED}} is reserved for
> the genuinely project-specific folds (dash target, quote folding,
> ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable
> skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
> h2. Design decisions (from OPENNLP-1850; assumed baseline on main)
> The following were settled during OPENNLP-1850 and are already shipped. This
> issue builds on them rather than reopening them.
> * *Offset-aware contract.* A parallel {{OffsetAwareNormalizer}} interface
> (extends {{CharSequenceNormalizer}}, adds {{normalizeAligned()}}).
> {{TextNormalizer.Builder.buildAligned()}} composes only rungs that implement
> it and throws {{IllegalStateException}} naming the offending rung otherwise.
> Callers probe with {{instanceof OffsetAwareNormalizer}} (same pattern as
> {{OffsetMappingNameFinder}} in the DL layer).
> * *Standard-engine folds in aligned pipelines.* NFC, NFKC, accent fold,
> confusable skeleton, and case fold delegate to {{java.text.Normalizer}} or
> JDK case mapping and *cannot* report per-character edits. They are excluded
> from {{buildAligned()}} in 1850; making them offset-aware is the core work of
> *this* issue.
> * *Default vs opt-in behavior.* {{TextNormalizer.defaultChain()}} is the
> conservative matching chain: strip invisible, NFC, whitespace, quotes,
> dashes, case fold, accent fold. DL input folding (whitespace and dashes) is
> opt-in via {{InferenceOptions}}. NFKC is not in {{defaultChain()}}. This
> issue does not change default normalization behavior; offset-aware capability
> and provenance-tagged lookups are additive.
> * *Emoji classification.* {{WordType.EMOJI}} is a dedicated category (not
> punctuation). Classification precedence is emoji > script >
> alphanumeric/numeric. {{Extended_Pictographic.txt}} is a filtered UCD subset
> bundled and cursor-parsed like the other 1850 data files.
> * *Bundled data format (1850 precedent).* Line-oriented {{.txt}} resources,
> cursor-parsed at load time, fail-loud on malformed input. No regex in the
> parse path. Provenance columns are not present in 1850 data files; they are
> introduced in this issue for CaseFolding and project-specific mappings.
> h2. Remaining design work
> * * *How NFC/NFKC/full case fold produce an {{Alignment}}.*
> {{java.text.Normalizer}} returns only a string. Choose and implement one of:
> ICU4J {{Normalizer2}} + {{Edits}}, input/output diff at decomposition
> boundaries, or a bundled CaseFolding lookup that generates the alignment
> during our own pass. Evaluate ICU dependency vs diff complexity in the first
> implementation PR.
> * *Provenance-tagged lookup format.* Extend the 1850 {{.txt}} + cursor-parse
> pattern with a documented header/schema for {{standard}}, {{fold_type}}, and
> {{UNSPECIFIED}} project choices; or introduce a small CSV if columnar
> provenance is clearer. One file per fold type for bundled tables (consistent
> with {{WordBreakProperty.txt}}, {{confusables.txt}}, etc.).
> * *{{UNSPECIFIED}} entries.* Allowed for genuine project choices (dash
> target, quote targets, emoji/emoticon mappings with no governing standard).
> Each must carry an inline note. Standard-backed rows must audit clean against
> the upstream file. No separate sign-off gate in the build for 1850-style
> project folds.
> * *Emoji/emoticon annotation layer.* Bidirectional emoji↔emoticon rung plus
> optional {{Term}} layers (name, sentiment, entity type, category). CLDR
> annotations for names where available; sentiment/entity/category sources and
> file layout decided in the first annotation PR. Annotations in a separate
> lookup file keyed by code point, not mixed into the fold table.
> h2. Scope and acceptance criteria
> * (DONE in OPENNLP-1850) {{Alignment.andThen}} composition with tests,
> including expand-then-contract and contract-then-expand chains.
> * (DONE in OPENNLP-1850) The offset-aware contract on {{CharClass}} returning
> {{AlignedText}}, with round-trip tests for collapse, supplementary-dash
> contraction, supplementary-replacement expansion, and edge deletions.
> * An offset-aware mode on the rest of the {{CharSequenceNormalizer}} family
> and on {{TextNormalizer}}, returning a composed {{AlignedText}} for a
> multi-rung chain (requires resolving the NFC/NFKC edit-tracking question).
> * A bundled, provenance-tagged CaseFolding lookup with a completeness audit
> test against upstream {{CaseFolding.txt}}.
> * Round-trip tests: for representative expanding (ellipsis, eszett, ligature,
> fraction) and contracting (run collapse, supplementary dash, NFC compose)
> inputs, a position in the normalized text maps back to the correct original
> characters through the full pipeline.
> * A bidirectional, opt-in emoji/emoticon rung backed by the same
> provenance-tagged lookup, offset-aware like every other rung, with round-trip
> tests in both directions.
> * A small, provenance-tagged annotation layer on the lookup (name, sentiment,
> entity type, category), surfaced through the {{Term}} model and wired as
> optional input to the Sentiment Detection, Name Finder, and Document
> Categorizer components.
> * Consistent {{WordType}}-level classification of emoji and emoticons so
> token counts and ML feature generators treat them as one class.
> h2. Design guidelines and codebase consistency
> OPENNLP-1850 merged the normalization engine the rest of this work builds on.
> The following are already in place on {{main}} and are assumed by this issue:
> * {{CharClass}} / {{CodePointSet}} cursor engine — UCD-sourced
> {{White_Space}} and {{Dash}} sets, no regex on text paths
> * {{Alignment}} / {{AlignedText}}, {{OffsetAwareNormalizer}},
> {{TextNormalizer.buildAligned()}}, and the {{CharClass}} {{*Aligned}} variants
> * UAX #29 word tokenizer ({{WordSegmenter}}, {{WordTokenizer}}, {{WordType}})
> * {{Term}} / {{TermAnalyzer}} and per-language {{NormalizationProfile(s)}}
> * DL integration: Unicode {{White_Space}} chunking, optional input folding,
> {{NameFinderDL.findInOriginal()}} via {{Alignment}}
> h3. Design principals
> Three design goals govern that engine and should govern new normalization
> work going forward:
> # *Standards-sourced character classes.* UCD properties ({{White_Space}},
> {{Dash}}, …), not {{Character.isWhitespace}} or regex character classes like
> {{\s}}.
> # *Cursor-based text transforms.* Single forward code-point scans for
> normalization and token-boundary logic on user text. Regex is fine for config
> parsing and other non-user-text utilities.
> # *Offset preservation where spans matter.* Length-changing folds compose
> through {{Alignment}} so consumers can map normalized positions back to the
> original input. That is this issue's job for the standard-engine folds
> {{buildAligned()}} rejects today; the {{CharClass}} rungs and DL path already
> do it.
> The manual (OPENNLP-1850 docs) should describe these as *design goals* the
> engine follows and that new code should follow. Do not claim every legacy
> class in {{opennlp.tools.util.normalizer}} or all of OpenNLP already meets
> them — several pre-existing normalizers still use regex and {{StringUtil}}
> still uses JVM whitespace predicates. That gap is tracked below, not a defect
> in the merged engine.
> Target manual wording ({{normalizer.xml}} intro):
> {quote}
> The normalization engine follows three design goals: standards-sourced
> character classes, cursor-based text transforms, and offset preservation
> where spans matter. New normalization code should align with these goals;
> older components are migrated on a best-effort basis (tracked in
> OPENNLP-1852).
> {quote}
> State the no-regex / UCD rationale once in the normalizer chapter; DL and
> tokenizer chapters should xref rather than restate.
> h3. Consistency scoreboard
> Sub-tasks to file under this epic when ready; until then this table is the
> scoreboard:
> || # || Summary || Component || Priority || Status ||
> | 1 | Audit {{split("\\s+")}} and {{Character.isWhitespace}} call sites
> outside the 1850 engine | codebase-wide | medium | todo |
> | 2 | Migrate {{StringUtil}} whitespace to UCD-backed API or document
> intentional exception | {{opennlp-api}} | medium | todo |
> | 3 | Replace regex in legacy {{TwitterCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 4 | Replace regex in legacy {{UrlCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 5 | Replace regex in legacy {{ShrinkCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 6 | Replace regex in legacy {{NumberCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 7 | Replace regex in legacy {{EmojiCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 8 | Audit spellcheck / extension normalizers for regex-on-text paths |
> extensions | low | todo |
> | 9 | Soften manual intro to design-goals wording; trim duplicated
> no-regex/ReDoS prose across chapters | {{opennlp-docs}} | medium | todo |
> | 10 | Add design-guidelines xref in {{normalizer.xml}} pointing here |
> {{opennlp-docs}} | low | todo |
> When the scoreboard is substantially green, tighten the manual from "design
> goals" to stronger statements. Scoreboard items do not block the offset-aware
> fold work in this issue.
> h2. Out of scope
> * * Changing default normalization behavior; this issue adds offset-aware
> capability and the provenance model, it does not turn expanding folds on by
> default.
> * Legacy regex normalizer migration, {{StringUtil}} whitespace
> standardization, and manual wording — tracked in the scoreboard above, not
> acceptance criteria for this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
