[
https://issues.apache.org/jira/browse/OPENNLP-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner updated OPENNLP-1852:
------------------------------------
Fix Version/s: 3.0.0
> Offset-aware normalization pipeline: handle length-changing folds (expansion
> and contraction) with a single offset model
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1852
> URL: https://issues.apache.org/jira/browse/OPENNLP-1852
> Project: OpenNLP
> Issue Type: Epic
> Components: Tokenizer
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
>
> h2. Summary
> Normalization can change string length in both directions, and today only one
> corner of the library accounts for it. Canonical and compatibility
> decompositions, full case folding, ligature and fraction expansion, and
> run-collapsing all change the number of characters between the input and the
> normalized output. The {{CharClass}} engine already produces an {{Alignment}}
> (carried in an {{AlignedText}}) for its whitespace/dash folds, used by the DL
> components to map entity spans back to the original text, but the broader
> {{CharSequenceNormalizer}} family returns a plain {{CharSequence}} with no
> offset information. The goal of this issue is to make the whole normalization
> pipeline offset-aware so that any chain of expanding or contracting rungs can
> map a result position (an entity span, a search hit, a highlight) back to
> original-document coordinates by construction, not case by case.
> This is a follow-up to OPENNLP-1850. That work delivered the offset *model*
> this issue builds on (see "Already delivered" below) and made the only
> offset-sensitive path that applies length-changing folds today (the DL
> whitespace/dash chunking in {{NameFinderDL}}) offset-safe. This issue
> generalizes the model across the rest of the normalizer family so it can be
> used safely in offset-sensitive contexts.
> h2. Already delivered in OPENNLP-1850 (do not re-scope here)
> The keystone is in place; this issue should build on it rather than reinvent
> it.
> * {{Alignment}} -- a bidirectional edit-sequence offset model
> ({{equal}}/{{replace}} runs) that represents expansion, contraction, deletion
> (as gaps), and 1:1 changes uniformly, and maps span-to-span with
> {{toOriginalSpan}}/{{toNormalizedSpan}}. It replaced the earlier
> per-output-character {{OffsetMap}}, which assumed the normalized text
> contiguously covered the original and over-covered deletions. ({{OffsetMap}}
> and {{NormalizedText}} were never released and are removed.)
> * {{Alignment.andThen(Alignment)}} -- associative composition of two stages
> into one original-to-final alignment. Built and tested, including
> expand-then-contract and contract-then-expand chains.
> * The offset-aware *contract* on {{CharClass}}: {{normalizeAligned}},
> {{collapseAligned}}, {{collapsePreservingAligned}}, {{trimAligned}},
> {{removeAllAligned}}, each returning an {{AlignedText}} (normalized text plus
> {{Alignment}}).
> * The DL path consuming it: {{AbstractDL.normalizeInputAligned}} and
> {{NameFinderDL.findInOriginal}} map decoded spans back via
> {{Alignment.toOriginalSpan}}.
> So the remaining work is to lift the offset-aware contract from {{CharClass}}
> up to the whole {{CharSequenceNormalizer}} family and the {{TextNormalizer}}
> pipeline, add the provenance-tagged data, and add the emoji/emoticon rung and
> annotation layer.
> h2. Background: there is a standard, and it classifies every character for us
> We do not need to hand-enumerate "special characters." Unicode already
> catalogs which characters expand or contract and to what, so the work is to
> consume that data, not invent it.
> || Source standard || What it covers || Direction || Notes ||
> | UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD) | Canonical and
> compatibility (de)composition | both | Implemented by
> {{java.text.Normalizer}} -- but see the edit-tracking caveat under Proposed
> architecture: it returns a plain string with no edit information |
> | UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX
> #44) | Per-character expansions, tagged by type: {{<fraction>}} (1/2),
> {{<super>}}/{{<sub>}} (squared to 2), {{<compat>}} (ligature fi, ellipsis to
> three dots, Roman numerals), {{<square>}}, {{<wide>}}/{{<narrow>}} | expand |
> This is the authoritative expansion catalog and the basis for NFKC |
> | CaseFolding.txt (UTS #21 section 5.18) | Case folding, including the full
> (expanding) folds with status F: eszett to ss, ligature ff to ff | expand and
> 1:1 | Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding,
> so full folds genuinely need this data |
> | Project folds (dash, quote, ellipsis, digit, confusable) | Our own target
> choices, some derived from Unicode properties (Dash, White_Space) and some
> hand-authored | both | These are the ones with no single governing standard;
> see provenance below |
> Contractions are the mirror image of the same data: NFC composition
> (combining marks fold into one precomposed character), our whitespace and
> dash run-collapses, and the UTF-16 case where a supplementary code point (two
> UTF-16 units) collapses to one unit.
> h2. The core idea: one offset model, not a per-character table
> Expansion and contraction are the same problem stated twice: the output
> length differs from the input length. {{Alignment}} solves it generically. It
> records the edits between input and output as a sequence of _equal_ runs
> (text copied through, length unchanged) and _replace_ runs (M original
> characters that produced P normalized characters), so 1-to-3 (ellipsis),
> 3-to-1 (run collapse), 2-to-1 (supplementary dash), and N-to-0 (deletion) are
> all just runs. It maps span-to-span, so a match that ends next to deleted
> text reports a tight original span instead of over-covering the deletion, and
> it composes with {{andThen}}. The character data only decides _what_ each
> fold produces; {{Alignment}} handles _where everything went_.
> h2. Proposed architecture
> # *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}}
> family an offset-aware mode that returns an {{AlignedText}} (normalized text
> plus {{Alignment}}), in addition to the existing {{CharSequence}} method. A
> length-preserving rung returns an identity alignment; a length-changing rung
> builds a real one. The {{CharClass}}-backed rungs derive it cheaply during
> the pass they already make; the standard-engine rungs are the hard case (next
> point).
> # *Map composition.* {{Alignment.andThen}} -- *already delivered*. A pipeline
> of N rungs collapses to a single original-to-final alignment by folding
> {{andThen}} over the stages.
> # *Offset-aware pipeline.* Add an offset-aware build/apply mode to
> {{TextNormalizer}} that runs the configured rungs in order, composes their
> alignments, and returns one {{AlignedText}} for the whole chain.
> # *Reuse the standard engines where possible -- with a known edit-tracking
> gap.* NFC/NFD/NFKC/NFKD and full case folding are the value of delegating to
> standards, but {{java.text.Normalizer.normalize(s, form)}} returns only the
> result string with *no edit information*, so an {{Alignment}} cannot be read
> out of it directly. The realistic options, to be decided in design review:
> (a) depend on ICU4J's {{Normalizer2}}, which can emit an {{Edits}} object
> that maps directly onto our {{equal}}/{{replace}} runs; (b) compute the
> alignment by diffing input against output at decomposition boundaries; or (c)
> restrict the offset-aware path to folds we generate ourselves (the
> {{CharClass}} family plus a bundled-table case fold) and treat
> {{java.text.Normalizer}}-only NFC/NFKC as offset-opaque. This is the single
> biggest design decision in the issue and should be settled first.
> The {{CharClass}} {{*Aligned}} family and the DL {{normalizeInputAligned}}
> are the first instances of the offset-aware contract; this issue lifts the
> pattern up to the whole family.
> h2. Data files and provenance metadata
> We will bundle a CaseFolding lookup (sourced from the Unicode
> {{CaseFolding.txt}}) and aim for completeness against the upstream file. The
> key addition is *provenance metadata*: every mapping records which standard
> it follows, so we can audit coverage, distinguish standard-mandated folds
> from project choices, and re-derive the table when Unicode revises.
> Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a
> documented header schema) with columns roughly:
> {noformat}
> source ; target ; fold_type ; standard ;
> sentiment ; unicode_version ; notes
> 00DF ; 0073 0073 ; CASE_FOLD ; UTS21:CaseFolding:F ;
> ; 17.0.0 ; sharp s -> ss
> 2026 ; 002E 002E 002E ; NFKC ; UAX44:Decomposition:compat ;
> ; 17.0.0 ; horizontal ellipsis
> 2014 2013 ; 002D ; DASH ; UNSPECIFIED ;
> ; - ; dash-to-hyphen target choice
> 1F600 ; 003A 0029 ; EMOJI ; CLDR:annotation ;
> positive ; 17.0.0 ; grinning face -> :) (expansion)
> 003A 0029 ; 1F642 ; EMOTICON ; UNSPECIFIED ;
> positive ; - ; :) -> slightly smiling face (contraction)
> {noformat}
> The {{standard}} field is the important one. Suggested values:
> * {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form
> folds
> * {{UAX44:Decomposition:<type>}} for a specific decomposition type
> * {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S
> (simple), T (Turkic)
> * {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a
> published standard." This is the explicit "we just did that" marker the team
> wants, so a reviewer or auditor immediately knows the entry is a project
> decision (for example our choice of ASCII hyphen-minus as the dash target, or
> expanding the ellipsis ourselves rather than relying on NFKC).
> h2. Emoji and emoticons, both directions, sentiment, and token classification
> Emoji-to-emoticon and emoticon-to-emoji are exactly the bidirectional,
> length-changing case this issue is built for: an emoji rendered as {{:)}} is
> an expansion, the reverse a contraction, and a supplementary-plane emoji is
> also the two-UTF-16-units-collapsing case. Both directions are in scope,
> riding the same offset-aware rung and lookup table. There is no governing
> standard for the mapping itself (emoticons are informal and the relation is
> many-to-many and culturally variable), so most rows are tagged
> {{UNSPECIFIED}}; where a direction can be anchored on published data we do
> (for example CLDR emoji annotations for the emoji short-name) and tag it
> accordingly.
> The bigger payoff is turning an opaque pictograph into *lexical signal*.
> Beyond the emoticon form, the same lookup can carry the emoji's actual text,
> its CLDR short-name (for example "grinning face"), so the rung can surface a
> real word sequence instead of an out-of-vocabulary symbol that most models
> simply drop. That text becomes the input that sentiment analysis, semantic
> search, embedding pipelines, and LLM-style consumers actually read and reason
> about, which is where this helps downstream understanding the most. The
> {{Term}} model is the natural carrier: one token can expose {{original}} (the
> emoji), a folded {{emoticon}}, a human-readable {{name}}, and a {{sentiment}}
> tag as parallel layers, and each consumer picks the layer it needs (a
> sentiment lexicon match, the name text for an embedding, or the original for
> display). Sentiment carries its own provenance via a {{sentiment_source}}
> field (an emoji sentiment ranking dataset, a project lexicon, or
> {{UNSPECIFIED}}), kept separate from the mapping's {{standard}} tag so each
> is independently auditable and replaceable.
> The same annotation mechanism generalizes well beyond sentiment. An emoji's
> name is frequently an entity or concept in its own right (an airplane, a
> tower, the Statue of Liberty, a national flag), so surfacing that name plus
> an optional coarse entity-type tag (for example VEHICLE, LANDMARK, ANIMAL,
> FLAG, FOOD) gives the Name Finder real lexical content and a gazetteer-like
> signal where it would otherwise see an unknown glyph. The same row can also
> carry a document-category hint (a pizza or an airplane is a strong cue for a
> food or a travel document), feeding Document Categorizer features. So the
> lookup is less a single sentiment column and more a small, provenance-tagged
> annotation layer per symbol: name, sentiment, entity type, and category, each
> optional and each with its own source, surfaced through the {{Term}} model
> and consumed by the Sentiment Detection, Name Finder, and Doccat components
> respectively. Each annotation dimension is opt-in, so a consumer that only
> wants the folded text pays nothing for the rest.
> Token classification ties in too. The UAX #29 tokenizer already gives emoji
> their own {{WordType}} category, so the same data lets downstream counting
> and feature generation treat emoji and emoticons as a consistent class rather
> than stray symbols, for example folding them into a punctuation-like bucket
> or a dedicated emoji feature for corpus statistics and the ML feature
> generators. That keeps a document's token counts and features stable whether
> sentiment is expressed as {{:)}}, as an emoji, or as the spelled-out name.
> h2. Reconciliation
> The runtime fold tables will be reconciled from several sources, so we need
> explicit rules:
> # *Prefer standard engines over tables.* For anything
> {{java.text.Normalizer}} already does (NFC/NFKC/NFD/NFKD), use it for the
> *text* and do not duplicate the mapping in a file (the offset-aware question
> for these is the edit-tracking gap above, not the mapping itself). The file
> is for what Java does not give us: full case folding and our project folds.
> # *Provenance on every entry.* Each bundled mapping carries its {{standard}}
> tag (above). A build/test step audits the table against the authoritative
> upstream file for that standard (for example: every CaseFolding.txt status C
> and F row is present and identical) and fails on drift or gaps, which is how
> we keep "try to make it complete" honest.
> # *Conflict precedence.* If two sources map the same code point differently
> for the same fold type, the more specific published standard wins over a
> general one, and any deviation we deliberately keep must be re-tagged
> {{UNSPECIFIED}} with a note, never left looking standard-backed.
> # *Do we need to roll our own?* Mostly no. The open question to settle in
> this issue is the exact boundary: confirm that NFC/NFKC and case folding are
> fully delegated to standard data, and that {{UNSPECIFIED}} is reserved for
> the genuinely project-specific folds (dash target, quote folding,
> ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable
> skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
> h2. Design decisions (from OPENNLP-1850; assumed baseline on main)
> The following were settled during OPENNLP-1850 and are already shipped. This
> issue builds on them rather than reopening them.
> * *Offset-aware contract.* A parallel {{OffsetAwareNormalizer}} interface
> (extends {{CharSequenceNormalizer}}, adds {{normalizeAligned()}}).
> {{TextNormalizer.Builder.buildAligned()}} composes only rungs that implement
> it and throws {{IllegalStateException}} naming the offending rung otherwise.
> Callers probe with {{instanceof OffsetAwareNormalizer}} (same pattern as
> {{OffsetMappingNameFinder}} in the DL layer).
> * *Standard-engine folds in aligned pipelines.* NFC, NFKC, accent fold,
> confusable skeleton, and case fold delegate to {{java.text.Normalizer}} or
> JDK case mapping and *cannot* report per-character edits. They are excluded
> from {{buildAligned()}} in 1850; making them offset-aware is the core work of
> *this* issue.
> * *Default vs opt-in behavior.* {{TextNormalizer.defaultChain()}} is the
> conservative matching chain: strip invisible, NFC, whitespace, quotes,
> dashes, case fold, accent fold. DL input folding (whitespace and dashes) is
> opt-in via {{InferenceOptions}}. NFKC is not in {{defaultChain()}}. This
> issue does not change default normalization behavior; offset-aware capability
> and provenance-tagged lookups are additive.
> * *Emoji classification.* {{WordType.EMOJI}} is a dedicated category (not
> punctuation). Classification precedence is emoji > script >
> alphanumeric/numeric. {{Extended_Pictographic.txt}} is a filtered UCD subset
> bundled and cursor-parsed like the other 1850 data files.
> * *Bundled data format (1850 precedent).* Line-oriented {{.txt}} resources,
> cursor-parsed at load time, fail-loud on malformed input. No regex in the
> parse path. Provenance columns are not present in 1850 data files; they are
> introduced in this issue for CaseFolding and project-specific mappings.
> h2. Remaining design work
> * * *How NFC/NFKC/full case fold produce an {{Alignment}}.*
> {{java.text.Normalizer}} returns only a string. Choose and implement one of:
> ICU4J {{Normalizer2}} + {{Edits}}, input/output diff at decomposition
> boundaries, or a bundled CaseFolding lookup that generates the alignment
> during our own pass. Evaluate ICU dependency vs diff complexity in the first
> implementation PR.
> * *Provenance-tagged lookup format.* Extend the 1850 {{.txt}} + cursor-parse
> pattern with a documented header/schema for {{standard}}, {{fold_type}}, and
> {{UNSPECIFIED}} project choices; or introduce a small CSV if columnar
> provenance is clearer. One file per fold type for bundled tables (consistent
> with {{WordBreakProperty.txt}}, {{confusables.txt}}, etc.).
> * *{{UNSPECIFIED}} entries.* Allowed for genuine project choices (dash
> target, quote targets, emoji/emoticon mappings with no governing standard).
> Each must carry an inline note. Standard-backed rows must audit clean against
> the upstream file. No separate sign-off gate in the build for 1850-style
> project folds.
> * *Emoji/emoticon annotation layer.* Bidirectional emoji↔emoticon rung plus
> optional {{Term}} layers (name, sentiment, entity type, category). CLDR
> annotations for names where available; sentiment/entity/category sources and
> file layout decided in the first annotation PR. Annotations in a separate
> lookup file keyed by code point, not mixed into the fold table.
> h2. Scope and acceptance criteria
> * (DONE in OPENNLP-1850) {{Alignment.andThen}} composition with tests,
> including expand-then-contract and contract-then-expand chains.
> * (DONE in OPENNLP-1850) The offset-aware contract on {{CharClass}} returning
> {{AlignedText}}, with round-trip tests for collapse, supplementary-dash
> contraction, supplementary-replacement expansion, and edge deletions.
> * An offset-aware mode on the rest of the {{CharSequenceNormalizer}} family
> and on {{TextNormalizer}}, returning a composed {{AlignedText}} for a
> multi-rung chain (requires resolving the NFC/NFKC edit-tracking question).
> * A bundled, provenance-tagged CaseFolding lookup with a completeness audit
> test against upstream {{CaseFolding.txt}}.
> * Round-trip tests: for representative expanding (ellipsis, eszett, ligature,
> fraction) and contracting (run collapse, supplementary dash, NFC compose)
> inputs, a position in the normalized text maps back to the correct original
> characters through the full pipeline.
> * A bidirectional, opt-in emoji/emoticon rung backed by the same
> provenance-tagged lookup, offset-aware like every other rung, with round-trip
> tests in both directions.
> * A small, provenance-tagged annotation layer on the lookup (name, sentiment,
> entity type, category), surfaced through the {{Term}} model and wired as
> optional input to the Sentiment Detection, Name Finder, and Document
> Categorizer components.
> * Consistent {{WordType}}-level classification of emoji and emoticons so
> token counts and ML feature generators treat them as one class.
> h2. Design guidelines and codebase consistency
> OPENNLP-1850 merged the normalization engine the rest of this work builds on.
> The following are already in place on {{main}} and are assumed by this issue:
> * {{CharClass}} / {{CodePointSet}} cursor engine — UCD-sourced
> {{White_Space}} and {{Dash}} sets, no regex on text paths
> * {{Alignment}} / {{AlignedText}}, {{OffsetAwareNormalizer}},
> {{TextNormalizer.buildAligned()}}, and the {{CharClass}} {{*Aligned}} variants
> * UAX #29 word tokenizer ({{WordSegmenter}}, {{WordTokenizer}}, {{WordType}})
> * {{Term}} / {{TermAnalyzer}} and per-language {{NormalizationProfile(s)}}
> * DL integration: Unicode {{White_Space}} chunking, optional input folding,
> {{NameFinderDL.findInOriginal()}} via {{Alignment}}
> h3. Design principals
> Three design goals govern that engine and should govern new normalization
> work going forward:
> # *Standards-sourced character classes.* UCD properties ({{White_Space}},
> {{Dash}}, …), not {{Character.isWhitespace}} or regex character classes like
> {{\s}}.
> # *Cursor-based text transforms.* Single forward code-point scans for
> normalization and token-boundary logic on user text. Regex is fine for config
> parsing and other non-user-text utilities.
> # *Offset preservation where spans matter.* Length-changing folds compose
> through {{Alignment}} so consumers can map normalized positions back to the
> original input. That is this issue's job for the standard-engine folds
> {{buildAligned()}} rejects today; the {{CharClass}} rungs and DL path already
> do it.
> The manual (OPENNLP-1850 docs) should describe these as *design goals* the
> engine follows and that new code should follow. Do not claim every legacy
> class in {{opennlp.tools.util.normalizer}} or all of OpenNLP already meets
> them — several pre-existing normalizers still use regex and {{StringUtil}}
> still uses JVM whitespace predicates. That gap is tracked below, not a defect
> in the merged engine.
> Target manual wording ({{normalizer.xml}} intro):
> {quote}
> The normalization engine follows three design goals: standards-sourced
> character classes, cursor-based text transforms, and offset preservation
> where spans matter. New normalization code should align with these goals;
> older components are migrated on a best-effort basis (tracked in
> OPENNLP-1852).
> {quote}
> State the no-regex / UCD rationale once in the normalizer chapter; DL and
> tokenizer chapters should xref rather than restate.
> h3. Consistency scoreboard
> Sub-tasks to file under this epic when ready; until then this table is the
> scoreboard:
> || # || Summary || Component || Priority || Status ||
> | 1 | Audit {{split("\\s+")}} and {{Character.isWhitespace}} call sites
> outside the 1850 engine | codebase-wide | medium | todo |
> | 2 | Migrate {{StringUtil}} whitespace to UCD-backed API or document
> intentional exception | {{opennlp-api}} | medium | todo |
> | 3 | Replace regex in legacy {{TwitterCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 4 | Replace regex in legacy {{UrlCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 5 | Replace regex in legacy {{ShrinkCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 6 | Replace regex in legacy {{NumberCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 7 | Replace regex in legacy {{EmojiCharSequenceNormalizer}} |
> {{opennlp-runtime}} | low | todo |
> | 8 | Audit spellcheck / extension normalizers for regex-on-text paths |
> extensions | low | todo |
> | 9 | Soften manual intro to design-goals wording; trim duplicated
> no-regex/ReDoS prose across chapters | {{opennlp-docs}} | medium | todo |
> | 10 | Add design-guidelines xref in {{normalizer.xml}} pointing here |
> {{opennlp-docs}} | low | todo |
> When the scoreboard is substantially green, tighten the manual from "design
> goals" to stronger statements. Scoreboard items do not block the offset-aware
> fold work in this issue.
> h2. Out of scope
> * * Changing default normalization behavior; this issue adds offset-aware
> capability and the provenance model, it does not turn expanding folds on by
> default.
> * Legacy regex normalizer migration, {{StringUtil}} whitespace
> standardization, and manual wording — tracked in the scoreboard above, not
> acceptance criteria for this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
