[
https://issues.apache.org/jira/browse/OPENNLP-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1852:
--------------------------------------
Issue Type: Epic (was: New Feature)
> Offset-aware normalization pipeline: handle length-changing folds (expansion
> and contraction) with a single offset model
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1852
> URL: https://issues.apache.org/jira/browse/OPENNLP-1852
> Project: OpenNLP
> Issue Type: Epic
> Components: Tokenizer
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
>
> h2. Summary
> Normalization can change string length in both directions, and today only one
> corner of the library accounts for it. Canonical and compatibility
> decompositions, full case folding, ligature and fraction expansion, and
> run-collapsing all change the number of characters between the input and the
> normalized output. The {{CharClass}} engine already produces an {{Alignment}}
> (carried in an {{AlignedText}}) for its whitespace/dash folds, used by the DL
> components to map entity spans back to the original text, but the broader
> {{CharSequenceNormalizer}} family returns a plain {{CharSequence}} with no
> offset information. The goal of this issue is to make the whole normalization
> pipeline offset-aware so that any chain of expanding or contracting rungs can
> map a result position (an entity span, a search hit, a highlight) back to
> original-document coordinates by construction, not case by case.
> This is a follow-up to OPENNLP-1850. That work delivered the offset *model*
> this issue builds on (see "Already delivered" below) and made the only
> offset-sensitive path that applies length-changing folds today (the DL
> whitespace/dash chunking in {{NameFinderDL}}) offset-safe. This issue
> generalizes the model across the rest of the normalizer family so it can be
> used safely in offset-sensitive contexts.
> h2. Already delivered in OPENNLP-1850 (do not re-scope here)
> The keystone is in place; this issue should build on it rather than reinvent
> it.
> * {{Alignment}} -- a bidirectional edit-sequence offset model
> ({{equal}}/{{replace}} runs) that represents expansion, contraction, deletion
> (as gaps), and 1:1 changes uniformly, and maps span-to-span with
> {{toOriginalSpan}}/{{toNormalizedSpan}}. It replaced the earlier
> per-output-character {{OffsetMap}}, which assumed the normalized text
> contiguously covered the original and over-covered deletions. ({{OffsetMap}}
> and {{NormalizedText}} were never released and are removed.)
> * {{Alignment.andThen(Alignment)}} -- associative composition of two stages
> into one original-to-final alignment. Built and tested, including
> expand-then-contract and contract-then-expand chains.
> * The offset-aware *contract* on {{CharClass}}: {{normalizeAligned}},
> {{collapseAligned}}, {{collapsePreservingAligned}}, {{trimAligned}},
> {{removeAllAligned}}, each returning an {{AlignedText}} (normalized text plus
> {{Alignment}}).
> * The DL path consuming it: {{AbstractDL.normalizeInputAligned}} and
> {{NameFinderDL.findInOriginal}} map decoded spans back via
> {{Alignment.toOriginalSpan}}.
> So the remaining work is to lift the offset-aware contract from {{CharClass}}
> up to the whole {{CharSequenceNormalizer}} family and the {{TextNormalizer}}
> pipeline, add the provenance-tagged data, and add the emoji/emoticon rung and
> annotation layer.
> h2. Background: there is a standard, and it classifies every character for us
> We do not need to hand-enumerate "special characters." Unicode already
> catalogs which characters expand or contract and to what, so the work is to
> consume that data, not invent it.
> || Source standard || What it covers || Direction || Notes ||
> | UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD) | Canonical and
> compatibility (de)composition | both | Implemented by
> {{java.text.Normalizer}} -- but see the edit-tracking caveat under Proposed
> architecture: it returns a plain string with no edit information |
> | UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX
> #44) | Per-character expansions, tagged by type: {{<fraction>}} (1/2),
> {{<super>}}/{{<sub>}} (squared to 2), {{<compat>}} (ligature fi, ellipsis to
> three dots, Roman numerals), {{<square>}}, {{<wide>}}/{{<narrow>}} | expand |
> This is the authoritative expansion catalog and the basis for NFKC |
> | CaseFolding.txt (UTS #21 section 5.18) | Case folding, including the full
> (expanding) folds with status F: eszett to ss, ligature ff to ff | expand and
> 1:1 | Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding,
> so full folds genuinely need this data |
> | Project folds (dash, quote, ellipsis, digit, confusable) | Our own target
> choices, some derived from Unicode properties (Dash, White_Space) and some
> hand-authored | both | These are the ones with no single governing standard;
> see provenance below |
> Contractions are the mirror image of the same data: NFC composition
> (combining marks fold into one precomposed character), our whitespace and
> dash run-collapses, and the UTF-16 case where a supplementary code point (two
> UTF-16 units) collapses to one unit.
> h2. The core idea: one offset model, not a per-character table
> Expansion and contraction are the same problem stated twice: the output
> length differs from the input length. {{Alignment}} solves it generically. It
> records the edits between input and output as a sequence of _equal_ runs
> (text copied through, length unchanged) and _replace_ runs (M original
> characters that produced P normalized characters), so 1-to-3 (ellipsis),
> 3-to-1 (run collapse), 2-to-1 (supplementary dash), and N-to-0 (deletion) are
> all just runs. It maps span-to-span, so a match that ends next to deleted
> text reports a tight original span instead of over-covering the deletion, and
> it composes with {{andThen}}. The character data only decides _what_ each
> fold produces; {{Alignment}} handles _where everything went_.
> h2. Proposed architecture
> # *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}}
> family an offset-aware mode that returns an {{AlignedText}} (normalized text
> plus {{Alignment}}), in addition to the existing {{CharSequence}} method. A
> length-preserving rung returns an identity alignment; a length-changing rung
> builds a real one. The {{CharClass}}-backed rungs derive it cheaply during
> the pass they already make; the standard-engine rungs are the hard case (next
> point).
> # *Map composition.* {{Alignment.andThen}} -- *already delivered*. A pipeline
> of N rungs collapses to a single original-to-final alignment by folding
> {{andThen}} over the stages.
> # *Offset-aware pipeline.* Add an offset-aware build/apply mode to
> {{TextNormalizer}} that runs the configured rungs in order, composes their
> alignments, and returns one {{AlignedText}} for the whole chain.
> # *Reuse the standard engines where possible -- with a known edit-tracking
> gap.* NFC/NFD/NFKC/NFKD and full case folding are the value of delegating to
> standards, but {{java.text.Normalizer.normalize(s, form)}} returns only the
> result string with *no edit information*, so an {{Alignment}} cannot be read
> out of it directly. The realistic options, to be decided in design review:
> (a) depend on ICU4J's {{Normalizer2}}, which can emit an {{Edits}} object
> that maps directly onto our {{equal}}/{{replace}} runs; (b) compute the
> alignment by diffing input against output at decomposition boundaries; or (c)
> restrict the offset-aware path to folds we generate ourselves (the
> {{CharClass}} family plus a bundled-table case fold) and treat
> {{java.text.Normalizer}}-only NFC/NFKC as offset-opaque. This is the single
> biggest design decision in the issue and should be settled first.
> The {{CharClass}} {{*Aligned}} family and the DL {{normalizeInputAligned}}
> are the first instances of the offset-aware contract; this issue lifts the
> pattern up to the whole family.
> h2. Data files and provenance metadata
> We will bundle a CaseFolding lookup (sourced from the Unicode
> {{CaseFolding.txt}}) and aim for completeness against the upstream file. The
> key addition is *provenance metadata*: every mapping records which standard
> it follows, so we can audit coverage, distinguish standard-mandated folds
> from project choices, and re-derive the table when Unicode revises.
> Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a
> documented header schema) with columns roughly:
> {noformat}
> source ; target ; fold_type ; standard ;
> sentiment ; unicode_version ; notes
> 00DF ; 0073 0073 ; CASE_FOLD ; UTS21:CaseFolding:F ;
> ; 17.0.0 ; sharp s -> ss
> 2026 ; 002E 002E 002E ; NFKC ; UAX44:Decomposition:compat ;
> ; 17.0.0 ; horizontal ellipsis
> 2014 2013 ; 002D ; DASH ; UNSPECIFIED ;
> ; - ; dash-to-hyphen target choice
> 1F600 ; 003A 0029 ; EMOJI ; CLDR:annotation ;
> positive ; 17.0.0 ; grinning face -> :) (expansion)
> 003A 0029 ; 1F642 ; EMOTICON ; UNSPECIFIED ;
> positive ; - ; :) -> slightly smiling face (contraction)
> {noformat}
> The {{standard}} field is the important one. Suggested values:
> * {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form
> folds
> * {{UAX44:Decomposition:<type>}} for a specific decomposition type
> * {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S
> (simple), T (Turkic)
> * {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a
> published standard." This is the explicit "we just did that" marker the team
> wants, so a reviewer or auditor immediately knows the entry is a project
> decision (for example our choice of ASCII hyphen-minus as the dash target, or
> expanding the ellipsis ourselves rather than relying on NFKC).
> h2. Emoji and emoticons, both directions, sentiment, and token classification
> Emoji-to-emoticon and emoticon-to-emoji are exactly the bidirectional,
> length-changing case this issue is built for: an emoji rendered as {{:)}} is
> an expansion, the reverse a contraction, and a supplementary-plane emoji is
> also the two-UTF-16-units-collapsing case. Both directions are in scope,
> riding the same offset-aware rung and lookup table. There is no governing
> standard for the mapping itself (emoticons are informal and the relation is
> many-to-many and culturally variable), so most rows are tagged
> {{UNSPECIFIED}}; where a direction can be anchored on published data we do
> (for example CLDR emoji annotations for the emoji short-name) and tag it
> accordingly.
> The bigger payoff is turning an opaque pictograph into *lexical signal*.
> Beyond the emoticon form, the same lookup can carry the emoji's actual text,
> its CLDR short-name (for example "grinning face"), so the rung can surface a
> real word sequence instead of an out-of-vocabulary symbol that most models
> simply drop. That text becomes the input that sentiment analysis, semantic
> search, embedding pipelines, and LLM-style consumers actually read and reason
> about, which is where this helps downstream understanding the most. The
> {{Term}} model is the natural carrier: one token can expose {{original}} (the
> emoji), a folded {{emoticon}}, a human-readable {{name}}, and a {{sentiment}}
> tag as parallel layers, and each consumer picks the layer it needs (a
> sentiment lexicon match, the name text for an embedding, or the original for
> display). Sentiment carries its own provenance via a {{sentiment_source}}
> field (an emoji sentiment ranking dataset, a project lexicon, or
> {{UNSPECIFIED}}), kept separate from the mapping's {{standard}} tag so each
> is independently auditable and replaceable.
> The same annotation mechanism generalizes well beyond sentiment. An emoji's
> name is frequently an entity or concept in its own right (an airplane, a
> tower, the Statue of Liberty, a national flag), so surfacing that name plus
> an optional coarse entity-type tag (for example VEHICLE, LANDMARK, ANIMAL,
> FLAG, FOOD) gives the Name Finder real lexical content and a gazetteer-like
> signal where it would otherwise see an unknown glyph. The same row can also
> carry a document-category hint (a pizza or an airplane is a strong cue for a
> food or a travel document), feeding Document Categorizer features. So the
> lookup is less a single sentiment column and more a small, provenance-tagged
> annotation layer per symbol: name, sentiment, entity type, and category, each
> optional and each with its own source, surfaced through the {{Term}} model
> and consumed by the Sentiment Detection, Name Finder, and Doccat components
> respectively. Each annotation dimension is opt-in, so a consumer that only
> wants the folded text pays nothing for the rest.
> Token classification ties in too. The UAX #29 tokenizer already gives emoji
> their own {{WordType}} category, so the same data lets downstream counting
> and feature generation treat emoji and emoticons as a consistent class rather
> than stray symbols, for example folding them into a punctuation-like bucket
> or a dedicated emoji feature for corpus statistics and the ML feature
> generators. That keeps a document's token counts and features stable whether
> sentiment is expressed as {{:)}}, as an emoji, or as the spelled-out name.
> h2. Reconciliation
> The runtime fold tables will be reconciled from several sources, so we need
> explicit rules:
> # *Prefer standard engines over tables.* For anything
> {{java.text.Normalizer}} already does (NFC/NFKC/NFD/NFKD), use it for the
> *text* and do not duplicate the mapping in a file (the offset-aware question
> for these is the edit-tracking gap above, not the mapping itself). The file
> is for what Java does not give us: full case folding and our project folds.
> # *Provenance on every entry.* Each bundled mapping carries its {{standard}}
> tag (above). A build/test step audits the table against the authoritative
> upstream file for that standard (for example: every CaseFolding.txt status C
> and F row is present and identical) and fails on drift or gaps, which is how
> we keep "try to make it complete" honest.
> # *Conflict precedence.* If two sources map the same code point differently
> for the same fold type, the more specific published standard wins over a
> general one, and any deviation we deliberately keep must be re-tagged
> {{UNSPECIFIED}} with a note, never left looking standard-backed.
> # *Do we need to roll our own?* Mostly no. The open question to settle in
> this issue is the exact boundary: confirm that NFC/NFKC and case folding are
> fully delegated to standard data, and that {{UNSPECIFIED}} is reserved for
> the genuinely project-specific folds (dash target, quote folding,
> ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable
> skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
> h2. Open questions for the design review
> * *How NFC/NFKC/case-fold rungs produce an {{Alignment}}* given
> {{java.text.Normalizer}} exposes no edits: ICU4J {{Normalizer2}}/{{Edits}},
> an input/output diff, or leaving those folds offset-opaque. (The decision
> that gates everything else.)
> * CSV vs annotated {{.txt}} for the lookup, and whether one combined file or
> one file per fold type.
> * Whether the offset-aware method lives on {{CharSequenceNormalizer}}
> directly (every rung implements it) or on a parallel interface that the
> pipeline adapts.
> * How aggressively to expand by default: full case folding (eszett to ss) and
> NFKC change tokens visibly, so they should likely stay opt-in rungs, not
> defaults.
> * Whether {{UNSPECIFIED}} entries should be allowed in a release build at
> all, or must each be reviewed and signed off.
> * Which dataset anchors each annotation dimension (an emoji sentiment ranking
> for sentiment, CLDR for names, and what for entity type and category), and
> whether the annotations live in the same lookup or a separate file keyed by
> code point.
> * Whether emoji and emoticons classify as punctuation or as a dedicated emoji
> {{WordType}} for counting and feature-generation purposes.
> h2. Scope and acceptance criteria
> * (DONE in OPENNLP-1850) {{Alignment.andThen}} composition with tests,
> including expand-then-contract and contract-then-expand chains.
> * (DONE in OPENNLP-1850) The offset-aware contract on {{CharClass}} returning
> {{AlignedText}}, with round-trip tests for collapse, supplementary-dash
> contraction, supplementary-replacement expansion, and edge deletions.
> * An offset-aware mode on the rest of the {{CharSequenceNormalizer}} family
> and on {{TextNormalizer}}, returning a composed {{AlignedText}} for a
> multi-rung chain (requires resolving the NFC/NFKC edit-tracking question).
> * A bundled, provenance-tagged CaseFolding lookup with a completeness audit
> test against upstream {{CaseFolding.txt}}.
> * Round-trip tests: for representative expanding (ellipsis, eszett, ligature,
> fraction) and contracting (run collapse, supplementary dash, NFC compose)
> inputs, a position in the normalized text maps back to the correct original
> characters through the full pipeline.
> * A bidirectional, opt-in emoji/emoticon rung backed by the same
> provenance-tagged lookup, offset-aware like every other rung, with round-trip
> tests in both directions.
> * A small, provenance-tagged annotation layer on the lookup (name, sentiment,
> entity type, category), surfaced through the {{Term}} model and wired as
> optional input to the Sentiment Detection, Name Finder, and Document
> Categorizer components.
> * Consistent {{WordType}}-level classification of emoji and emoticons so
> token counts and ML feature generators treat them as one class.
> h2. Out of scope
> * The DL whitespace/dash offset-safety, already handled under OPENNLP-1850
> ({{NameFinderDL.findInOriginal}} plus {{AbstractDL.normalizeInputAligned}}).
> * The {{Alignment}} model and {{andThen}} composition themselves, delivered
> under OPENNLP-1850; this issue consumes them.
> * Changing any default normalization behavior; this issue adds the
> offset-aware capability and the provenance model, it does not turn expanding
> folds on by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
