Kristian Rickert created OPENNLP-1852:
-----------------------------------------
Summary: Offset-aware normalization pipeline: handle
length-changing folds (expansion and contraction) with a single offset model
Key: OPENNLP-1852
URL: https://issues.apache.org/jira/browse/OPENNLP-1852
Project: OpenNLP
Issue Type: New Feature
Components: Tokenizer
Reporter: Kristian Rickert
Assignee: Kristian Rickert
h2. Summary
Normalization can change string length in both directions, and today only one
corner of the library accounts for it. Canonical and compatibility
decompositions, full case folding, ligature and fraction expansion, and
run-collapsing all change the number of characters between the input and the
normalized output. The {{CharClass}} engine already produces an {{OffsetMap}}
for its whitespace/dash folds (used by the DL components to map entity spans
back to the original text), but the broader {{CharSequenceNormalizer}} family
returns a plain {{CharSequence}} with no offset information. The goal of this
issue is to make the whole normalization pipeline offset-aware so that any
chain of expanding or contracting rungs can map a result position (an entity
span, a search hit, a highlight) back to original-document coordinates by
construction, not case by case.
This is a follow-up to OPENNLP-1850. The in-flight PRs already make the only
offset-sensitive path that applies length-changing folds today (the DL
whitespace/dash chunking in {{{}NameFinderDL{}}}) offset-safe. This issue
generalizes that so the rest of the normalizer family can be used safely in
offset-sensitive contexts.
h2. Background: there is a standard, and it classifies every character for us
We do not need to hand-enumerate "special characters." Unicode already catalogs
which characters expand or contract and to what, so the work is to consume that
data, not invent it.
||Source standard||What it covers||Direction||Notes||
|UAX #15 Normalization Forms (NFC/NFD/NFKC/NFKD)|Canonical and compatibility
(de)composition|both|Implemented by {{{}java.text.Normalizer{}}}; no lookup
table of our own required|
|UCD {{UnicodeData.txt}} Decomposition_Mapping + Decomposition_Type (UAX
#44)|Per-character expansions, tagged by type: {{<fraction>}} (1/2),
{{{}<super>{}}}/{{{}<sub>{}}} (squared to 2), {{<compat>}} (ligature fi,
ellipsis to three dots, Roman numerals), {{{}<square>{}}},
{{{}<wide>{}}}/{{{}<narrow>{}}}|expand|This is the authoritative expansion
catalog and the basis for NFKC|
|CaseFolding.txt (UTS #21 section 5.18)|Case folding, including the full
(expanding) folds with status F: eszett to ss, ligature ff to ff|expand and
1:1|Java's {{toLowerCase}} is locale lowercasing, not Unicode case folding, so
full folds genuinely need this data|
|Project folds (dash, quote, ellipsis, digit, confusable)|Our own target
choices, some derived from Unicode properties (Dash, White_Space) and some
hand-authored|both|These are the ones with no single governing standard; see
provenance below|
Contractions are the mirror image of the same data: NFC composition (combining
marks fold into one precomposed character), our whitespace and dash
run-collapses, and the UTF-16 case where a supplementary code point (two UTF-16
units) collapses to one unit.
h2. The core idea: one offset model, not a per-character table
Expansion and contraction are the same problem stated twice. They both mean
"the output length differs from the input length," and {{OffsetMap}} already
solves it generically: its builder records one original offset per output
character, so 1-to-3 (ellipsis), 3-to-1 (run collapse), and 2-to-1
(supplementary dash) are all just "map each output position back to its
source." Nothing in the runtime path is per-character. The character data only
decides _what_ each fold produces; {{OffsetMap}} handles {_}where everything
went{_}.
h2. Proposed architecture
# *Offset-aware normalizer contract.* Give the {{CharSequenceNormalizer}}
family an offset-aware mode that returns a {{NormalizedText}} (normalized text
plus {{{}OffsetMap{}}}), in addition to the existing {{CharSequence}} method. A
length-preserving rung returns an identity map; a length-changing rung builds a
real one. Most rungs can derive the map cheaply during the pass they already
make.
# *Map composition.* Add {{OffsetMap.andThen(OffsetMap inner)}} (or an
equivalent composer) so a pipeline of N rungs collapses to a single
original-to-final map. Composition is associative and is what lets an arbitrary
chain stay offset-safe.
# *Offset-aware pipeline.* Add an offset-aware build/apply mode to
{{TextNormalizer}} that runs the configured rungs in order, composes their
maps, and returns one {{NormalizedText}} for the whole chain.
# *Reuse the standard engines where possible.* NFC/NFD/NFKC/NFKD go through
{{java.text.Normalizer}} with an offset map derived from the decomposition
boundaries; we do not maintain a decomposition table. Case folding uses the
bundled CaseFolding data. Only project-specific folds carry their own small
tables.
The existing {{{}CharClass.normalizeMapped{}}}/{{{}collapseMapped{}}} and the
DL {{normalizeInputMapped}} are the first instances of this pattern; this issue
lifts the pattern up to the whole family.
h2. Data files and provenance metadata
We will bundle a CaseFolding lookup (sourced from the Unicode
{{{}CaseFolding.txt{}}}) and aim for completeness against the upstream file.
The key addition is {*}provenance metadata{*}: every mapping records which
standard it follows, so we can audit coverage, distinguish standard-mandated
folds from project choices, and re-derive the table when Unicode revises.
Proposed move from the raw {{.txt}} to a small CSV (or a {{.txt}} with a
documented header schema) with columns roughly:
{noformat}
source ; target ; fold_type ; standard ;
unicode_version ; notes
00DF ; 0073 0073 ; CASE_FOLD ; UTS21:CaseFolding:F ;
17.0.0 ; LATIN SMALL LETTER SHARP S -> ss
2026 ; 002E 002E 002E ; NFKC ; UAX44:Decomposition:compat ;
17.0.0 ; HORIZONTAL ELLIPSIS
2014 2013 ; 002D ; DASH ; UNSPECIFIED ; -
; project dash-to-hyphen target choice
{noformat}
The {{standard}} field is the important one. Suggested values:
* {{UAX15:NFC}} / {{UAX15:NFKC}} (and the D forms) for normalization-form folds
* {{UAX44:Decomposition:<type>}} for a specific decomposition type
* {{UTS21:CaseFolding:<status>}} where status is C (common), F (full), S
(simple), T (Turkic)
* {{UNSPECIFIED}} meaning "we chose this mapping; it is not mandated by a
published standard." This is the explicit "we just did that" marker the team
wants, so a reviewer or auditor immediately knows the entry is a project
decision (for example our choice of ASCII hyphen-minus as the dash target, or
expanding the ellipsis ourselves rather than relying on NFKC).
h2. Reconciliation
The runtime fold tables will be reconciled from several sources, so we need
explicit rules:
# *Prefer standard engines over tables.* For anything {{java.text.Normalizer}}
already does (NFC/NFKC/NFD/NFKD), use it and do not duplicate the mapping in a
file. The file is for what Java does not give us: full case folding and our
project folds.
# *Provenance on every entry.* Each bundled mapping carries its {{standard}}
tag (above). A build/test step audits the table against the authoritative
upstream file for that standard (for example: every CaseFolding.txt status C
and F row is present and identical) and fails on drift or gaps, which is how we
keep "try to make it complete" honest.
# *Conflict precedence.* If two sources map the same code point differently
for the same fold type, the more specific published standard wins over a
general one, and any deviation we deliberately keep must be re-tagged
{{UNSPECIFIED}} with a note, never left looking standard-backed.
# *Do we need to roll our own?* Mostly no. The open question to settle in this
issue is the exact boundary: confirm that NFC/NFKC and case folding are fully
delegated to standard data, and that {{UNSPECIFIED}} is reserved for the
genuinely project-specific folds (dash target, quote folding,
ellipsis-if-we-want-it-independent-of-NFKC, digit folding, confusable
skeletons). Minimizing the {{UNSPECIFIED}} set is a goal.
h2. Open questions for the design review
* CSV vs annotated {{.txt}} for the lookup, and whether one combined file or
one file per fold type.
* Whether the offset-aware method lives on {{CharSequenceNormalizer}} directly
(every rung implements it) or on a parallel interface that the pipeline adapts.
* How aggressively to expand by default: full case folding (eszett to ss) and
NFKC change tokens visibly, so they should likely stay opt-in rungs, not
defaults.
* Whether {{UNSPECIFIED}} entries should be allowed in a release build at all,
or must each be reviewed and signed off.
h2. Scope and acceptance criteria
* {{OffsetMap.andThen}} composition with tests, including expand-then-contract
and contract-then-expand chains.
* An offset-aware mode on the normalizer family and on {{{}TextNormalizer{}}},
returning a composed {{NormalizedText}} for a multi-rung chain.
* A bundled, provenance-tagged CaseFolding lookup with a completeness audit
test against upstream {{{}CaseFolding.txt{}}}.
* Round-trip tests: for representative expanding (ellipsis, eszett, ligature,
fraction) and contracting (run collapse, supplementary dash, NFC compose)
inputs, a position in the normalized text maps back to the correct original
characters.
h2. Out of scope
* The DL whitespace/dash offset-safety, already handled under OPENNLP-1850
({{{}NameFinderDL.findInOriginal{}}} plus
{{{}AbstractDL.normalizeInputMapped{}}}).
* Changing any default normalization behavior; this issue adds the
offset-aware capability and the provenance model, it does not turn expanding
folds on by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
