Kristian Rickert created OPENNLP-1867:
-----------------------------------------
Summary: Offset-aware normalization pipeline: AlignedText contract
across CharSequenceNormalizer and TextNormalizer
Key: OPENNLP-1867
URL: https://issues.apache.org/jira/browse/OPENNLP-1867
Project: OpenNLP
Issue Type: Task
Components: dl, Sentiment Detection
Affects Versions: 3.0.0-M4
Reporter: Kristian Rickert
Assignee: Kristian Rickert
*This is one of 4 tickets we will work on in a similar fashion to 1850. The
main requirements are on OPENNLP-1852*
This would be referred to as 1852-a until all tickets are created at which time
I'll correct it.
*Stacks on:* main (post-1850). *Blocks:* 1852-b, c, d. (I'll adjust these to
proper OPENNLP names afterward)
*Summary.* Give the {{CharSequenceNormalizer}} family an offset-aware mode that
returns an {{AlignedText}} (normalized text plus {{{}Alignment{}}}) alongside
the existing {{CharSequence}} method: a length-preserving rung returns an
identity alignment, a length-changing rung builds a real one. Add an
offset-aware build/apply mode to {{TextNormalizer}} that runs the configured
rungs in order, folds {{andThen}} over their alignments, and returns one
{{AlignedText}} for the whole chain. Settle and implement the gate decision
(recommendation: keep NFC/NFKC offset-opaque via the existing
{{buildAligned()}} rejection; author-able folds are offset-aware).
*Design note.* Decide whether the offset-aware method lives on
{{CharSequenceNormalizer}} directly (every rung implements it) or on a parallel
capability interface that the pipeline adapts (mirrors how
{{OffsetMappingNameFinder}} was done in 1850). The CharClass-backed rungs
already have the machinery; the standard-engine rungs are where the gate
applies.
*Acceptance criteria.*
* Offset-aware mode on the {{CharSequenceNormalizer}} family and on
{{{}TextNormalizer{}}}, returning a composed {{AlignedText}} for a multi-rung
chain.
* Length-preserving rungs return identity alignments; the pipeline composes
via {{{}andThen{}}}.
* NFC/NFKC (and any fold that cannot emit edits) fail loud in the offset-aware
path, consistent with today's {{{}buildAligned(){}}}.
* Round-trip tests: a position in the normalized output maps back to the
correct original characters through a multi-rung chain (mix of identity and
length-changing rungs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
