[
https://issues.apache.org/jira/browse/OPENNLP-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092827#comment-18092827
]
Kristian Rickert commented on OPENNLP-1867:
-------------------------------------------
Closing. Not needed for OPENNLP-1852. Adding some tests in OPENNLP-1868
instead.
> Offset-aware normalization pipeline: AlignedText contract across
> CharSequenceNormalizer and TextNormalizer
> ----------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1867
> URL: https://issues.apache.org/jira/browse/OPENNLP-1867
> Project: OpenNLP
> Issue Type: Task
> Components: dl, Sentiment Detection
> Affects Versions: 3.0.0-M4
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
>
> *This is one of 4 tickets we will work on in a similar fashion to 1850. The
> main requirements are on OPENNLP-1852*
> This would be referred to as 1852-a until all tickets are created at which
> time I'll correct it.
> *Stacks on:* main (post-1850). *Blocks:* OPENNLP-1868, OPENNLP-1869,
> OPENNLP-1870.
> *Summary.* Give the {{CharSequenceNormalizer}} family an offset-aware mode
> that returns an {{AlignedText}} (normalized text plus {{{}Alignment{}}})
> alongside the existing {{CharSequence}} method: a length-preserving rung
> returns an identity alignment, a length-changing rung builds a real one. Add
> an offset-aware build/apply mode to {{TextNormalizer}} that runs the
> configured rungs in order, folds {{andThen}} over their alignments, and
> returns one {{AlignedText}} for the whole chain. Settle and implement the
> gate decision (recommendation: keep NFC/NFKC offset-opaque via the existing
> {{buildAligned()}} rejection; author-able folds are offset-aware).
> *Design note.* Decide whether the offset-aware method lives on
> {{CharSequenceNormalizer}} directly (every rung implements it) or on a
> parallel capability interface that the pipeline adapts (mirrors how
> {{OffsetMappingNameFinder}} was done in OPENNLP-1850). The CharClass-backed
> rungs already have the machinery; the standard-engine rungs are where the
> gate applies.
> *Acceptance criteria.*
> * Offset-aware mode on the {{CharSequenceNormalizer}} family and on
> {{{}TextNormalizer{}}}, returning a composed {{AlignedText}} for a multi-rung
> chain.
> * Length-preserving rungs return identity alignments; the pipeline composes
> via {{{}andThen{}}}.
> * NFC/NFKC (and any fold that cannot emit edits) fail loud in the
> offset-aware path, consistent with today's {{{}buildAligned(){}}}.
> * Round-trip tests: a position in the normalized output maps back to the
> correct original characters through a multi-rung chain (mix of identity and
> length-changing rungs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
