[
https://issues.apache.org/jira/browse/OPENNLP-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner resolved OPENNLP-1876.
-------------------------------------
Resolution: Delivered
> Replace regex with cursor scans in the legacy CharSequenceNormalizers
> ---------------------------------------------------------------------
>
> Key: OPENNLP-1876
> URL: https://issues.apache.org/jira/browse/OPENNLP-1876
> Project: OpenNLP
> Issue Type: Improvement
> Affects Versions: 3.0.0-M4
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M5
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> h2. Summary
> The legacy normalizers {{TwitterCharSequenceNormalizer}},
> {{UrlCharSequenceNormalizer}}, {{ShrinkCharSequenceNormalizer}}, and
> {{NumberCharSequenceNormalizer}} apply regular expressions to user text,
> which runs against the OPENNLP-1850 design goal of cursor-based text
> transforms (a single forward code-point scan on user text, with regex
> reserved for config parsing). This ticket replaces those regex passes with
> cursor scans on the {{CharClass}} engine where they operate on user text, and
> audits the spellcheck and extension normalizers for the same pattern. These
> are pre-1850 rungs and are not in the blessed default chain, so the change is
> low-risk and output-preserving rather than a redesign.
> h2. Scope
> * Replace regex-on-text with a cursor scan (on the
> {{CharClass}}/{{CodePointSet}} primitives from OPENNLP-1850) in
> {{TwitterCharSequenceNormalizer}}, {{UrlCharSequenceNormalizer}},
> {{ShrinkCharSequenceNormalizer}}, and {{NumberCharSequenceNormalizer}}.
> * Where a normalizer's behavior is subtle, add characterization tests that
> pin the pre-change output first, then refactor, so the output is preserved
> byte for byte.
> * Audit the spellcheck and extension normalizers for regex-on-text paths;
> convert the clear cases and file follow-ups for anything larger.
> h2. Acceptance criteria
> * The four normalizers use cursor scans on the text path; no regex is applied
> to user text in them.
> * Output-preserving: characterization tests capture the prior behavior and
> pass unchanged after the refactor.
> * The spellcheck and extension audit is complete, with conversions applied or
> follow-ups filed.
> h2. Out of scope
> * Making these rungs offset-aware. They are not in the blessed chain, and
> offset-safety for them is a separate concern.
> * The {{StringUtil}} whitespace migration, which is a separate ticket.
> * Regex in config parsing and non-user-text utilities.
> * Adding or removing any of these rungs from a default chain.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
