Kristian Rickert created OPENNLP-1876:
-----------------------------------------
Summary: Remove regex over cursors TwitterCharSequenceNormalizer,
UrlCharSequenceNormalizer, ShrinkCharSequenceNormalizer, and
NumberCharSequenceNormalizer
Key: OPENNLP-1876
URL: https://issues.apache.org/jira/browse/OPENNLP-1876
Project: OpenNLP
Issue Type: Improvement
Affects Versions: 3.0.0-M4
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Fix For: 3.0.0-M5
h2. Summary
The legacy normalizers {{TwitterCharSequenceNormalizer}},
{{UrlCharSequenceNormalizer}}, {{ShrinkCharSequenceNormalizer}}, and
{{NumberCharSequenceNormalizer}} apply regular expressions to user text, which
runs against the OPENNLP-1850 design goal of cursor-based text transforms (a
single forward code-point scan on user text, with regex reserved for config
parsing). This ticket replaces those regex passes with cursor scans on the
{{CharClass}} engine where they operate on user text, and audits the spellcheck
and extension normalizers for the same pattern. These are pre-1850 rungs and
are not in the blessed default chain, so the change is low-risk and
output-preserving rather than a redesign.
h2. Scope
* Replace regex-on-text with a cursor scan (on the
{{CharClass}}/{{CodePointSet}} primitives from OPENNLP-1850) in
{{TwitterCharSequenceNormalizer}}, {{UrlCharSequenceNormalizer}},
{{ShrinkCharSequenceNormalizer}}, and {{NumberCharSequenceNormalizer}}.
* Where a normalizer's behavior is subtle, add characterization tests that pin
the pre-change output first, then refactor, so the output is preserved byte for
byte.
* Audit the spellcheck and extension normalizers for regex-on-text paths;
convert the clear cases and file follow-ups for anything larger.
h2. Acceptance criteria
* The four normalizers use cursor scans on the text path; no regex is applied
to user text in them.
* Output-preserving: characterization tests capture the prior behavior and pass
unchanged after the refactor.
* The spellcheck and extension audit is complete, with conversions applied or
follow-ups filed.
h2. Out of scope
* Making these rungs offset-aware. They are not in the blessed chain, and
offset-safety for them is a separate concern.
* The {{StringUtil}} whitespace migration, which is a separate ticket.
* Regex in config parsing and non-user-text utilities.
* Adding or removing any of these rungs from a default chain.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
