krickert opened a new pull request, #1151: URL: https://github.com/apache/opennlp/pull/1151
Replaces the regex passes in the four legacy normalizers (Twitter, Url, Shrink, Number) with single forward cursor scans on the OPENNLP-1850 `CharClass`/`CodePointSet` primitives, output byte for byte. **Method:** characterization tests pinning the regex behavior (typical, edge, supplementary-plane, and pathological inputs, probed against the original implementations) were committed before each refactor, and each suite carries a seeded randomized differential check against the former patterns. **Fidelity:** the scans reproduce the engine's exact semantics where they leak into output: ASCII `\s`/`\d`/case-insensitive folding, the JDK `\b` word-boundary rules including the non-spacing-mark clause, code-point backreference matching around lone surrogates, and the mail regex's lookbehind and domain backtracking. These rungs feed the language detector factory chain, so output preservation keeps model behavior stable. **Also in this PR:** the spellcheck and extension normalizers were audited; punctuation peeling, the number-like guard, and the token-stream delimiter split are converted. The spellcheck URL-like guard stays a regex with a justification comment and is filed as a follow-up. Null text now fails loud with `IllegalArgumentException` instead of an undocumented NPE (documented and tested). Verification: opennlp-runtime 1324/0, tools 265, api 286, spellcheck 114, uima 49, all green. https://issues.apache.org/jira/browse/OPENNLP-1876 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
