krickert opened a new pull request, #1150: URL: https://github.com/apache/opennlp/pull/1150
UCD-backs the whitespace predicates outside the OPENNLP-1850 engine (OPENNLP-1852 consistency item; generalizes the OPENNLP-205 correction). Every `StringUtil.isWhitespace`, `Character.isWhitespace`, `Character.isSpaceChar`, and `\s`-split site outside the 1850 engine was audited and classified: - **Migrated to the UCD `White_Space` set:** both rule-based tokenizers, `Span.trim`, the sentence detector's abbreviation guard, the stopword and spellcheck CLIs, `SymSpell.lookupCompound`, the spellcheck `PER_TOKEN` normalizer, and the UIMA `NumberUtil`. - **Frozen with documented rationale and permanent pins:** trained-model feature generation and the BERT reference definition (model stability). - **Kept JVM/ASCII semantics with per-site justification comments:** format parsers, where the format fixes the definition. - `StringUtil.isWhitespace` itself stays frozen as the documented legacy predicate, since its remaining callers are exactly the frozen sites; user-text code is directed to `UnicodeWhitespace`. Method is characterize-then-refactor: the first commit pins pre-change behavior at NBSP, figure and narrow space, U+2028/U+2029, NEL, and U+001C..U+001F for every touched site, so the migration commits flip only the deliberate deltas. This is a behavior change; the release-note text is in the JIRA issue. Verification: opennlp-runtime 1313/0, plus green suites in api, formats, cli, tools, uima, and spellcheck. https://issues.apache.org/jira/browse/OPENNLP-1875 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
