krickert commented on PR #1150: URL: https://github.com/apache/opennlp/pull/1150#issuecomment-4954861780
Thanks, all three are in. **A)** Dropped the per-call-site justification comments. Kept and trimmed to a sentence the notes that document a real invariant: the frozen-predicate rationale on `StringUtil.isWhitespace` and the two context generators, the BERT pin in `BertNormalization`, and a one-line "White_Space since 3.0" note in the javadoc of `WhitespaceTokenizer`, `SimpleTokenizer`, and `Span#trim`. The JIRA key and the U+xxxx enumerations are out of the source. The `SimpleTokenizer` keepNewLines fix comment stays as one line. **B)** Added `StringUtil.isUnicodeWhitespace(int)` and the `char` overload, both delegating to `UnicodeWhitespace`, and switched the migrated call sites (`WhitespaceTokenizer`, `SimpleTokenizer`, `Span#trim`, spellcheck, UIMA `NumberUtil`) to them. `UnicodeWhitespace` stays the reference-data home, and `CharClass.whitespace()` in `SentenceDetectorME` is left as is. **C)** Renamed the PR and the issue to "Align whitespace handling with the Unicode White_Space property". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
