krickert commented on PR #1150: URL: https://github.com/apache/opennlp/pull/1150#issuecomment-4940132905
Thanks. Addressed in 9fd57a85, one pushback: - Indirect sample-parser sites: each now carries the same per-site justification comment as the frozen direct call sites, and characterization tests pin the new behavior where it is deliberate: a NameSample, POSSample, and TokenSample line separated by U+0085 parses identically to its ASCII-space equivalent, token indices and spans included. The brat, OntoNotes, and doccat streams got the comment treatment; their separators are format-fixed ASCII in practice, and the comments say so. - keepNewLines and NEL/LS/PS: fixed, not just documented. The line separator set under keepNewLines is now the ASCII pair plus U+0085, U+2028, and U+2029 in both WhitespaceTokenizer and SimpleTokenizer, so those code points are emitted as newline tokens instead of silently dropped; the SimpleTokenizer path also anchors the emitted span at the separator's own position. Javadoc states the emitted set, tests pin it with keepNewLines enabled. - SymSpell compound distance: the distance path now trims with the same Unicode White_Space set the split uses; an NBSP-only input reports distance 0 like an ASCII-space-only input, tested. - CorrectTextTool blank guard: a whitespace-only line now emits an empty output line instead of disappearing; guarded with the same Unicode definition as tokenization, tested through the command line tool. - SentenceDetectorME.isAcceptableBreak and NumberUtil.parse: both carry the since-3.0 Unicode White_Space javadoc note the other migrated public sites received. - cp(int) duplication: kept as-is. The three-line helper names the intent at call sites full of hex literals, and consolidating across modules would need a shared test artifact for a helper that small; the duplication predates this PR across the suite and is better swept in one test-wide cleanup than piecemeal here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
