krickert commented on PR #1150: URL: https://github.com/apache/opennlp/pull/1150#issuecomment-4965186551
@jzonthemtn @rzo1 Pushed the compat flag in 61498f6f. Jeff - thanks for the heads up - the failure was real and I went with the neutral stance: 1. Default to unicode 2. Legacy flag at startup 3. Test pin legacy, 13 pass as before Quick version: whitespace defaults to the Unicode `White_Space` set, and `-Dopennlp.whitespace.mode=LEGACY` restores the exact 1.x/2.x definition process-wide, since the switch sits inside `StringUtil.isWhitespace` and covers every consumer at once. One-time warning when LEGACY is selected, scheduled for deletion in 4.0 (one class, one branch, no call-site changes), documented in the manual next to the other system properties. For pre-3.0 models: on default settings they shift at exactly five code points (NEL becomes a boundary, U+001C-U+001F stop being one); under LEGACY they're byte-identical. `SourceForgeModelEval` pins LEGACY, the training-data fingerprint from main stands untouched, and the full eval suite passes locally against the SourceForge models, 13 for 13. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
