krickert commented on PR #1150: URL: https://github.com/apache/opennlp/pull/1150#issuecomment-4959817643
Yes, I reverted them. I'd moved the general tokenizers to Unicode `White_Space`, which shifted the `SourceForgeModelEval` fingerprints, and since that changes trained-model tokenization I reverted `SimpleTokenizer`, `WhitespaceTokenizer`, `Span.trim` and `SentenceDetectorME` back to current behavior. They're byte-identical to main now, so the eval needs no changes and existing models are unaffected. `White_Space` applies only to the user-text paths that don't feed models: the spellcheck normalizer, `SymSpell` splitting, and the UIMA number normalizer. Honestly I'd lean toward just moving the tokenizers to `White_Space` for 3.0 and updating the eval baselines, rather than carrying two definitions. It's a major version, so a reasonable time to standardize. It was simple to keep both - and just as simple to update the fingerprints. Should I switch it back and refresh the fingerprints? Or keep the legacy behavior for this release if you'd rather not change model tokenization. Your call. I'm open either way... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
