krickert commented on PR #1150: URL: https://github.com/apache/opennlp/pull/1150#issuecomment-4958610871
Yes, a pre-3.0 tokenizer model can behave differently under 3.0, but only for the five code points where the two definitions differ. `TokenizerME` uses `WhitespaceTokenizer` to establish hard candidate boundaries before consulting the model: U+0085 (`NEL`) becomes a boundary, while U+001C-U+001F no longer are. Concretely, text containing `NEL` now splits where it previously did not, and text joined by the `C0` separator controls (rare in practice) now reaches the model as one candidate instead of being force-split. The model feature generator retains the legacy predicate, so its serialized feature semantics remain compatible, and ordinary whitespace is unchanged. What this gains is one standards-based definition for user text. The JDK has no single Character predicate that matches the Unicode `White_Space` property: `Character.isWhitespace()` omits `NEL` and the no-break spaces while including U+001C-U+001F, and `Character.isSpaceChar()` omits control whitespace such as tab and newline. Using the explicit Unicode property removes that ambiguity and makes tokenization consistent across the affected OpenNLP APIs. What motivated me: thousands of PDFs I've parsed carry unusual whitespace in their extracted text. They were always edge cases and at times caused poison documents: `NEL` line breaks the tokenizer didn't split on, and the typographic spaces from newspaper and book layouts that some code paths didn't recognize as whitespace at all. With every API on the same `White_Space` predicate, those documents tokenize consistently instead of choking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
