krickert commented on code in PR #1150:
URL: https://github.com/apache/opennlp/pull/1150#discussion_r3593784583
##########
opennlp-docs/src/docbkx/introduction.xml:
##########
@@ -335,6 +335,29 @@ Arguments description:
the interface 'StringInterner' and specify this class via
'opennlp.interner.class'.
</para>
</section>
+ <section xml:id="intro.sysprops.whitespace">
+ <title>Whitespace Definition</title>
+ <para>
+ Tokenization, corpus format parsing, and feature generation
classify whitespace
+ using the Unicode <code>White_Space</code> property by
default. A small set of
Review Comment:
Reworded to say models trained with 1.x/2.x used the legacy definition, and
as of this release that covers all published OpenNLP models. Also expanded the
section a bit on why Unicode `White_Space` (25 code points) and the JVM/`\s`
mismatch matter, with a pointer to the Wikipedia whitespace page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
