This is an automated email from the ASF dual-hosted git repository.
mawiesne pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 2ae4146b9 OPENNLP-1875: Align whitespace handling with the Unicode
White_Space property (#1150)
2ae4146b9 is described below
commit 2ae4146b999bb3c4091cb0497d0b3a321e4fb66c
Author: Kristian Rickert <[email protected]>
AuthorDate: Fri Jul 17 23:14:04 2026 -0400
OPENNLP-1875: Align whitespace handling with the Unicode White_Space
property (#1150)
* Migrate CLI, spellcheck and UIMA whitespace splits to Unicode White_Space
= Replace the ASCII \s regex whitespace handling on the remaining
user-text paths with the standards-sourced Unicode White_Space set from
OPENNLP-1850:
Behavior delta, pinned by the flipped characterization tests: for the
regex sites the change is purely additive (NBSP, figure/narrow space,
NEL, the Unicode line and paragraph separators and the other Zs
separators now separate or strip; U+001C..U+001F never matched \s and
still do not split). For the PER_TOKEN normalizer, which used
Character.isWhitespace, U+001C..U+001F additionally stop being token
boundaries. None of these components feeds trained-model features.
* SimpleTokenizer now splits on the Unicode White_Space set, so the NEL
(U+0085) and C0 separator (U+001C-U+001F) characters in the eng_news_2010_300K
corpus retokenize. Update the pinned MD5 in verifyTrainingData to match. The
accuracy evals and the sentence-model digest are unchanged.
* Narrow to user-text paths; keep the trained-model tokenizers on legacy
whitespace
* The general tokenizers (SimpleTokenizer, WhitespaceTokenizer), Span.trim
and SentenceDetectorME feed trained models, so moving them to the Unicode
White_Space set changed their tokenization on the NEL and C0-separator code
points and shifted
every SourceForgeModelEval fingerprint. Revert those to main so existing
models are unaffected, and apply White_Space only to the genuine user-text
paths: the spellcheck normalizer and SymSpell splitting, and the UIMA number
normalizer. StringUtil gains
isUnicodeWhitespace as the standard user-text predicate; the legacy
isWhitespace stays for trained-model feature generation.
* Switch whitespace classification to Unicode White_Space with a legacy
compatibility mode
* StringUtil.isWhitespace now resolves through the new WhitespaceMode: the
Unicode White_Space set by default, or the 1.x/2.x definition
(Character.isWhitespace plus the Zs category) when the opennlp.whitespace.mode
system property selects LEGACY.
Every caller shares that one decision, so the general tokenizers,
Span.trim, corpus parsing and the context generators follow White_Space by
default, and models trained
under the legacy definition decode identically by selecting LEGACY. The
mode resolves once at class initialization and applies process-wide; setActive
and reset let tests
and embedders override it, selecting LEGACY logs a one-time removal
warning, and the mode is scheduled for removal in 4.0. SourceForgeModelEval and
the sentence-span characterization tests pin LEGACY for their legacy-trained
models.
* Documented in the
manual next to the other system properties.
---
.../main/java/opennlp/tools/util/StringUtil.java | 173 +++++-
.../java/opennlp/tools/util/WhitespaceMode.java | 134 +++++
.../opennlp/tools/util/WhitespaceModeTest.java | 130 +++++
.../tools/cmdline/stopword/StopwordFilterTool.java | 5 +-
.../sentdetect/DefaultSDContextGenerator.java | 6 +-
.../tools/stopword/DictionaryStopwordFilter.java | 9 +-
.../tokenize/DefaultTokenContextGenerator.java | 6 +
.../SentenceDetectorMESpanMappingTest.java | 14 +
.../tokenize/DefaultTokenContextGeneratorTest.java | 80 +++
.../tools/tokenize/SimpleTokenizerTest.java | 46 ++
.../tools/tokenize/WhitespaceTokenizerTest.java | 46 ++
opennlp-docs/src/docbkx/introduction.xml | 50 ++
opennlp-docs/src/docbkx/tokenizer.xml | 5 +
.../opennlp/tools/eval/SourceForgeModelEval.java | 19 +
.../spellcheck/cmdline/CorrectTextTool.java | 6 +-
.../SpellCheckingCharSequenceNormalizer.java | 19 +-
.../java/opennlp/spellcheck/symspell/SymSpell.java | 9 +-
.../SpellCheckingCharSequenceNormalizerTest.java | 48 ++
.../spellcheck/symspell/SymSpellCompoundTest.java | 64 +++
.../java/opennlp/uima/normalizer/NumberUtil.java | 26 +-
.../opennlp/uima/normalizer/NumberUtilTest.java | 32 ++
.../java/opennlp/tools/util/StringUtilTest.java | 623 ++++++++++++++++++++-
22 files changed, 1499 insertions(+), 51 deletions(-)
diff --git a/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
b/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
index c21020e28..5de9ce6b5 100644
--- a/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
+++ b/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
@@ -18,55 +18,182 @@
package opennlp.tools.util;
import java.nio.CharBuffer;
+import java.util.ArrayList;
+import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import opennlp.tools.util.normalizer.UnicodeWhitespace;
+
public class StringUtil {
private static final Logger logger =
LoggerFactory.getLogger(StringUtil.class);
/**
- * Determines if the specified {@link Character} is a whitespace.
- * A character is considered a whitespace when one of the following
conditions is met:
- * <ul>
- * <li>It's a {@link Character#isWhitespace(int)} whitespace.</li>
- * <li>It's a part of the Unicode Zs category ({@link
Character#SPACE_SEPARATOR}).</li>
- * </ul>
+ * Determines if the specified {@link Character} is a whitespace under the
active
+ * {@link WhitespaceMode}: the Unicode {@code White_Space} property by
default, or, when
+ * the {@value WhitespaceMode#MODE_PROPERTY} system property selects
+ * {@link WhitespaceMode#LEGACY}, the union of {@link
Character#isWhitespace(int)} and the
+ * Unicode {@code Zs} category ({@link Character#SPACE_SEPARATOR}) that
OpenNLP 1.x and 2.x
+ * used.
*
- * {@link Character#isWhitespace(int)} does not include no-break spaces.
- * In OpenNLP no-break spaces are also considered as white spaces.
+ * <p>Tokenization, corpus format parsing, and feature generation all
resolve whitespace
+ * through this method, so they share one {@link WhitespaceMode} for the
life of the
+ * process; use {@link #isUnicodeWhitespace(char)} directly where the
Unicode definition
+ * should apply unconditionally, such as user-text normalization with no
trained-model
+ * dependency.</p>
*
* @param charCode The character to check.
- *
- * @return {@code true} if {@code charCode} represents a white space, {@code
false} otherwise.
+ *
+ * @return {@code true} if {@code charCode} represents a white space under
the active
+ * {@link WhitespaceMode}, {@code false} otherwise.
*/
public static boolean isWhitespace(char charCode) {
- return Character.isWhitespace(charCode) ||
+ return WhitespaceMode.current() == WhitespaceMode.LEGACY
+ ? isLegacyWhitespace(charCode) : isUnicodeWhitespace(charCode);
+ }
+
+ /**
+ * Determines if the specified code point is a whitespace under the active
+ * {@link WhitespaceMode}; see {@link #isWhitespace(char)} for details.
+ *
+ * @param charCode An int representation of a character to check.
+ *
+ * @return {@code true} if {@code charCode} represents a white space under
the active
+ * {@link WhitespaceMode}, {@code false} otherwise.
+ */
+ public static boolean isWhitespace(int charCode) {
+ return WhitespaceMode.current() == WhitespaceMode.LEGACY
+ ? isLegacyWhitespace(charCode) : isUnicodeWhitespace(charCode);
+ }
+
+ /**
+ * The OpenNLP 1.x/2.x definition: {@link Character#isWhitespace(int)} or
the Unicode
+ * {@code Zs} category.
+ */
+ private static boolean isLegacyWhitespace(int charCode) {
+ return Character.isWhitespace(charCode) ||
Character.getType(charCode) == Character.SPACE_SEPARATOR;
}
/**
- * Determines if the specified {@link Character} is a whitespace.
- * A character is considered a whitespace when one of the following
conditions is met:
+ * Determines if the specified {@link Character} is a whitespace under the
Unicode
+ * {@code White_Space} property; delegates to
+ * {@link
opennlp.tools.util.normalizer.UnicodeWhitespace#isWhitespace(int)}. This is also
+ * what {@link #isWhitespace(char)} resolves to under the default {@link
WhitespaceMode}.
*
- * <ul>
- * <li>Its a {@link Character#isWhitespace(int)} whitespace.</li>
- * <li>Its a part of the Unicode Zs category ({@link
Character#SPACE_SEPARATOR}).</li>
- * </ul>
+ * @param charCode The character to check.
*
- * {@link Character#isWhitespace(int)} does not include no-break spaces.
- * In OpenNLP no-break spaces are also considered as white spaces.
+ * @return {@code true} if {@code charCode} has the {@code White_Space}
property,
+ * {@code false} otherwise.
+ */
+ public static boolean isUnicodeWhitespace(char charCode) {
+ return UnicodeWhitespace.isWhitespace(charCode);
+ }
+
+ /**
+ * Determines if the specified code point is a whitespace under the Unicode
+ * {@code White_Space} property; delegates to
+ * {@link
opennlp.tools.util.normalizer.UnicodeWhitespace#isWhitespace(int)}. This is also
+ * what {@link #isWhitespace(int)} resolves to under the default {@link
WhitespaceMode}.
*
* @param charCode An int representation of a character to check.
*
- * @return {@code true} if {@code charCode} represents a white space, {@code
false} otherwise.
+ * @return {@code true} if {@code charCode} has the {@code White_Space}
property,
+ * {@code false} otherwise.
*/
- public static boolean isWhitespace(int charCode) {
- return Character.isWhitespace(charCode) ||
- Character.getType(charCode) == Character.SPACE_SEPARATOR;
+ public static boolean isUnicodeWhitespace(int charCode) {
+ return UnicodeWhitespace.isWhitespace(charCode);
+ }
+
+ /**
+ * Splits {@code input} on runs of Unicode {@code White_Space}. Leading and
trailing
+ * runs are ignored, so whitespace-only input yields an empty array. This is
a
+ * code-point scan, not a regular expression.
+ *
+ * @param input The text to split. Must not be {@code null}.
+ * @return The non-whitespace terms in order.
+ * @throws IllegalArgumentException If {@code input} is {@code null}.
+ */
+ public static String[] splitOnUnicodeWhitespace(CharSequence input) {
+ if (input == null) {
+ throw new IllegalArgumentException("input must not be null");
+ }
+ final List<String> terms = new ArrayList<>();
+ final int n = input.length();
+ int start = -1;
+ int i = 0;
+ while (i < n) {
+ final int cp = Character.codePointAt(input, i);
+ if (isUnicodeWhitespace(cp)) {
+ if (start >= 0) {
+ terms.add(input.subSequence(start, i).toString());
+ start = -1;
+ }
+ } else if (start < 0) {
+ start = i;
+ }
+ i += Character.charCount(cp);
+ }
+ if (start >= 0) {
+ terms.add(input.subSequence(start, n).toString());
+ }
+ return terms.toArray(new String[0]);
}
+ /**
+ * Trims leading and trailing runs of Unicode {@code White_Space}, the same
set
+ * {@link #splitOnUnicodeWhitespace(CharSequence)} breaks terms on.
+ *
+ * @param input The text to trim. Must not be {@code null}.
+ * @return The trimmed string; may be empty when {@code input} is
whitespace-only.
+ * @throws IllegalArgumentException If {@code input} is {@code null}.
+ */
+ public static String trimUnicodeWhitespace(CharSequence input) {
+ if (input == null) {
+ throw new IllegalArgumentException("input must not be null");
+ }
+ int start = 0;
+ int end = input.length();
+ while (start < end) {
+ final int cp = Character.codePointAt(input, start);
+ if (!isUnicodeWhitespace(cp)) {
+ break;
+ }
+ start += Character.charCount(cp);
+ }
+ while (end > start) {
+ final int cp = Character.codePointBefore(input, end);
+ if (!isUnicodeWhitespace(cp)) {
+ break;
+ }
+ end -= Character.charCount(cp);
+ }
+ return input.subSequence(start, end).toString();
+ }
+
+ /**
+ * {@code true} when {@code input} is {@code null}, empty, or consists only
of Unicode
+ * {@code White_Space} code points.
+ *
+ * @param input The text to test; {@code null} is treated as blank.
+ * @return {@code true} if there is no non-whitespace code point.
+ */
+ public static boolean isUnicodeBlank(CharSequence input) {
+ if (input == null || input.length() == 0) {
+ return true;
+ }
+ int i = 0;
+ while (i < input.length()) {
+ final int cp = Character.codePointAt(input, i);
+ if (!isUnicodeWhitespace(cp)) {
+ return false;
+ }
+ i += Character.charCount(cp);
+ }
+ return true;
+ }
/**
* Converts a {@link CharSequence} to lower case, independent of the current
diff --git a/opennlp-api/src/main/java/opennlp/tools/util/WhitespaceMode.java
b/opennlp-api/src/main/java/opennlp/tools/util/WhitespaceMode.java
new file mode 100644
index 000000000..c359569c3
--- /dev/null
+++ b/opennlp-api/src/main/java/opennlp/tools/util/WhitespaceMode.java
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.util.Locale;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Selects the whitespace definition used by {@link
StringUtil#isWhitespace(char)} and
+ * {@link StringUtil#isWhitespace(int)}: the Unicode {@code White_Space}
property, or
+ * OpenNLP's legacy definition from 1.x/2.x.
+ * <p>
+ * Resolved from the {@value #MODE_PROPERTY} system property when this class
is initialized
+ * and shared process-wide, so a model is trained and decoded under one
definition. Tests and
+ * embedders may override the mode via {@link #setActive(WhitespaceMode)} and
{@link #reset()}.
+ *
+ * @since 3.0.0
+ */
+public enum WhitespaceMode {
+
+ /**
+ * OpenNLP 1.x/2.x whitespace: the union of {@link
Character#isWhitespace(int)} and the
+ * Unicode {@code Zs} category. Restores byte-identical tokenization, corpus
parsing, and
+ * feature generation for models trained under this definition.
+ */
+ LEGACY,
+
+ /**
+ * The Unicode {@code White_Space} property. The default from 3.0 onward.
+ */
+ UNICODE;
+
+ /**
+ * System property that selects the active {@link WhitespaceMode} at
startup. Accepts
+ * {@code LEGACY} or {@code UNICODE}, case-insensitive; unset or blank
resolves to
+ * {@link #UNICODE}, any other value raises an {@link
IllegalArgumentException} when the
+ * mode is resolved.
+ */
+ public static final String MODE_PROPERTY = "opennlp.whitespace.mode";
+
+ private static final Logger logger =
LoggerFactory.getLogger(WhitespaceMode.class);
+ private static final AtomicBoolean LEGACY_WARNED = new AtomicBoolean();
+
+ private static volatile WhitespaceMode active = fromProperty();
+
+ /**
+ * Returns the active {@link WhitespaceMode}: the value resolved from the
+ * {@value #MODE_PROPERTY} system property when this class was initialized,
or the value
+ * most recently passed to {@link #setActive(WhitespaceMode)}.
+ *
+ * @return The active {@link WhitespaceMode}.
+ */
+ public static WhitespaceMode current() {
+ return active;
+ }
+
+ /**
+ * Overrides the active {@link WhitespaceMode} for the whole process, taking
precedence
+ * over the {@value #MODE_PROPERTY} system property. Intended for tests and
embedders;
+ * callers pinning a mode temporarily should call {@link #reset()} afterward.
+ *
+ * @param mode The {@link WhitespaceMode} to activate. Must not be {@code
null}.
+ *
+ * @throws IllegalArgumentException If {@code mode} is {@code null}.
+ */
+ public static void setActive(WhitespaceMode mode) {
+ if (mode == null) {
+ throw new IllegalArgumentException("mode must not be null");
+ }
+ active = mode;
+ warnIfLegacy(mode);
+ }
+
+ /**
+ * Discards any override set via {@link #setActive(WhitespaceMode)} and
re-resolves the
+ * active mode from the {@value #MODE_PROPERTY} system property.
+ *
+ * @throws IllegalArgumentException If the property holds a value other than
+ * {@code LEGACY} or {@code UNICODE} (case-insensitive); the previous
mode is retained.
+ */
+ public static void reset() {
+ active = fromProperty();
+ }
+
+ /**
+ * Resolves the mode from the {@value #MODE_PROPERTY} system property; unset
or blank
+ * resolves to {@link #UNICODE}. Warns once per process when {@link #LEGACY}
is selected.
+ */
+ private static WhitespaceMode fromProperty() {
+ String value = System.getProperty(MODE_PROPERTY);
+ WhitespaceMode mode;
+ if (value == null || value.isBlank()) {
+ mode = UNICODE;
+ } else {
+ try {
+ mode = WhitespaceMode.valueOf(value.trim().toUpperCase(Locale.ROOT));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Invalid value '" + value + "' for
system property '"
+ + MODE_PROPERTY + "': expected LEGACY or UNICODE", e);
+ }
+ }
+ warnIfLegacy(mode);
+ return mode;
+ }
+
+ /**
+ * Logs the legacy-mode removal warning, once per process.
+ */
+ private static void warnIfLegacy(WhitespaceMode mode) {
+ if (mode == LEGACY && LEGACY_WARNED.compareAndSet(false, true)) {
+ logger.warn("Using the legacy (pre-3.0) whitespace definition for
tokenization, corpus " +
+ "format parsing, and feature generation. This compatibility mode is
scheduled for " +
+ "removal in 4.0.");
+ }
+ }
+}
diff --git
a/opennlp-api/src/test/java/opennlp/tools/util/WhitespaceModeTest.java
b/opennlp-api/src/test/java/opennlp/tools/util/WhitespaceModeTest.java
new file mode 100644
index 000000000..f3bc41b5b
--- /dev/null
+++ b/opennlp-api/src/test/java/opennlp/tools/util/WhitespaceModeTest.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.util;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for the {@link WhitespaceMode} class.
+ */
+public class WhitespaceModeTest {
+
+ /**
+ * Initializes {@link WhitespaceMode} while the property is unset, so tests
that set an
+ * invalid value exercise {@link WhitespaceMode#reset()} rather than class
initialization.
+ */
+ @BeforeAll
+ static void initializeWithCleanProperty() {
+ System.clearProperty(WhitespaceMode.MODE_PROPERTY);
+ WhitespaceMode.reset();
+ }
+
+ /**
+ * Restores property resolution after each test, so no mode or property
state leaks.
+ */
+ @AfterEach
+ void resetWhitespaceMode() {
+ System.clearProperty(WhitespaceMode.MODE_PROPERTY);
+ WhitespaceMode.reset();
+ }
+
+ @Test
+ void testDefaultsToUnicode() {
+ WhitespaceMode.reset();
+ assertEquals(WhitespaceMode.UNICODE, WhitespaceMode.current());
+ }
+
+ @Test
+ void testBlankPropertyDefaultsToUnicode() {
+ System.setProperty(WhitespaceMode.MODE_PROPERTY, " ");
+ WhitespaceMode.reset();
+ assertEquals(WhitespaceMode.UNICODE, WhitespaceMode.current());
+ }
+
+ @Test
+ void testPropertyResolvesLegacy() {
+ System.setProperty(WhitespaceMode.MODE_PROPERTY, "LEGACY");
+ WhitespaceMode.reset();
+ assertEquals(WhitespaceMode.LEGACY, WhitespaceMode.current());
+ }
+
+ @Test
+ void testPropertyResolvesCaseInsensitively() {
+ System.setProperty(WhitespaceMode.MODE_PROPERTY, "legacy");
+ WhitespaceMode.reset();
+ assertEquals(WhitespaceMode.LEGACY, WhitespaceMode.current());
+ }
+
+ @Test
+ void testPropertyResolvesUnicodeExplicitly() {
+ System.setProperty(WhitespaceMode.MODE_PROPERTY, "unicode");
+ WhitespaceMode.reset();
+ assertEquals(WhitespaceMode.UNICODE, WhitespaceMode.current());
+ }
+
+ @Test
+ void testInvalidPropertyValueThrows() {
+ WhitespaceMode before = WhitespaceMode.current();
+ System.setProperty(WhitespaceMode.MODE_PROPERTY, "sloppy");
+ IllegalArgumentException e = assertThrows(IllegalArgumentException.class,
WhitespaceMode::reset);
+ assertTrue(e.getMessage().contains(WhitespaceMode.MODE_PROPERTY));
+ assertTrue(e.getMessage().contains("sloppy"));
+ // The previous mode is retained when re-resolution fails.
+ assertEquals(before, WhitespaceMode.current());
+ }
+
+ @Test
+ void testCurrentCachesUntilReset() {
+ WhitespaceMode.reset();
+ assertEquals(WhitespaceMode.UNICODE, WhitespaceMode.current());
+
+ // The property is only read during resolution; setting it later has no
effect until reset().
+ System.setProperty(WhitespaceMode.MODE_PROPERTY, "LEGACY");
+ assertEquals(WhitespaceMode.UNICODE, WhitespaceMode.current());
+
+ WhitespaceMode.reset();
+ assertEquals(WhitespaceMode.LEGACY, WhitespaceMode.current());
+ }
+
+ @Test
+ void testSetActiveOverridesProperty() {
+ System.setProperty(WhitespaceMode.MODE_PROPERTY, "UNICODE");
+ WhitespaceMode.reset();
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+ assertEquals(WhitespaceMode.LEGACY, WhitespaceMode.current());
+ }
+
+ @Test
+ void testSetActiveRejectsNull() {
+ assertThrows(IllegalArgumentException.class, () ->
WhitespaceMode.setActive(null));
+ }
+
+ @Test
+ void testResetReturnsToPropertyResolution() {
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+ assertEquals(WhitespaceMode.LEGACY, WhitespaceMode.current());
+
+ WhitespaceMode.reset();
+ assertEquals(WhitespaceMode.UNICODE, WhitespaceMode.current());
+ }
+}
diff --git
a/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/stopword/StopwordFilterTool.java
b/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/stopword/StopwordFilterTool.java
index 39700dfc5..9391583e3 100644
---
a/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/stopword/StopwordFilterTool.java
+++
b/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/stopword/StopwordFilterTool.java
@@ -34,6 +34,7 @@ import opennlp.tools.cmdline.CLI;
import opennlp.tools.cmdline.TerminateToolException;
import opennlp.tools.stopword.StopwordFilter;
import opennlp.tools.stopword.StopwordLists;
+import opennlp.tools.util.StringUtil;
/**
* A command line tool that filters stop words from whitespace-separated
@@ -85,11 +86,11 @@ public final class StopwordFilterTool extends
BasicCmdLineTool {
String line;
while ((line = reader.readLine()) != null) {
- if (line.isEmpty()) {
+ final String[] tokens = StringUtil.splitOnUnicodeWhitespace(line);
+ if (tokens.length == 0) {
writer.println();
continue;
}
- final String[] tokens = line.split("\\s+");
final String[] kept = filter.filter(tokens);
writer.println(String.join(" ", kept));
}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/DefaultSDContextGenerator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/DefaultSDContextGenerator.java
index afb2066aa..3a57d965c 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/DefaultSDContextGenerator.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/DefaultSDContextGenerator.java
@@ -27,7 +27,11 @@ import opennlp.tools.util.StringUtil;
/**
* Generate event contexts for maxent decisions for sentence detection.
- *
+ * <p>
+ * Features reflect {@link StringUtil#isWhitespace(char)}, whose result
depends on the
+ * active {@link opennlp.tools.util.WhitespaceMode}; they are part of every
trained
+ * sentence detector model, so training and decoding must use the same
+ * {@link opennlp.tools.util.WhitespaceMode}.
*/
public class DefaultSDContextGenerator implements SDContextGenerator {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stopword/DictionaryStopwordFilter.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stopword/DictionaryStopwordFilter.java
index 13d857591..b144ca5c4 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stopword/DictionaryStopwordFilter.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stopword/DictionaryStopwordFilter.java
@@ -34,6 +34,7 @@ import java.util.Set;
import opennlp.tools.commons.ThreadSafe;
import opennlp.tools.dictionary.Dictionary;
import opennlp.tools.util.StringList;
+import opennlp.tools.util.StringUtil;
/**
* An immutable, thread-safe {@link StopwordFilter} backed by an OpenNLP
@@ -254,11 +255,11 @@ public final class DictionaryStopwordFilter implements
StopwordFilter {
BufferedReader lineReader = new BufferedReader(reader)) {
String line;
while ((line = lineReader.readLine()) != null) {
- final String trimmed = line.trim();
+ final String trimmed = StringUtil.trimUnicodeWhitespace(line);
if (trimmed.isEmpty() || trimmed.startsWith(COMMENT_PREFIX)) {
continue;
}
- final String[] tokens = trimmed.split("\\s+");
+ final String[] tokens = StringUtil.splitOnUnicodeWhitespace(trimmed);
if (tokens.length > 0) {
dict.put(new StringList(tokens));
}
@@ -390,11 +391,11 @@ public final class DictionaryStopwordFilter implements
StopwordFilter {
BufferedReader lineReader = new BufferedReader(reader)) {
String line;
while ((line = lineReader.readLine()) != null) {
- final String trimmed = line.trim();
+ final String trimmed = StringUtil.trimUnicodeWhitespace(line);
if (trimmed.isEmpty() || trimmed.startsWith(COMMENT_PREFIX)) {
continue;
}
- addEntries.add(trimmed.split("\\s+"));
+ addEntries.add(StringUtil.splitOnUnicodeWhitespace(trimmed));
}
}
return this;
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/DefaultTokenContextGenerator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/DefaultTokenContextGenerator.java
index f334e2872..01a6c23de 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/DefaultTokenContextGenerator.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/DefaultTokenContextGenerator.java
@@ -114,6 +114,12 @@ public class DefaultTokenContextGenerator implements
TokenContextGenerator {
* a fixed text sequence depending on {@code c}. The resulting combination
is added
* to the given list {@code preds}.
*
+ * <p>
+ * The {@code _ws} class uses {@link StringUtil#isWhitespace(char)}, whose
result depends
+ * on the active {@link opennlp.tools.util.WhitespaceMode}; the feature
strings it produces
+ * are part of every trained tokenizer model, so training and decoding must
use the same
+ * {@link opennlp.tools.util.WhitespaceMode}.
+ *
* @param key The input string to process.
* @param c A character used to discriminate which fixed text shall be
appended.
* @param preds The list into which the resulting combinations will be added.
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentdetect/SentenceDetectorMESpanMappingTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentdetect/SentenceDetectorMESpanMappingTest.java
index d081c1839..aa3f5fd70 100644
---
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentdetect/SentenceDetectorMESpanMappingTest.java
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/sentdetect/SentenceDetectorMESpanMappingTest.java
@@ -21,12 +21,14 @@ import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
+import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import opennlp.tools.dictionary.Dictionary;
import opennlp.tools.util.Span;
+import opennlp.tools.util.WhitespaceMode;
/**
* Characterization tests for the end-of-sentence position to {@link Span}
mapping of
@@ -43,6 +45,9 @@ public class SentenceDetectorMESpanMappingTest extends
AbstractSentenceDetectorT
@BeforeAll
static void prepareResources() throws IOException {
+ // DefaultSDContextGenerator's features depend on the active
WhitespaceMode, so training
+ // must be pinned to WhitespaceMode.LEGACY to reproduce the expectations
pinned below.
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
Dictionary abb = loadAbbDictionary(Locale.ENGLISH);
tokenEnd = new SentenceDetectorME(
train(new SentenceDetectorFactory("eng", true, abb, null),
Locale.ENGLISH));
@@ -305,4 +310,13 @@ public class SentenceDetectorMESpanMappingTest extends
AbstractSentenceDetectorT
assertSpans(tokenEnd, input, new Span(0, 5), new Span(6, 7));
assertSpans(noTokenEnd, input, new Span(0, 5), new Span(6, 7));
}
+
+ /**
+ * Reverts the {@link WhitespaceMode} pin, so later test classes sharing
this JVM fork
+ * resolve the mode from the system property again.
+ */
+ @AfterAll
+ static void resetWhitespaceMode() {
+ WhitespaceMode.reset();
+ }
}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/DefaultTokenContextGeneratorTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/DefaultTokenContextGeneratorTest.java
new file mode 100644
index 000000000..326487ffe
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/DefaultTokenContextGeneratorTest.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.tokenize;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import opennlp.tools.util.WhitespaceMode;
+
+/**
+ * Tests for the {@link DefaultTokenContextGenerator} class.
+ */
+public class DefaultTokenContextGeneratorTest {
+
+ private final DefaultTokenContextGenerator cg = new
DefaultTokenContextGenerator();
+
+ /**
+ * Restores {@link WhitespaceMode} property resolution after each test, so
no mode
+ * state leaks.
+ */
+ @AfterEach
+ void resetWhitespaceMode() {
+ WhitespaceMode.reset();
+ }
+
+ /**
+ * The {@code _ws} feature suffix (whitespace class; underscore is only a
name separator,
+ * as in {@code f1_ws}) follows the active {@link WhitespaceMode}. These
features are part
+ * of every trained tokenizer model, so this pins the default behavior at
the code points
+ * where the modes disagree: the next line control ({@code U+0085}) is
classified as
+ * whitespace, the information separator ({@code U+001C}) is not.
+ */
+ @Test
+ void testNextLineControlProducesWhitespaceFeatureByDefault() {
+ char nextLine = (char) 0x0085;
+ List<String> preds = Arrays.asList(cg.getContext("a" + nextLine + "b", 1));
+ Assertions.assertTrue(preds.contains("f1_ws"), preds.toString());
+
+ char infoSeparator = (char) 0x001C;
+ preds = Arrays.asList(cg.getContext("a" + infoSeparator + "b", 1));
+ Assertions.assertFalse(preds.contains("f1_ws"), preds.toString());
+ }
+
+ /**
+ * Under {@link WhitespaceMode#LEGACY} the classification flips: the
information separator
+ * ({@code U+001C}) is whitespace and the next line control ({@code U+0085})
is not,
+ * reproducing the features of OpenNLP 1.x/2.x tokenizer models.
+ */
+ @Test
+ void testInformationSeparatorProducesWhitespaceFeatureUnderLegacyMode() {
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+
+ char infoSeparator = (char) 0x001C;
+ List<String> preds = Arrays.asList(cg.getContext("a" + infoSeparator +
"b", 1));
+ Assertions.assertTrue(preds.contains("f1_ws"), preds.toString());
+
+ char nextLine = (char) 0x0085;
+ preds = Arrays.asList(cg.getContext("a" + nextLine + "b", 1));
+ Assertions.assertFalse(preds.contains("f1_ws"), preds.toString());
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/SimpleTokenizerTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/SimpleTokenizerTest.java
index d56d02d41..7ea9768d5 100644
---
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/SimpleTokenizerTest.java
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/SimpleTokenizerTest.java
@@ -17,9 +17,12 @@
package opennlp.tools.tokenize;
+import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import opennlp.tools.util.WhitespaceMode;
+
/**
* Tests for the {@link SimpleTokenizer} class.
*/
@@ -28,6 +31,15 @@ public class SimpleTokenizerTest {
// The SimpleTokenizer is thread safe
private final SimpleTokenizer mTokenizer = SimpleTokenizer.INSTANCE;
+ /**
+ * Restores {@link WhitespaceMode} property resolution after each test, so
no mode
+ * state leaks.
+ */
+ @AfterEach
+ void resetWhitespaceMode() {
+ WhitespaceMode.reset();
+ }
+
/**
* Tests if it can tokenize whitespace separated tokens.
*/
@@ -146,4 +158,38 @@ public class SimpleTokenizerTest {
Assertions.assertEquals("بنجاح", tokenizedText[3]);
Assertions.assertEquals(".", tokenizedText[4]);
}
+
+ /**
+ * Under the default {@link WhitespaceMode#UNICODE}, the next line control
({@code U+0085})
+ * separates tokens like whitespace, and the information separators ({@code
U+001C..U+001F})
+ * do not.
+ */
+ @Test
+ void testNextLineControlIsWhitespaceByDefault() {
+ char nextLine = (char) 0x0085;
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ mTokenizer.tokenize("a" + nextLine + "b"));
+
+ char infoSeparator = (char) 0x001C;
+ Assertions.assertArrayEquals(new String[] {"a",
String.valueOf(infoSeparator), "b"},
+ mTokenizer.tokenize("a" + infoSeparator + "b"));
+ }
+
+ /**
+ * Under {@link WhitespaceMode#LEGACY}, the information separators ({@code
U+001C..U+001F})
+ * separate tokens like whitespace, and the next line control ({@code
U+0085}) does not,
+ * matching OpenNLP 1.x/2.x tokenizer models.
+ */
+ @Test
+ void testInformationSeparatorsAreWhitespaceUnderLegacyMode() {
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+
+ char infoSeparator = (char) 0x001C;
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ mTokenizer.tokenize("a" + infoSeparator + "b"));
+
+ char nextLine = (char) 0x0085;
+ Assertions.assertArrayEquals(new String[] {"a", String.valueOf(nextLine),
"b"},
+ mTokenizer.tokenize("a" + nextLine + "b"));
+ }
}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/WhitespaceTokenizerTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/WhitespaceTokenizerTest.java
index 8dbcde68f..793f97b1a 100644
---
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/WhitespaceTokenizerTest.java
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/WhitespaceTokenizerTest.java
@@ -17,14 +17,26 @@
package opennlp.tools.tokenize;
+import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import opennlp.tools.util.WhitespaceMode;
+
/**
* Tests for the {@link WhitespaceTokenizer} class.
*/
public class WhitespaceTokenizerTest {
+ /**
+ * Restores {@link WhitespaceMode} property resolution after each test, so
no mode
+ * state leaks.
+ */
+ @AfterEach
+ void resetWhitespaceMode() {
+ WhitespaceMode.reset();
+ }
+
@Test
void testOneToken() {
Assertions.assertEquals("one",
WhitespaceTokenizer.INSTANCE.tokenize("one")[0]);
@@ -98,4 +110,38 @@ public class WhitespaceTokenizerTest {
Assertions.assertArrayEquals(new String[] {"a", "\r", "\n", "\r", "\n",
"b", "\r", "\n", "\r", "\n", "c"},
tokenizer.tokenize("a\r\n\r\n b\r\n\r\n c"));
}
+
+ /**
+ * Under the default {@link WhitespaceMode#UNICODE}, the next line control
({@code U+0085})
+ * separates tokens like whitespace, and the information separators ({@code
U+001C..U+001F})
+ * do not.
+ */
+ @Test
+ void testNextLineControlIsWhitespaceByDefault() {
+ char nextLine = (char) 0x0085;
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ WhitespaceTokenizer.INSTANCE.tokenize("a" + nextLine + "b"));
+
+ char infoSeparator = (char) 0x001C;
+ Assertions.assertArrayEquals(new String[] {"a" + infoSeparator + "b"},
+ WhitespaceTokenizer.INSTANCE.tokenize("a" + infoSeparator + "b"));
+ }
+
+ /**
+ * Under {@link WhitespaceMode#LEGACY}, the information separators ({@code
U+001C..U+001F})
+ * separate tokens like whitespace, and the next line control ({@code
U+0085}) does not,
+ * matching OpenNLP 1.x/2.x tokenizer models.
+ */
+ @Test
+ void testInformationSeparatorsAreWhitespaceUnderLegacyMode() {
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+
+ char infoSeparator = (char) 0x001C;
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ WhitespaceTokenizer.INSTANCE.tokenize("a" + infoSeparator + "b"));
+
+ char nextLine = (char) 0x0085;
+ Assertions.assertArrayEquals(new String[] {"a" + nextLine + "b"},
+ WhitespaceTokenizer.INSTANCE.tokenize("a" + nextLine + "b"));
+ }
}
diff --git a/opennlp-docs/src/docbkx/introduction.xml
b/opennlp-docs/src/docbkx/introduction.xml
index 51afe02a7..11a252942 100644
--- a/opennlp-docs/src/docbkx/introduction.xml
+++ b/opennlp-docs/src/docbkx/introduction.xml
@@ -335,6 +335,56 @@ Arguments description:
the interface 'StringInterner' and specify this class via
'opennlp.interner.class'.
</para>
</section>
+ <section xml:id="intro.sysprops.whitespace">
+ <title>Whitespace Definition</title>
+ <para>
+ What counts as whitespace is a common interoperability gap
across programming
+ languages and libraries. The Unicode Character Database
defines a binary
+ <code>White_Space</code> property. As documented on
+ <link
xlink:href="https://en.wikipedia.org/wiki/Whitespace_character";>Wikipedia's
+ whitespace character page</link>, Unicode defines exactly
twenty-five characters
+ with <code>White_Space=yes</code> (<quote>WSpace=Y</quote> /
<quote>WS</quote>),
+ including ordinary space and tab, the next-line control
(<code>U+0085</code>),
+ no-break and typographic spaces such as <code>U+00A0</code> and
+ <code>U+2007</code>, and the line and paragraph separators
+ (<code>U+2028</code>/<code>U+2029</code>). That page also
states the cross-language
+ weakness directly: <quote>Most languages only recognize
whitespace characters that
+ have an ASCII code. They disallow most or all of the Unicode
codes listed
+ above</quote>. Java is a concrete example of that mismatch:
+ <code>Character.isWhitespace</code> omits the no-break spaces
and
+ <code>U+0085</code> while including the information separators
+ <code>U+001C</code>..<code>U+001F</code>, and Java's default
<code>\s</code> regex
+ class is likewise an ASCII-oriented subset unless Unicode
character-class mode is
+ enabled. Text extracted from PDFs, HTML, and word processors
regularly carries the
+ wider set, so a tokenizer that follows the JVM predicate and
one that follows
+ Unicode will disagree on those documents.
+ </para>
+ <para>
+ Tokenization, corpus format parsing, and feature generation
therefore classify
+ whitespace using the Unicode <code>White_Space</code> property
by default, via
+ <code>StringUtil.isWhitespace</code> and the related
+ <code>StringUtil.splitOnUnicodeWhitespace</code> /
+ <code>trimUnicodeWhitespace</code> helpers (code-point scans,
not regular
+ expressions). Models trained with OpenNLP 1.x or 2.x used the
legacy definition
+ (the union of <code>Character.isWhitespace</code> and the
Unicode <code>Zs</code>
+ category). As of this release, that covers all published
OpenNLP models, so set
+ the property below when decoding those models if you need
byte-identical
+ tokenization at the few code points where the two definitions
differ.
+ </para>
+ <para>
+ Users may switch to the legacy definition by setting the
following system
+ property:
+ <screen>
+<![CDATA[-Dopennlp.whitespace.mode=LEGACY]]>
+ </screen>
+ </para>
+ <para>
+ Accepted values are <code>LEGACY</code> and
<code>UNICODE</code> (the default),
+ case-insensitive; any other value throws an
<code>IllegalArgumentException</code>.
+ The setting applies process-wide, so training and decoding of
a given model must
+ use the same value. Legacy support is scheduled for removal in
4.0.
+ </para>
+ </section>
</section>
</chapter>
diff --git a/opennlp-docs/src/docbkx/tokenizer.xml
b/opennlp-docs/src/docbkx/tokenizer.xml
index 7dd09a489..cd1d8a2dd 100644
--- a/opennlp-docs/src/docbkx/tokenizer.xml
+++ b/opennlp-docs/src/docbkx/tokenizer.xml
@@ -32,6 +32,11 @@
components apply that machinery automatically for
document chunking; see
<xref linkend="tools.normalizer.dl"/>.
</para>
+ <para>
+ Tokenizers classify whitespace using the Unicode
<code>White_Space</code> property by
+ default; see <xref
linkend="intro.sysprops.whitespace"/> for the system property that
+ restores OpenNLP's legacy definition for models trained
with 1.x or 2.x.
+ </para>
<para>
<screen>
<![CDATA[Pierre Vinken, 61 years old, will join the board as a nonexecutive
director Nov. 29.
diff --git
a/opennlp-eval-tests/src/test/java/opennlp/tools/eval/SourceForgeModelEval.java
b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/SourceForgeModelEval.java
index cf9f93213..ff444cfcc 100644
---
a/opennlp-eval-tests/src/test/java/opennlp/tools/eval/SourceForgeModelEval.java
+++
b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/SourceForgeModelEval.java
@@ -27,6 +27,7 @@ import java.util.Arrays;
import java.util.List;
import java.util.Objects;
+import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
@@ -61,6 +62,7 @@ import opennlp.tools.util.MarkableFileInputStreamFactory;
import opennlp.tools.util.ObjectStream;
import opennlp.tools.util.PlainTextByLineStream;
import opennlp.tools.util.Span;
+import opennlp.tools.util.WhitespaceMode;
/**
* This tests ensures that the existing SourceForge models perform
@@ -159,8 +161,25 @@ public class SourceForgeModelEval extends AbstractEvalTest
{
}
}
+ /**
+ * Reverts the {@link WhitespaceMode} override pinned in {@link
#verifyTrainingData()}, so
+ * later test classes sharing this JVM fork resolve the mode from the system
property again
+ * instead of inheriting this class's pin.
+ */
+ @AfterAll
+ static void resetWhitespaceMode() {
+ WhitespaceMode.reset();
+ }
+
+ /**
+ * Pins {@link WhitespaceMode#LEGACY} before any corpus access: the
SourceForge models
+ * evaluated here were trained under the legacy whitespace definition, so
decoding must use
+ * the same definition. The pin stays first in this method because JUnit
does not order
+ * multiple {@code @BeforeAll} methods.
+ */
@BeforeAll
static void verifyTrainingData() throws Exception {
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
verifyTrainingData(new LeipzigTestSampleStream(25,
SimpleTokenizer.INSTANCE,
new MarkableFileInputStreamFactory(new File(getOpennlpDataDir(),
"leipzig/eng_news_2010_300K-sentences.txt"))),
diff --git
a/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/cmdline/CorrectTextTool.java
b/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/cmdline/CorrectTextTool.java
index c628011d7..49f7fce5b 100644
---
a/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/cmdline/CorrectTextTool.java
+++
b/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/cmdline/CorrectTextTool.java
@@ -45,6 +45,7 @@ import opennlp.tools.cmdline.ArgumentParser;
import opennlp.tools.cmdline.BasicCmdLineTool;
import opennlp.tools.cmdline.CmdLineUtil;
import opennlp.tools.cmdline.TerminateToolException;
+import opennlp.tools.util.StringUtil;
/**
* A command line tool that corrects spelling in text using a {@link
SymSpellModel}.
@@ -136,11 +137,12 @@ public class CorrectTextTool extends BasicCmdLineTool {
final int effectiveMax = Math.min(maxEditDistance,
checker.maxEditDistance());
String line;
while ((line = in.readLine()) != null) {
- if (line.isBlank()) {
+ final String[] tokens = StringUtil.splitOnUnicodeWhitespace(line);
+ if (tokens.length == 0) {
out.newLine();
continue;
}
- for (String token : line.trim().split("\\s+")) {
+ for (String token : tokens) {
final List<SuggestItem> suggestions =
checker.lookup(token.toLowerCase(Locale.ROOT), verbosity,
effectiveMax);
final StringBuilder terms = new StringBuilder();
diff --git
a/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizer.java
b/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizer.java
index 42177e8c4..d5a1f5d27 100644
---
a/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizer.java
+++
b/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizer.java
@@ -25,6 +25,7 @@ import opennlp.spellcheck.SpellChecker;
import opennlp.spellcheck.SuggestItem;
import opennlp.spellcheck.Verbosity;
import opennlp.spellcheck.dictionary.SymSpellModel;
+import opennlp.tools.util.StringUtil;
import opennlp.tools.util.normalizer.AggregateCharSequenceNormalizer;
import opennlp.tools.util.normalizer.CharSequenceNormalizer;
@@ -35,7 +36,9 @@ import opennlp.tools.util.normalizer.CharSequenceNormalizer;
* <p>The normalizer works in one of two {@linkplain Mode modes}:</p>
* <ul>
* <li>{@link Mode#PER_TOKEN PER_TOKEN} (default) – the input is split
into
- * whitespace-delimited tokens and each token is corrected independently
with
+ * whitespace-delimited tokens (whitespace being the Unicode {@code
White_Space}
+ * set, {@link StringUtil#isUnicodeWhitespace(int)}) and each token is
corrected
+ * independently with
* {@link SpellChecker#lookup}. The original whitespace runs between
tokens are
* preserved verbatim, so the shape of the line is kept. Tokens the
dictionary
* already contains (best suggestion at edit distance {@code 0}) are left
@@ -102,6 +105,7 @@ public class SpellCheckingCharSequenceNormalizer implements
CharSequenceNormaliz
* {@link SpellChecker}.
*
* @param spellChecker the engine used to correct tokens; must not be {@code
null}
+ * @throws IllegalArgumentException if {@code spellChecker} is {@code null}
*/
public SpellCheckingCharSequenceNormalizer(SpellChecker spellChecker) {
this(builder(spellChecker));
@@ -113,6 +117,7 @@ public class SpellCheckingCharSequenceNormalizer implements
CharSequenceNormaliz
* engine}).
*
* @param model the loaded model whose engine is used; must not be {@code
null}
+ * @throws IllegalArgumentException if {@code model} is {@code null}
*/
public SpellCheckingCharSequenceNormalizer(SymSpellModel model) {
this(builder(model));
@@ -134,6 +139,7 @@ public class SpellCheckingCharSequenceNormalizer implements
CharSequenceNormaliz
/**
* @param spellChecker the engine to wrap; must not be {@code null}
* @return a new {@link Builder} seeded with sensible defaults
+ * @throws IllegalArgumentException if {@code spellChecker} is {@code null}
*/
public static Builder builder(SpellChecker spellChecker) {
if (spellChecker == null) {
@@ -145,6 +151,7 @@ public class SpellCheckingCharSequenceNormalizer implements
CharSequenceNormaliz
/**
* @param model the loaded model whose engine to wrap; must not be {@code
null}
* @return a new {@link Builder} seeded with sensible defaults
+ * @throws IllegalArgumentException if {@code model} is {@code null}
*/
public static Builder builder(SymSpellModel model) {
if (model == null) {
@@ -160,6 +167,7 @@ public class SpellCheckingCharSequenceNormalizer implements
CharSequenceNormaliz
*
* @param checker the engine to attach; must not be {@code null}
* @return a new, ready-to-use normalizer with this instance's settings
+ * @throws IllegalArgumentException if {@code checker} is {@code null}
*/
public SpellCheckingCharSequenceNormalizer withSpellChecker(SpellChecker
checker) {
return builder(checker)
@@ -223,10 +231,10 @@ public class SpellCheckingCharSequenceNormalizer
implements CharSequenceNormaliz
int i = 0;
final int n = input.length();
while (i < n) {
- // Copy a run of whitespace verbatim.
- if (Character.isWhitespace(input.charAt(i))) {
+ // Copy a run of whitespace (the Unicode White_Space set) verbatim.
+ if (StringUtil.isUnicodeWhitespace(input.charAt(i))) {
final int start = i;
- while (i < n && Character.isWhitespace(input.charAt(i))) {
+ while (i < n && StringUtil.isUnicodeWhitespace(input.charAt(i))) {
i++;
}
out.append(input, start, i);
@@ -234,7 +242,7 @@ public class SpellCheckingCharSequenceNormalizer implements
CharSequenceNormaliz
}
// Take a non-whitespace token and correct it.
final int start = i;
- while (i < n && !Character.isWhitespace(input.charAt(i))) {
+ while (i < n && !StringUtil.isUnicodeWhitespace(input.charAt(i))) {
i++;
}
out.append(correctToken(input.substring(start, i)));
@@ -433,6 +441,7 @@ public class SpellCheckingCharSequenceNormalizer implements
CharSequenceNormaliz
/**
* @param value the correction mode; must not be {@code null}
* @return this builder
+ * @throws IllegalArgumentException if {@code value} is {@code null}
*/
public Builder mode(Mode value) {
if (value == null) {
diff --git
a/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/symspell/SymSpell.java
b/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/symspell/SymSpell.java
index 52f9ec5e5..385788d40 100644
---
a/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/symspell/SymSpell.java
+++
b/opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/symspell/SymSpell.java
@@ -30,6 +30,7 @@ import opennlp.spellcheck.SpellChecker;
import opennlp.spellcheck.SuggestItem;
import opennlp.spellcheck.Verbosity;
import opennlp.spellcheck.distance.EditDistance;
+import opennlp.tools.util.StringUtil;
/**
* Symmetric Delete spelling correction engine (SymSpell).
@@ -394,6 +395,9 @@ public final class SymSpell implements SpellChecker {
return suggestions;
}
+ /**
+ * {@inheritDoc}
+ */
@Override
public List<SuggestItem> lookupCompound(String input, int maxEditDistance) {
Objects.requireNonNull(input, "input must not be null");
@@ -401,7 +405,7 @@ public final class SymSpell implements SpellChecker {
throw new IllegalArgumentException("maxEditDistance must not be
negative: " + maxEditDistance);
}
- final String[] termList = input.trim().isEmpty() ? new String[0] :
input.trim().split("\\s+");
+ final String[] termList = StringUtil.splitOnUnicodeWhitespace(input);
final List<SuggestItem> suggestionParts = new ArrayList<>();
boolean lastCombi = false;
@@ -531,8 +535,9 @@ public final class SymSpell implements SpellChecker {
}
final String corrected = joined.toString();
+ // Trim on the same set the terms were split on, so the distance is not
inflated.
final int distance = editDistance.distance(
- input.trim(), corrected, Integer.MAX_VALUE - 1);
+ StringUtil.trimUnicodeWhitespace(input), corrected, Integer.MAX_VALUE
- 1);
final long frequency = (long) freqProduct;
return Collections.singletonList(
new SuggestItem(corrected, Math.max(distance, 0), frequency));
diff --git
a/opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizerTest.java
b/opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizerTest.java
index 5c8f5c5e5..c731f8abb 100644
---
a/opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizerTest.java
+++
b/opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizerTest.java
@@ -249,4 +249,52 @@ public class SpellCheckingCharSequenceNormalizerTest {
() -> "core: " + core);
}
}
+
+ @Test
+ void perTokenTreatsNoBreakSpaceAsTokenBoundary() {
+ // Since 3.0 PER_TOKEN boundaries use the Unicode White_Space set, which
includes the
+ // no-break spaces; both sides are corrected and the separator is copied
verbatim
+ // (Character.isWhitespace excluded the Zs no-break spaces).
+ final var normalizer =
SpellCheckingCharSequenceNormalizer.builder(symSpell)
+ .minTokenLength(3).build();
+ assertEquals("the" + cp(0x00A0) + "fox",
+ norm(normalizer, "teh" + cp(0x00A0) + "fxo"));
+ assertEquals("the" + cp(0x202F) + "fox",
+ norm(normalizer, "teh" + cp(0x202F) + "fxo"));
+ }
+
+ @Test
+ void perTokenTreatsNextLineControlAsTokenBoundary() {
+ // U+0085 NEL carries the Unicode White_Space property, so since 3.0 it
separates
+ // tokens (Character.isWhitespace excludes it).
+ final var normalizer =
SpellCheckingCharSequenceNormalizer.builder(symSpell)
+ .minTokenLength(3).build();
+ assertEquals("the" + cp(0x0085) + "fox",
+ norm(normalizer, "teh" + cp(0x0085) + "fxo"));
+ }
+
+ @Test
+ void perTokenTreatsInformationSeparatorAsPartOfTheToken() {
+ // The U+001C..U+001F information separators are not Unicode White_Space,
so since 3.0
+ // they no longer separate tokens; the joined token has no dictionary
entry within
+ // reach and passes through unchanged (Character.isWhitespace treated
U+001C as
+ // whitespace and corrected both sides).
+ final var normalizer =
SpellCheckingCharSequenceNormalizer.builder(symSpell)
+ .minTokenLength(3).build();
+ String input = "teh" + cp(0x001C) + "fxo";
+ assertEquals(input, norm(normalizer, input));
+ }
+
+ @Test
+ void perTokenTreatsLineSeparatorAsTokenBoundary() {
+ // U+2028 is whitespace under both Character.isWhitespace and Unicode
White_Space.
+ final var normalizer =
SpellCheckingCharSequenceNormalizer.builder(symSpell)
+ .minTokenLength(3).build();
+ assertEquals("the" + cp(0x2028) + "fox",
+ norm(normalizer, "teh" + cp(0x2028) + "fxo"));
+ }
+
+ private static String cp(int codePoint) {
+ return new String(Character.toChars(codePoint));
+ }
}
diff --git
a/opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/symspell/SymSpellCompoundTest.java
b/opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/symspell/SymSpellCompoundTest.java
index 897181907..5d78f534e 100644
---
a/opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/symspell/SymSpellCompoundTest.java
+++
b/opennlp-extensions/opennlp-spellcheck/src/test/java/opennlp/spellcheck/symspell/SymSpellCompoundTest.java
@@ -139,4 +139,68 @@ public class SymSpellCompoundTest {
void nullInputIsRejected() {
assertThrows(NullPointerException.class, () -> tiny.lookupCompound(null,
2));
}
+
+ // ------------------------------------------------------------------
+ // Whitespace boundary code points.
+ // ------------------------------------------------------------------
+
+ @Test
+ void noBreakSpaceSeparatesTerms() {
+ // Since 3.0 lookupCompound splits terms on the Unicode White_Space set,
so the
+ // NBSP-joined pair is two known terms. The reported edit distance is
still 1 because
+ // it is measured against the trimmed input, where the interior NBSP
differs from the
+ // space the corrected phrase is joined with.
+ final List<SuggestItem> r = tiny.lookupCompound("hello" + cp(0x00A0) +
"world", 2);
+ assertEquals(1, r.size());
+ assertEquals("hello world", r.get(0).term());
+ assertEquals(1, r.get(0).editDistance());
+ }
+
+ @Test
+ void noBreakSpaceJoinedTripleIsFullyRepaired() {
+ // Since 3.0 the three NBSP-separated words are three terms; under the
previous ASCII
+ // \s+ split this was one term and the single-split heuristic could not
repair it.
+ String input = "hello" + cp(0x00A0) + "world" + cp(0x00A0) + "hello";
+ assertEquals("hello world hello", correct(tiny, input));
+ }
+
+ @Test
+ void nextLineControlJoinedTripleIsFullyRepaired() {
+ // U+0085 NEL carries the Unicode White_Space property, so since 3.0 it
separates terms.
+ String input = "hello" + cp(0x0085) + "world" + cp(0x0085) + "hello";
+ assertEquals("hello world hello", correct(tiny, input));
+ }
+
+ @Test
+ void lineSeparatorJoinedTripleIsFullyRepaired() {
+ // U+2028 LINE SEPARATOR is Unicode White_Space, so since 3.0 it separates
terms.
+ String input = "hello" + cp(0x2028) + "world" + cp(0x2028) + "hello";
+ assertEquals("hello world hello", correct(tiny, input));
+ }
+
+ @Test
+ void noBreakSpaceOnlyInputReportsDistanceZero() {
+ // The distance path trims the input on the same Unicode White_Space set
the terms are
+ // split on: a whitespace-only input corrects to the empty phrase at
distance 0, exactly
+ // like an ASCII-space-only input (the ASCII String.trim() kept the NBSP
and reported 1).
+ final List<SuggestItem> r = tiny.lookupCompound(cp(0x00A0), 2);
+ assertEquals(1, r.size());
+ assertEquals("", r.get(0).term());
+ assertEquals(0, r.get(0).editDistance());
+ }
+
+ @Test
+ void leadingAndTrailingNoBreakSpacesDoNotInflateDistance() {
+ // Leading and trailing NBSP runs are trimmed before the distance is
measured, so a
+ // correct NBSP-padded phrase reports distance 0 like its
ASCII-space-padded twin.
+ final String padded = cp(0x00A0) + "hello world" + cp(0x00A0) + cp(0x0085);
+ final List<SuggestItem> r = tiny.lookupCompound(padded, 2);
+ assertEquals(1, r.size());
+ assertEquals("hello world", r.get(0).term());
+ assertEquals(0, r.get(0).editDistance());
+ }
+
+ private static String cp(int codePoint) {
+ return new String(Character.toChars(codePoint));
+ }
}
diff --git
a/opennlp-extensions/opennlp-uima/src/main/java/opennlp/uima/normalizer/NumberUtil.java
b/opennlp-extensions/opennlp-uima/src/main/java/opennlp/uima/normalizer/NumberUtil.java
index bc1dc6b9c..c421b6633 100644
---
a/opennlp-extensions/opennlp-uima/src/main/java/opennlp/uima/normalizer/NumberUtil.java
+++
b/opennlp-extensions/opennlp-uima/src/main/java/opennlp/uima/normalizer/NumberUtil.java
@@ -20,15 +20,14 @@ package opennlp.uima.normalizer;
import java.text.NumberFormat;
import java.text.ParseException;
import java.util.Locale;
-import java.util.regex.Pattern;
+
+import opennlp.tools.util.StringUtil;
/**
* Provides methods to parse numbers which occur in natural language texts.
*/
public final class NumberUtil {
- private final static Pattern WHITESPACE_PATTERN = Pattern.compile("\\s");
-
/**
* Checks if the language is supported.
*
@@ -53,6 +52,9 @@ public final class NumberUtil {
/**
* Parses a specified {@link String number} for a certain {@code
languageCode}.
+ * <p>
+ * Before parsing, every Unicode {@code White_Space} code point is removed
from
+ * {@code number}.
*
* @param number The suspected number to parse.
* @param languageCode A ISO conform language code, e.g. "en", "pt"
@@ -71,7 +73,23 @@ public final class NumberUtil {
}
NumberFormat numberFormat = NumberFormat.getInstance(locale);
- number = WHITESPACE_PATTERN.matcher(number).replaceAll("");
+ number = removeWhitespace(number);
return numberFormat.parse(number);
}
+
+ /**
+ * Removes every Unicode {@code White_Space} code point from the given
string.
+ */
+ private static String removeWhitespace(String s) {
+ final StringBuilder sb = new StringBuilder(s.length());
+ int i = 0;
+ while (i < s.length()) {
+ final int cp = s.codePointAt(i);
+ if (!StringUtil.isUnicodeWhitespace(cp)) {
+ sb.appendCodePoint(cp);
+ }
+ i += Character.charCount(cp);
+ }
+ return sb.toString();
+ }
}
diff --git
a/opennlp-extensions/opennlp-uima/src/test/java/opennlp/uima/normalizer/NumberUtilTest.java
b/opennlp-extensions/opennlp-uima/src/test/java/opennlp/uima/normalizer/NumberUtilTest.java
index 4491fa74b..4e6213021 100644
---
a/opennlp-extensions/opennlp-uima/src/test/java/opennlp/uima/normalizer/NumberUtilTest.java
+++
b/opennlp-extensions/opennlp-uima/src/test/java/opennlp/uima/normalizer/NumberUtilTest.java
@@ -63,4 +63,36 @@ class NumberUtilTest {
} , "java.lang.IllegalArgumentException: Language INVALID is not
supported!");
}
+ @Test
+ void parse_withNoBreakSpaces() throws ParseException {
+ // Since 3.0 the pre-parse strip removes every Unicode White_Space code
point, so the
+ // no-break spaces used as digit grouping separators no longer stop the
parse (the
+ // previous ASCII \s strip left them in place and en parsing stopped at
them).
+ Assertions.assertEquals(1234L, NumberUtil.parse("1" + cp(0x00A0) + "234",
+ VALID_LANGUAGE_CODE).longValue());
+ Assertions.assertEquals(1234L, NumberUtil.parse("1" + cp(0x202F) + "234",
+ VALID_LANGUAGE_CODE).longValue());
+ }
+
+ @Test
+ void parse_withLineSeparatorAndNextLineControl() throws ParseException {
+ // U+2028 and U+0085 carry the Unicode White_Space property, so since 3.0
they are
+ // stripped as well.
+ Assertions.assertEquals(1234L, NumberUtil.parse("12" + cp(0x2028) + "34",
+ VALID_LANGUAGE_CODE).longValue());
+ Assertions.assertEquals(1234L, NumberUtil.parse("12" + cp(0x0085) + "34",
+ VALID_LANGUAGE_CODE).longValue());
+ }
+
+ @Test
+ void parse_withInformationSeparator() throws ParseException {
+ // U+001C is neither ASCII whitespace nor Unicode White_Space; it is never
stripped.
+ Assertions.assertEquals(12L, NumberUtil.parse("12" + cp(0x001C) + "34",
+ VALID_LANGUAGE_CODE).longValue());
+ }
+
+ private static String cp(int codePoint) {
+ return new String(Character.toChars(codePoint));
+ }
+
}
diff --git a/opennlp-tools/src/test/java/opennlp/tools/util/StringUtilTest.java
b/opennlp-tools/src/test/java/opennlp/tools/util/StringUtilTest.java
index b6f02339c..a47306d3d 100644
--- a/opennlp-tools/src/test/java/opennlp/tools/util/StringUtilTest.java
+++ b/opennlp-tools/src/test/java/opennlp/tools/util/StringUtilTest.java
@@ -17,15 +17,71 @@
package opennlp.tools.util;
+import java.nio.CharBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.util.normalizer.UnicodeWhitespace;
+import opennlp.tools.util.normalizer.UnicodeWhitespace.RelatedCharacter;
+import opennlp.tools.util.normalizer.UnicodeWhitespace.WhitespaceCharacter;
/**
* Tests for the {@link StringUtil} class.
*/
-
public class StringUtilTest {
+ private static final int[] INFO_SEPARATORS = {0x001C, 0x001D, 0x001E,
0x001F};
+
+ private static final int DESERET_CAPITAL_BEE = 0x10412; //
supplementary-plane letter
+ private static final int DESERET_SMALL_BEE = 0x1043A;
+ private static final int GRINNING_FACE = 0x1F600; // emoji, two chars
+
+ /**
+ * Restores {@link WhitespaceMode} property resolution after each test, so
no mode
+ * state leaks.
+ */
+ @AfterEach
+ void resetWhitespaceMode() {
+ WhitespaceMode.reset();
+ }
+
+ private static List<WhitespaceCharacter> whitespace() {
+ return UnicodeWhitespace.all();
+ }
+
+ private static List<RelatedCharacter> lookalikes() {
+ return UnicodeWhitespace.lookalikes();
+ }
+
+ private static Stream<CharSequence> nonStringCharSequences() {
+ return Stream.of(
+ new StringBuilder("a\u00A0b"),
+ new StringBuffer("a\u00A0b"),
+ CharBuffer.wrap("a\u00A0b".toCharArray()));
+ }
+
+ private static String cp(int codePoint) {
+ return new String(Character.toChars(codePoint));
+ }
+
+ private static String join(int... codePoints) {
+ return new String(codePoints, 0, codePoints.length);
+ }
+
+ // -------------------------------------------------------------------------
+ // isWhitespace / isUnicodeWhitespace (existing pins, kept)
+ // -------------------------------------------------------------------------
+
@Test
void testNoBreakSpace() {
Assertions.assertTrue(StringUtil.isWhitespace(0x00A0));
@@ -37,6 +93,561 @@ public class StringUtilTest {
Assertions.assertTrue(StringUtil.isWhitespace((char) 0x202F));
}
+ /**
+ * Pins the exact semantics of {@link StringUtil#isWhitespace(int)} under
the default
+ * {@link WhitespaceMode#UNICODE}, at the code points where the JVM
predicates and the
+ * Unicode {@code White_Space} property disagree, so the predicate cannot
drift silently:
+ * it excludes the {@code U+001C..U+001F} information separators and
includes the next
+ * line control {@code U+0085}, agreeing with {@link
StringUtil#isUnicodeWhitespace(int)}.
+ */
+ @Test
+ void testIsWhitespaceBoundaryCodePointsDefaultIsUnicode() {
+ for (int cp = 0x0009; cp <= 0x000D; cp++) {
+ Assertions.assertTrue(StringUtil.isWhitespace(cp), "U+" +
Integer.toHexString(cp));
+ }
+ Assertions.assertTrue(StringUtil.isWhitespace(0x0020));
+
+ for (int cp : INFO_SEPARATORS) {
+ Assertions.assertFalse(StringUtil.isWhitespace(cp), "U+" +
Integer.toHexString(cp));
+ Assertions.assertFalse(StringUtil.isWhitespace((char) cp), "U+" +
Integer.toHexString(cp));
+ }
+
+ Assertions.assertTrue(StringUtil.isWhitespace(0x0085));
+ Assertions.assertTrue(StringUtil.isWhitespace((char) 0x0085));
+
+ int[] separators = {0x1680, 0x2000, 0x2001, 0x2002, 0x2003, 0x2004,
0x2005, 0x2006,
+ 0x2008, 0x2009, 0x200A, 0x2028, 0x2029, 0x205F, 0x3000};
+ for (int cp : separators) {
+ Assertions.assertTrue(StringUtil.isWhitespace(cp), "U+" +
Integer.toHexString(cp));
+ }
+
+ Assertions.assertFalse(StringUtil.isWhitespace(0x200B));
+ Assertions.assertFalse(StringUtil.isWhitespace(0xFEFF));
+ }
+
+ /**
+ * Pins the exact semantics of {@link StringUtil#isWhitespace(int)} under
+ * {@link WhitespaceMode#LEGACY}: the union of {@link
Character#isWhitespace(int)} and the
+ * {@code Zs} category, the opposite of the Unicode {@code White_Space} set
at
+ * {@code U+001C..U+001F} and {@code U+0085}. Trained sentence-detector and
tokenizer
+ * models built under this definition depend on it.
+ */
+ @Test
+ void testIsWhitespaceBoundaryCodePointsUnderLegacyMode() {
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+
+ for (int cp = 0x0009; cp <= 0x000D; cp++) {
+ Assertions.assertTrue(StringUtil.isWhitespace(cp), "U+" +
Integer.toHexString(cp));
+ }
+ Assertions.assertTrue(StringUtil.isWhitespace(0x0020));
+
+ for (int cp : INFO_SEPARATORS) {
+ Assertions.assertTrue(StringUtil.isWhitespace(cp), "U+" +
Integer.toHexString(cp));
+ Assertions.assertTrue(StringUtil.isWhitespace((char) cp), "U+" +
Integer.toHexString(cp));
+ }
+
+ Assertions.assertFalse(StringUtil.isWhitespace(0x0085));
+ Assertions.assertFalse(StringUtil.isWhitespace((char) 0x0085));
+
+ int[] separators = {0x1680, 0x2000, 0x2001, 0x2002, 0x2003, 0x2004,
0x2005, 0x2006,
+ 0x2008, 0x2009, 0x200A, 0x2028, 0x2029, 0x205F, 0x3000};
+ for (int cp : separators) {
+ Assertions.assertTrue(StringUtil.isWhitespace(cp), "U+" +
Integer.toHexString(cp));
+ }
+
+ Assertions.assertFalse(StringUtil.isWhitespace(0x200B));
+ Assertions.assertFalse(StringUtil.isWhitespace(0xFEFF));
+ }
+
+ /**
+ * The {@link StringUtil#isUnicodeWhitespace(int)} facade must agree with
the Unicode
+ * {@code White_Space} reference implementation on every code point,
including the
+ * deltas to the legacy predicate ({@code U+0085} in, {@code U+001C..U+001F}
out).
+ */
+ @Test
+ void testIsUnicodeWhitespaceDelegates() {
+ for (int cp = 0x0000; cp <= 0x3000; cp++) {
+ Assertions.assertEquals(
+ UnicodeWhitespace.isWhitespace(cp),
+ StringUtil.isUnicodeWhitespace(cp), "U+" + Integer.toHexString(cp));
+ }
+ Assertions.assertTrue(StringUtil.isUnicodeWhitespace(0x0085));
+ Assertions.assertTrue(StringUtil.isUnicodeWhitespace((char) 0x0085));
+ for (int cp : INFO_SEPARATORS) {
+ Assertions.assertFalse(StringUtil.isUnicodeWhitespace(cp), "U+" +
Integer.toHexString(cp));
+ Assertions.assertFalse(StringUtil.isUnicodeWhitespace((char) cp),
+ "U+" + Integer.toHexString(cp));
+ }
+ Assertions.assertTrue(StringUtil.isUnicodeWhitespace(' '));
+ Assertions.assertTrue(StringUtil.isUnicodeWhitespace((char) 0x00A0));
+ Assertions.assertTrue(StringUtil.isUnicodeWhitespace(0x2028));
+ Assertions.assertFalse(StringUtil.isUnicodeWhitespace('a'));
+ Assertions.assertFalse(StringUtil.isUnicodeWhitespace(0x200B));
+ }
+
+ // -------------------------------------------------------------------------
+ // splitOnUnicodeWhitespace
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testSplitOnUnicodeWhitespaceNullThrows() {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> StringUtil.splitOnUnicodeWhitespace(null));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceEmptyAndWhitespaceOnly() {
+ Assertions.assertArrayEquals(new String[0],
StringUtil.splitOnUnicodeWhitespace(""));
+ Assertions.assertArrayEquals(new String[0],
StringUtil.splitOnUnicodeWhitespace(" "));
+ Assertions.assertArrayEquals(new String[0],
StringUtil.splitOnUnicodeWhitespace("\t\n\r"));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceAsciiBasics() {
+ Assertions.assertArrayEquals(new String[] {"a"},
StringUtil.splitOnUnicodeWhitespace("a"));
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ StringUtil.splitOnUnicodeWhitespace("a b"));
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ StringUtil.splitOnUnicodeWhitespace(" a\tb "));
+ Assertions.assertArrayEquals(new String[] {"hello", "world"},
+ StringUtil.splitOnUnicodeWhitespace("hello world"));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceCollapsesMixedRuns() {
+ // Tab, space, NBSP, NEL, ideographic space between two tokens — one
boundary.
+ final String input = "left" + "\t \u00A0\u0085\u3000" + "right";
+ Assertions.assertArrayEquals(new String[] {"left", "right"},
+ StringUtil.splitOnUnicodeWhitespace(input));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespacePreservesInternalNonWhitespace() {
+ // Hyphen, punctuation, digits stay inside the term.
+ Assertions.assertArrayEquals(new String[] {"well-known", "C++", "3.14"},
+ StringUtil.splitOnUnicodeWhitespace("well-known C++ 3.14"));
+ }
+
+ @ParameterizedTest
+ @MethodSource("whitespace")
+ void
testSplitOnUnicodeWhitespaceEveryWhiteSpaceCodePointSeparates(WhitespaceCharacter
ws) {
+ final String sep = cp(ws.codePoint());
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ StringUtil.splitOnUnicodeWhitespace("a" + sep + "b"),
+ () -> ws.toUnicodeNotation() + " must separate tokens");
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ StringUtil.splitOnUnicodeWhitespace(sep + "a" + sep + sep + "b" + sep),
+ () -> ws.toUnicodeNotation() + " leading/trailing/repeated runs must
collapse");
+ }
+
+ @ParameterizedTest
+ @MethodSource("lookalikes")
+ void testSplitOnUnicodeWhitespaceLookalikesDoNotSeparate(RelatedCharacter
related) {
+ // ZWSP/BOM/etc. are not White_Space — they stay glued inside the token.
+ final String glue = cp(related.codePoint());
+ Assertions.assertArrayEquals(new String[] {"a" + glue + "b"},
+ StringUtil.splitOnUnicodeWhitespace("a" + glue + "b"),
+ () -> related.toUnicodeNotation() + " must remain inside the token");
+ }
+
+ @ParameterizedTest
+ @ValueSource(ints = {0x001C, 0x001D, 0x001E, 0x001F})
+ void testSplitOnUnicodeWhitespaceInfoSeparatorsDoNotSeparate(int infoSep) {
+ // Character.isWhitespace includes these; Unicode White_Space does not.
+ Assertions.assertArrayEquals(new String[] {"a" + cp(infoSep) + "b"},
+ StringUtil.splitOnUnicodeWhitespace("a" + cp(infoSep) + "b"),
+ () -> "U+" + Integer.toHexString(infoSep));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespacePdfStyleTypographicSpaces() {
+ // Newspaper/PDF extracts often mix NBSP, figure space, thin/hair/narrow
spaces.
+ final String input = "New" + cp(0x00A0) + "York" + cp(0x2007) + "Times"
+ + cp(0x2009) + cp(0x200A) + cp(0x202F) + "Inc.";
+ Assertions.assertArrayEquals(new String[] {"New", "York", "Times", "Inc."},
+ StringUtil.splitOnUnicodeWhitespace(input));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceLineAndParagraphSeparators() {
+ Assertions.assertArrayEquals(new String[] {"a", "b", "c"},
+ StringUtil.splitOnUnicodeWhitespace("a\u2028b\u2029c"));
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ StringUtil.splitOnUnicodeWhitespace("a\r\nb"));
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ StringUtil.splitOnUnicodeWhitespace("a\n\nb"));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceSupplementaryPlaneTerms() {
+ // Terms may contain supplementary-plane code points (2 chars each); the
scanner
+ // must advance by charCount so it does not split inside a surrogate pair.
+ final String bee = cp(DESERET_CAPITAL_BEE);
+ final String face = cp(GRINNING_FACE);
+ Assertions.assertEquals(2, bee.length());
+ Assertions.assertEquals(2, face.length());
+
+ Assertions.assertArrayEquals(new String[] {bee, face},
+ StringUtil.splitOnUnicodeWhitespace(bee + " " + face));
+ Assertions.assertArrayEquals(new String[] {bee + face},
+ StringUtil.splitOnUnicodeWhitespace(bee + face));
+ Assertions.assertArrayEquals(new String[] {bee, "x", face},
+ StringUtil.splitOnUnicodeWhitespace(bee + "\u00A0x\u3000" + face));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceManyAlternatingTokens() {
+ final StringBuilder sb = new StringBuilder();
+ for (int i = 0; i < 50; i++) {
+ if (i > 0) {
+ sb.append(i % 2 == 0 ? ' ' : '\u00A0');
+ }
+ sb.append((char) ('a' + (i % 26)));
+ }
+ final String[] tokens = StringUtil.splitOnUnicodeWhitespace(sb.toString());
+ Assertions.assertEquals(50, tokens.length);
+ for (int i = 0; i < 50; i++) {
+ Assertions.assertEquals(String.valueOf((char) ('a' + (i % 26))),
tokens[i]);
+ }
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceLongWhitespaceRun() {
+ final String run = " \t\u00A0".repeat(200);
+ Assertions.assertArrayEquals(new String[] {"x", "y"},
+ StringUtil.splitOnUnicodeWhitespace(run + "x" + run + "y" + run));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceSingleWhitespaceBetweenEmptyYieldsEmpty() {
+ // Only whitespace → empty array, never [""].
+ for (WhitespaceCharacter ws : UnicodeWhitespace.all()) {
+ Assertions.assertArrayEquals(new String[0],
+ StringUtil.splitOnUnicodeWhitespace(cp(ws.codePoint())),
+ () -> ws.toUnicodeNotation());
+ }
+ }
+
+ @ParameterizedTest
+ @MethodSource("nonStringCharSequences")
+ void
testSplitOnUnicodeWhitespaceAcceptsCharSequenceImplementations(CharSequence
input) {
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ StringUtil.splitOnUnicodeWhitespace(input));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceIgnoresLegacyMode() {
+ // The Unicode helpers are unconditional; LEGACY must not change them.
+ final String withNel = "a\u0085b";
+ final String withInfo = "a\u001Cb";
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ StringUtil.splitOnUnicodeWhitespace(withNel));
+ Assertions.assertArrayEquals(new String[] {"a\u001Cb"},
+ StringUtil.splitOnUnicodeWhitespace(withInfo));
+
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
+ StringUtil.splitOnUnicodeWhitespace(withNel));
+ Assertions.assertArrayEquals(new String[] {"a\u001Cb"},
+ StringUtil.splitOnUnicodeWhitespace(withInfo));
+ }
+
+ @Test
+ void testSplitOnUnicodeWhitespaceAllTwentyFiveAsMixedSeparators() {
+ // Build "t0 <ws0> t1 <ws1> ... t24 <ws24> t25" using every White_Space
code point.
+ final List<WhitespaceCharacter> all = UnicodeWhitespace.all();
+ final StringBuilder sb = new StringBuilder();
+ for (int i = 0; i < all.size(); i++) {
+ sb.append('t').append(i);
+ sb.appendCodePoint(all.get(i).codePoint());
+ }
+ sb.append('t').append(all.size());
+
+ final String[] expected = IntStream.rangeClosed(0, all.size())
+ .mapToObj(i -> "t" + i)
+ .toArray(String[]::new);
+ Assertions.assertArrayEquals(expected,
StringUtil.splitOnUnicodeWhitespace(sb.toString()));
+ }
+
+ // -------------------------------------------------------------------------
+ // trimUnicodeWhitespace
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testTrimUnicodeWhitespaceNullThrows() {
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> StringUtil.trimUnicodeWhitespace(null));
+ }
+
+ @Test
+ void testTrimUnicodeWhitespaceEmptyAndWhitespaceOnly() {
+ Assertions.assertEquals("", StringUtil.trimUnicodeWhitespace(""));
+ Assertions.assertEquals("", StringUtil.trimUnicodeWhitespace(" "));
+ Assertions.assertEquals("",
StringUtil.trimUnicodeWhitespace("\t\n\r\u00A0\u0085"));
+ }
+
+ @Test
+ void testTrimUnicodeWhitespacePreservesInternalWhitespace() {
+ Assertions.assertEquals("a b", StringUtil.trimUnicodeWhitespace(" a b
"));
+ Assertions.assertEquals("a\u00A0b",
StringUtil.trimUnicodeWhitespace("\ta\u00A0b\n"));
+ Assertions.assertEquals("a \t b", StringUtil.trimUnicodeWhitespace("a
\t b"));
+ }
+
+ @ParameterizedTest
+ @MethodSource("whitespace")
+ void testTrimUnicodeWhitespaceEveryWhiteSpaceCodePoint(WhitespaceCharacter
ws) {
+ final String sep = cp(ws.codePoint());
+ Assertions.assertEquals("x", StringUtil.trimUnicodeWhitespace(sep + "x" +
sep),
+ () -> ws.toUnicodeNotation());
+ Assertions.assertEquals("x y", StringUtil.trimUnicodeWhitespace(sep + "x
y" + sep),
+ () -> ws.toUnicodeNotation() + " must not trim internal ASCII space");
+ Assertions.assertEquals("", StringUtil.trimUnicodeWhitespace(sep + sep +
sep),
+ () -> ws.toUnicodeNotation() + " whitespace-only");
+ }
+
+ @ParameterizedTest
+ @MethodSource("lookalikes")
+ void testTrimUnicodeWhitespaceLookalikesAreNotTrimmed(RelatedCharacter
related) {
+ final String glue = cp(related.codePoint());
+ // Leading/trailing lookalikes stay — they are not White_Space.
+ Assertions.assertEquals(glue + "x" + glue,
+ StringUtil.trimUnicodeWhitespace(glue + "x" + glue),
+ () -> related.toUnicodeNotation());
+ // Surrounded by real whitespace, lookalike still survives as content.
+ Assertions.assertEquals(glue + "x" + glue,
+ StringUtil.trimUnicodeWhitespace(" " + glue + "x" + glue + " "),
+ () -> related.toUnicodeNotation() + " after real trim");
+ }
+
+ @ParameterizedTest
+ @ValueSource(ints = {0x001C, 0x001D, 0x001E, 0x001F})
+ void testTrimUnicodeWhitespaceInfoSeparatorsAreNotTrimmed(int infoSep) {
+ final String sep = cp(infoSep);
+ Assertions.assertEquals(sep + "x" + sep,
StringUtil.trimUnicodeWhitespace(sep + "x" + sep));
+ Assertions.assertEquals(sep + "x" + sep,
+ StringUtil.trimUnicodeWhitespace(" " + sep + "x" + sep + " "));
+ }
+
+ @Test
+ void testTrimUnicodeWhitespaceSupplementaryPlaneEdges() {
+ final String bee = cp(DESERET_CAPITAL_BEE);
+ final String face = cp(GRINNING_FACE);
+ Assertions.assertEquals(bee,
+ StringUtil.trimUnicodeWhitespace("\u00A0" + bee + "\u3000"));
+ Assertions.assertEquals(bee + " " + face,
+ StringUtil.trimUnicodeWhitespace("\u0085" + bee + " " + face + "\t"));
+ // Trim must not chop a surrogate pair when the edge is a supplementary
char.
+ Assertions.assertEquals(bee, StringUtil.trimUnicodeWhitespace(bee));
+ Assertions.assertEquals(face, StringUtil.trimUnicodeWhitespace(face));
+ }
+
+ @Test
+ void testTrimUnicodeWhitespaceMixedLeadingTrailing() {
+ final String leading = "\t\u00A0\u2007\u202F\u3000";
+ final String trailing = "\u0085\u2028\u2029\n";
+ Assertions.assertEquals("keep me",
+ StringUtil.trimUnicodeWhitespace(leading + "keep me" + trailing));
+ }
+
+ @Test
+ void testTrimUnicodeWhitespaceOnlyLeadingOrOnlyTrailing() {
+ Assertions.assertEquals("x",
StringUtil.trimUnicodeWhitespace("\u00A0\u00A0x"));
+ Assertions.assertEquals("x",
StringUtil.trimUnicodeWhitespace("x\u00A0\u00A0"));
+ Assertions.assertEquals("x", StringUtil.trimUnicodeWhitespace("x"));
+ }
+
+ @Test
+ void testTrimUnicodeWhitespaceIgnoresLegacyMode() {
+ Assertions.assertEquals("a\u0085b", StringUtil.trimUnicodeWhitespace("
a\u0085b "));
+ Assertions.assertEquals("\u001Cx\u001C",
+ StringUtil.trimUnicodeWhitespace("\u001Cx\u001C"));
+
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+ Assertions.assertEquals("a\u0085b", StringUtil.trimUnicodeWhitespace("
a\u0085b "));
+ // Info separators still not trimmed under LEGACY — Unicode helper is
unconditional.
+ Assertions.assertEquals("\u001Cx\u001C",
+ StringUtil.trimUnicodeWhitespace("\u001Cx\u001C"));
+ // But NEL is still trimmed (Unicode), even though LEGACY
isWhitespace(NEL) is false.
+ Assertions.assertEquals("x",
StringUtil.trimUnicodeWhitespace("\u0085x\u0085"));
+ }
+
+ @ParameterizedTest
+ @MethodSource("nonStringCharSequences")
+ void
testTrimUnicodeWhitespaceAcceptsCharSequenceImplementations(CharSequence input)
{
+ Assertions.assertEquals("a\u00A0b",
StringUtil.trimUnicodeWhitespace(input));
+ }
+
+ @Test
+ void testTrimUnicodeWhitespaceDoesNotMatchStringTrimOnNbsp() {
+ // String.trim() only strips <= U+0020; NBSP must still be trimmed by our
helper.
+ final String padded = "\u00A0hello\u00A0";
+ Assertions.assertEquals(padded, padded.trim());
+ Assertions.assertEquals("hello", StringUtil.trimUnicodeWhitespace(padded));
+ }
+
+ // -------------------------------------------------------------------------
+ // isUnicodeBlank
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testIsUnicodeBlankNullAndEmpty() {
+ Assertions.assertTrue(StringUtil.isUnicodeBlank(null));
+ Assertions.assertTrue(StringUtil.isUnicodeBlank(""));
+ }
+
+ @Test
+ void testIsUnicodeBlankAsciiWhitespace() {
+ Assertions.assertTrue(StringUtil.isUnicodeBlank(" "));
+ Assertions.assertTrue(StringUtil.isUnicodeBlank(" \t\n\r\f"));
+ Assertions.assertFalse(StringUtil.isUnicodeBlank("a"));
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(" a "));
+ }
+
+ @ParameterizedTest
+ @MethodSource("whitespace")
+ void testIsUnicodeBlankEveryWhiteSpaceCodePoint(WhitespaceCharacter ws) {
+ Assertions.assertTrue(StringUtil.isUnicodeBlank(cp(ws.codePoint())),
+ () -> ws.toUnicodeNotation());
+
Assertions.assertTrue(StringUtil.isUnicodeBlank(cp(ws.codePoint()).repeat(3)),
+ () -> ws.toUnicodeNotation() + " repeated");
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(cp(ws.codePoint()) + "x"),
+ () -> ws.toUnicodeNotation() + " with content");
+ }
+
+ @Test
+ void testIsUnicodeBlankAllTwentyFiveTogether() {
+ final String allWs = UnicodeWhitespace.all().stream()
+ .map(ws -> cp(ws.codePoint()))
+ .collect(Collectors.joining());
+ Assertions.assertTrue(StringUtil.isUnicodeBlank(allWs));
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(allWs + "."));
+ }
+
+ @ParameterizedTest
+ @MethodSource("lookalikes")
+ void testIsUnicodeBlankLookalikesAreNotBlank(RelatedCharacter related) {
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(cp(related.codePoint())),
+ () -> related.toUnicodeNotation());
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(" " +
cp(related.codePoint()) + " "),
+ () -> related.toUnicodeNotation() + " surrounded by real whitespace");
+ }
+
+ @ParameterizedTest
+ @ValueSource(ints = {0x001C, 0x001D, 0x001E, 0x001F})
+ void testIsUnicodeBlankInfoSeparatorsAreNotBlank(int infoSep) {
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(cp(infoSep)));
+ // JVM String.isBlank uses Character.isWhitespace, which *does* treat info
separators
+ // as blank — pin that we disagree.
+ Assertions.assertTrue(cp(infoSep).isBlank());
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(cp(infoSep)));
+ }
+
+ @Test
+ void testIsUnicodeBlankDisagreesWithStringIsBlankOnNbsp() {
+ Assertions.assertFalse("\u00A0".isBlank()); // JDK: NBSP is not
Character.isWhitespace
+ Assertions.assertTrue(StringUtil.isUnicodeBlank("\u00A0"));
+ Assertions.assertFalse("\u0085".isBlank()); // JDK: NEL likewise
+ Assertions.assertTrue(StringUtil.isUnicodeBlank("\u0085"));
+ }
+
+ @Test
+ void testIsUnicodeBlankSupplementaryPlaneContent() {
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(cp(DESERET_CAPITAL_BEE)));
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(cp(GRINNING_FACE)));
+ Assertions.assertFalse(StringUtil.isUnicodeBlank("\u00A0" +
cp(GRINNING_FACE)));
+ }
+
+ @Test
+ void testIsUnicodeBlankIgnoresLegacyMode() {
+ Assertions.assertTrue(StringUtil.isUnicodeBlank("\u0085"));
+ Assertions.assertFalse(StringUtil.isUnicodeBlank("\u001C"));
+
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+ Assertions.assertTrue(StringUtil.isUnicodeBlank("\u0085"));
+ Assertions.assertFalse(StringUtil.isUnicodeBlank("\u001C"));
+ }
+
+ @ParameterizedTest
+ @MethodSource("nonStringCharSequences")
+ void testIsUnicodeBlankAcceptsCharSequenceImplementations(CharSequence
input) {
+ // "a\u00A0b" is not blank.
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(input));
+ }
+
+ // -------------------------------------------------------------------------
+ // Cross-helper invariants
+ // -------------------------------------------------------------------------
+
+ @Test
+ void testBlankIffSplitEmptyAndTrimEmpty() {
+ final String[] samples = {
+ "",
+ " ",
+ "\t\n\u00A0\u0085\u3000",
+ "x",
+ " x ",
+ "a\u00A0b",
+ "\u001C",
+ "\u200B",
+ cp(DESERET_CAPITAL_BEE),
+ join(0x00A0, DESERET_CAPITAL_BEE, 0x3000),
+ UnicodeWhitespace.all().stream().map(ws -> cp(ws.codePoint()))
+ .collect(Collectors.joining())
+ };
+ for (String sample : samples) {
+ final boolean blank = StringUtil.isUnicodeBlank(sample);
+ final String[] split = StringUtil.splitOnUnicodeWhitespace(sample);
+ final String trimmed = StringUtil.trimUnicodeWhitespace(sample);
+ Assertions.assertEquals(blank, split.length == 0,
+ () -> "blank iff split empty for: " +
Arrays.toString(sample.codePoints().toArray()));
+ Assertions.assertEquals(blank, trimmed.isEmpty(),
+ () -> "blank iff trim empty for: " +
Arrays.toString(sample.codePoints().toArray()));
+ }
+ }
+
+ @Test
+ void testTrimThenSplitEqualsSplit() {
+ // Leading/trailing whitespace is ignored by split, so trim-then-split is
a no-op.
+ final String[] samples = {
+ " a b ",
+ "\u00A0hello\u3000world\u0085",
+ "\t\t alone \n",
+ "a\u2007b\u2009c"
+ };
+ for (String sample : samples) {
+ Assertions.assertArrayEquals(
+ StringUtil.splitOnUnicodeWhitespace(sample),
+
StringUtil.splitOnUnicodeWhitespace(StringUtil.trimUnicodeWhitespace(sample)),
+ () -> sample);
+ }
+ }
+
+ @Test
+ void testJoinSplitTokensWithSpaceRoundTripShape() {
+ final String[] tokens = StringUtil.splitOnUnicodeWhitespace(
+ "alpha\u00A0beta\u3000gamma");
+ Assertions.assertArrayEquals(new String[] {"alpha", "beta", "gamma"},
tokens);
+ Assertions.assertArrayEquals(tokens,
+ StringUtil.splitOnUnicodeWhitespace(String.join(" ", tokens)));
+ }
+
+ @Test
+ void testSplitDoesNotEmitEmptyTokensAroundSeparators() {
+ // Regex split("\\s+") on a leading-space string yields a leading "" with
limit -1;
+ // our scanner must never emit empty strings.
+ for (String sample : List.of(" a", "a ", " a ", " a b ",
"\u00A0a\u00A0")) {
+ for (String token : StringUtil.splitOnUnicodeWhitespace(sample)) {
+ Assertions.assertFalse(token.isEmpty(), () -> "empty token from: '" +
sample + "'");
+ Assertions.assertFalse(StringUtil.isUnicodeBlank(token),
+ () -> "whitespace-only token from: '" + sample + "'");
+ }
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Existing non-whitespace StringUtil coverage
+ // -------------------------------------------------------------------------
+
@Test
void testToLowerCase() {
Assertions.assertEquals("test", StringUtil.toLowerCase("TEST"));
@@ -57,17 +668,13 @@ public class StringUtilTest {
@Test
void testIsEmptyWithNullString() {
- // should raise a NPE
- Assertions.assertThrows(NullPointerException.class, () -> {
- // should raise a NPE
- StringUtil.isEmpty(null);
- });
+ Assertions.assertThrows(NullPointerException.class, () ->
StringUtil.isEmpty(null));
}
@Test
void testLowercaseBeyondBMP() {
- int[] codePoints = new int[] {65, 66578, 67}; //A,Deseret capital BEE,C
- int[] expectedCodePoints = new int[] {97, 66618, 99};//a,Deseret lowercase
b,c
+ int[] codePoints = new int[] {65, DESERET_CAPITAL_BEE, 67};
+ int[] expectedCodePoints = new int[] {97, DESERET_SMALL_BEE, 99};
String input = new String(codePoints, 0, codePoints.length);
String lc = StringUtil.toLowerCase(input);
Assertions.assertArrayEquals(expectedCodePoints,
lc.codePoints().toArray());
