krickert commented on code in PR #1151:
URL: https://github.com/apache/opennlp/pull/1151#discussion_r3564915034
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/ShrinkCharSequenceNormalizer.java:
##########
@@ -16,28 +16,131 @@
*/
package opennlp.tools.util.normalizer;
-import java.util.regex.Pattern;
-
/**
- * A {@link ShrinkCharSequenceNormalizer} implementation that shrinks repeated
spaces / chars in text.
+ * A {@link CharSequenceNormalizer} implementation that shrinks repeated
spaces / chars
+ * in text.
+ *
+ * <p>Two forward cursor passes reproduce, byte for byte, the output of the
former regex
+ * implementation ({@code "\\s{2,}"} replaced by one space, then {@code
"(.)\\1{2,}"} with
+ * {@code CASE_INSENSITIVE} replaced by {@code "$1$1"}, then {@link
String#trim()}):</p>
+ * <ol>
+ * <li>Each run of two or more ASCII whitespace characters (the six
characters the regex
+ * {@code \s} class matches; this legacy rung predates the Unicode
{@code White_Space}
+ * based {@link CharClass#whitespace()}) collapses to a single space. A
lone whitespace
+ * character is kept as it is.</li>
+ * <li>Each run of three or more repeats of one code point shrinks to two
copies of its first
+ * occurrence. Repeats compare case-insensitively over ASCII only, and a
run never starts
+ * on a regex line terminator, because the former {@code .} did not
match one.</li>
+ * </ol>
+ * <p>The result is finally trimmed with {@link String#trim()}, preserving the
legacy contract
+ * that every leading or trailing char up to {@code U+0020} is dropped.</p>
*/
public class ShrinkCharSequenceNormalizer implements CharSequenceNormalizer {
private static final long serialVersionUID = -4511969661556543048L;
- private static final Pattern REPEATED_CHAR_REGEX =
Pattern.compile("(.)\\1{2,}",
- Pattern.CASE_INSENSITIVE);
- private static final Pattern SPACE_REGEX = Pattern.compile("\\s{2,}",
- Pattern.CASE_INSENSITIVE);
+ // The code points the former "." (without DOTALL) refused to match: the
regex line terminators.
+ private static final CodePointSet REGEX_LINE_TERMINATORS =
+ CodePointSet.of(0x000A, 0x000D, 0x0085, 0x2028, 0x2029);
private static final ShrinkCharSequenceNormalizer INSTANCE = new
ShrinkCharSequenceNormalizer();
public static ShrinkCharSequenceNormalizer getInstance() {
return INSTANCE;
}
- public CharSequence normalize (CharSequence text) {
- text = SPACE_REGEX.matcher(text).replaceAll(" ");
- return REPEATED_CHAR_REGEX.matcher(text).replaceAll("$1$1").trim();
+ /**
+ * @throws IllegalArgumentException Thrown if {@code text} is {@code null}.
+ */
+ @Override
+ public CharSequence normalize(CharSequence text) {
+ if (text == null) {
Review Comment:
Fixed on the other side: `NumberCharSequenceNormalizer` now has the explicit
guard too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
