krickert commented on code in PR #1151:
URL: https://github.com/apache/opennlp/pull/1151#discussion_r3593904401
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NumberCharSequenceNormalizer.java:
##########
@@ -16,26 +16,43 @@
*/
package opennlp.tools.util.normalizer;
-import java.util.regex.Pattern;
-
/**
- * A {@link NumberCharSequenceNormalizer} implementation that normalizes text
- * in terms of numbers. Every encounter will be replaced by a whitespace.
+ * A {@link CharSequenceNormalizer} implementation that normalizes text in
terms of numbers:
+ * every maximal run of ASCII digits ({@code 0} to {@code 9}), that is the
longest unbroken
+ * stretch of consecutive digits, is replaced by a single space. For example,
{@code "a1234b56"}
+ * becomes {@code "a b "}. Non-ASCII digits, for example Arabic-Indic or
fullwidth digits, are
+ * not treated as digits and are left unchanged.
*/
public class NumberCharSequenceNormalizer implements CharSequenceNormalizer {
private static final long serialVersionUID = -782056416383201122L;
Review Comment:
I hand rolled that when I did it :( I'll get that going.. apologies. (ditto
on the rest) Although I do remember not doing it for a few classes that had
identical JVM output - I'll double check on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
