krickert commented on code in PR #1151:
URL: https://github.com/apache/opennlp/pull/1151#discussion_r3594007959
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/ShrinkCharSequenceNormalizer.java:
##########
@@ -16,28 +16,136 @@
*/
package opennlp.tools.util.normalizer;
-import java.util.regex.Pattern;
-
/**
- * A {@link ShrinkCharSequenceNormalizer} implementation that shrinks repeated
spaces / chars in text.
+ * A {@link CharSequenceNormalizer} implementation that shrinks repeated
whitespace and repeated
+ * characters in text, in three steps:
+ * <ol>
+ * <li>Each run of two or more ASCII whitespace characters (tab, line feed,
vertical tab,
+ * form feed, carriage return, or space) collapses to a single space. A
lone whitespace
+ * character is kept as it is.</li>
+ * <li>Each run of three or more repeats of one code point shrinks to two
copies of its first
+ * occurrence, comparing repeats case-insensitively over ASCII; a run
never starts on a
+ * line terminator. For example, {@code "coooool"} becomes {@code
"cool"}.</li>
+ * <li>Every leading and trailing character at or below {@code U+0020} is
dropped, the same
+ * rule {@link String#trim()} applies.</li>
+ * </ol>
*/
public class ShrinkCharSequenceNormalizer implements CharSequenceNormalizer {
private static final long serialVersionUID = -4511969661556543048L;
Review Comment:
I sure hope I'm right, I re-ran `serialver` on all twelve serializable
normalizers against the current class files. And then I asked Grok to do it.
Elon agrees.
Every declared `serialVersionUID` already matches. Please double check as I
enjoy shaming a JDK tools :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
