This is an automated email from the ASF dual-hosted git repository.
mawiesne pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 7a560bd7a OPENNLP-1935: Regex removal (10/10): Forbid java.util.regex
in production code (#1348)
7a560bd7a is described below
commit 7a560bd7a89478ea6d5c3768bf5c5c41d67052b4
Author: Kristian Rickert <[email protected]>
AuthorDate: Sat Oct 10 04:45:40 2026 -0500
OPENNLP-1935: Regex removal (10/10): Forbid java.util.regex in production
code (#1348)
Add a forbiddenapis check for production classes in every module. Reject
java.util.regex and regex-based String and Scanner methods at
process-classes, before tests run. Exclude test classes and point error
messages to the StringUtil replacements.
Add SuppressForbidden for documented APIs that accept user-supplied
regular expressions. Require a reason for each exception and apply it
to RegexNameFinder and RegexNameFinderFactory.
Document the rule, dev/forbidden-regex.txt and allowed exceptions in
CONTRIBUTING.md. The build will fail until the remaining production
regex uses are removed by the related PRs.
* OPENNLP-1935: Share one non-empty split loop in StringUtil
EventFields, MascIdentifiers, MecabDictionary and HunspellDictionary each
had the same
split loop with a different character test. StringUtil has it as one
private loop behind
splitNonEmpty, splitOnUnicodeWhitespace and the new mode-aware
splitOnWhitespace, and the
four classes call those.
* OPENNLP-1935: Name the StringUtil helpers in the forbidden list message
* OPENNLP-1935: Allow the user-supplied tokenizer pattern and head-rule tags
Mark the tokenizer alphanumeric-pattern API and the AnCora head-rule tag
matching with
@SuppressForbidden, since both regular expressions are public,
user-supplied contracts.
---
.github/CONTRIBUTING.md | 14 ++++++
dev/forbidden-regex.txt | 30 ++++++++++++
.../opennlp/tools/commons/SuppressForbidden.java | 51 ++++++++++++++++++++
.../main/java/opennlp/tools/util/StringUtil.java | 56 +++++++++++++---------
.../tools/util/StringUtilSplitNonEmptyTest.java | 7 ++-
.../java/opennlp/tools/util/StringUtilTest.java | 40 ++++++++++++++++
.../opennlp/tools/namefind/RegexNameFinder.java | 8 ++++
.../tools/namefind/RegexNameFinderFactory.java | 6 +++
.../parser/lang/es/AncoraSpanishHeadRules.java | 13 +++--
.../tools/stemmer/hunspell/HunspellDictionary.java | 29 ++---------
.../opennlp/tools/tokenize/TokSpanEventStream.java | 4 ++
.../opennlp/tools/tokenize/TokenizerFactory.java | 7 +++
.../java/opennlp/tools/tokenize/TokenizerME.java | 3 ++
.../java/opennlp/tools/tokenize/lang/Factory.java | 34 +++++++++----
pom.xml | 19 ++++++++
15 files changed, 260 insertions(+), 61 deletions(-)
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index 577eb1615..0b02ac4ca 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -9,3 +9,17 @@ To make the process smooth for the project *committers* (those
who review and ac
Please check out the [How to get
involved](http://opennlp.apache.org/get-involved.html) to understand how
contributions are made.
A detailed list of coding standards can be found at [Apache OpenNLP Code
Conventions](http://opennlp.apache.org/code-conventions.html) which also
contains a list of coding guidelines that you should follow.
For pull requests, there is a [check list](PULL_REQUEST_TEMPLATE.md) with
criteria for acceptable contributions.
+
+## Regular expressions in production code
+
+OpenNLP production code scans text with code-point loops and the helpers in
`opennlp.tools.util.StringUtil`
+(for example `splitOnUnicodeWhitespace` and `split(CharSequence, char)`)
instead of `java.util.regex`.
+The build enforces this with forbiddenapis: `dev/forbidden-regex.txt` lists
the forbidden signatures,
+and the `forbidden-regex-main` execution in the root `pom.xml` checks
production classes in the `process-classes`
+phase (tests are not checked).
+An exception exists only for a member with a documented public contract that
is a user-supplied regular expression,
+such as the patterns given to `RegexNameFinder`;
+mark that member (not the class) with
`@opennlp.tools.commons.SuppressForbidden("reason")`.
+A private pattern, or one that is an implementation detail, is converted to a
scan, not annotated:
+the check's report is the list of what remains to convert.
+See OPENNLP-1935 for the background.
diff --git a/dev/forbidden-regex.txt b/dev/forbidden-regex.txt
new file mode 100644
index 000000000..3eecc2937
--- /dev/null
+++ b/dev/forbidden-regex.txt
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# forbiddenapis signatures for the forbidden-regex-main execution in the root
pom.xml.
+# Production code scans text with code-point loops and the
opennlp.tools.util.StringUtil
+# helpers instead of java.util.regex (OPENNLP-1935). A member with a public
contract that is
+# a user-supplied pattern (RegexNameFinder, the tokenizer factories) is marked
with
+# @opennlp.tools.commons.SuppressForbidden and is exempt. Only production
classes are
+# checked; tests may use regular expressions.
+
+@defaultMessage Use a code-point scan or a StringUtil helper
(splitOnUnicodeWhitespace, splitOnWhitespace, split(CharSequence, char),
splitNonEmpty) instead of java.util.regex; a member with a user-supplied
pattern as its public contract is marked
@opennlp.tools.commons.SuppressForbidden (OPENNLP-1935)
+java.util.regex.**
+java.lang.String#split(java.lang.String)
+java.lang.String#split(java.lang.String,int)
+java.lang.String#matches(java.lang.String)
+java.lang.String#replaceAll(java.lang.String,java.lang.String)
+java.lang.String#replaceFirst(java.lang.String,java.lang.String)
+java.util.Scanner#useDelimiter(java.lang.String)
diff --git
a/opennlp-api/src/main/java/opennlp/tools/commons/SuppressForbidden.java
b/opennlp-api/src/main/java/opennlp/tools/commons/SuppressForbidden.java
new file mode 100644
index 000000000..8bde26a2a
--- /dev/null
+++ b/opennlp-api/src/main/java/opennlp/tools/commons/SuppressForbidden.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.commons;
+
+import java.lang.annotation.Documented;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Marks a member with a public contract that is a user-supplied regular
expression,
+ * for example the patterns given to {@code RegexNameFinder} or the
alphanumeric
+ * {@link java.util.regex.Pattern} accepted by the tokenizer factories.
+ * <p>
+ * Production code scans text with code-point loops and the
+ * {@code opennlp.tools.util.StringUtil} helpers instead of {@code
java.util.regex}.
+ * The build enforces this rule with forbiddenapis, using the signatures in
+ * {@code dev/forbidden-regex.txt} (OPENNLP-1935). An annotated member is
exempt
+ * from that check. Apply it to the narrowest member (field, constructor, or
method)
+ * that needs it rather than to the enclosing class, so a regular expression
added to
+ * another member of the same class is still reported.
+ * <p>
+ * The annotation is kept in class files and read by the build only; it has no
+ * effect at runtime.
+ */
+@Documented
+@Retention(RetentionPolicy.CLASS)
+@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
ElementType.CONSTRUCTOR})
+public @interface SuppressForbidden {
+
+ /**
+ * @return The public contract that requires the regular expression.
+ */
+ String value();
+}
diff --git a/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
b/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
index 6b7839d10..c84d98bde 100644
--- a/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
+++ b/opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java
@@ -21,6 +21,7 @@ import java.nio.CharBuffer;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
+import java.util.function.IntPredicate;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -130,26 +131,22 @@ public class StringUtil {
*/
public static String[] splitOnUnicodeWhitespace(CharSequence input) {
ParamChecks.requireNonNullArg(input, "input");
- final List<String> terms = new ArrayList<>();
- final int n = input.length();
- int start = -1;
- int i = 0;
- while (i < n) {
- final int cp = Character.codePointAt(input, i);
- if (isUnicodeWhitespace(cp)) {
- if (start >= 0) {
- terms.add(input.subSequence(start, i).toString());
- start = -1;
- }
- } else if (start < 0) {
- start = i;
- }
- i += Character.charCount(cp);
- }
- if (start >= 0) {
- terms.add(input.subSequence(start, n).toString());
- }
- return terms.toArray(new String[0]);
+ return splitNonEmpty(input, StringUtil::isUnicodeWhitespace);
+ }
+
+ /**
+ * Splits {@code input} on runs of whitespace under the active {@link
WhitespaceMode}, see
+ * {@link #isWhitespace(char)}. Leading and trailing runs are ignored, so
whitespace-only
+ * input yields an empty array. Use {@link
#splitOnUnicodeWhitespace(CharSequence)} where
+ * the Unicode definition should apply regardless of the mode.
+ *
+ * @param input The text to split. Must not be {@code null}.
+ * @return The non-whitespace fields in order.
+ * @throws IllegalArgumentException If {@code input} is {@code null}.
+ */
+ public static String[] splitOnWhitespace(CharSequence input) {
+ ParamChecks.requireNonNullArg(input, "input");
+ return splitNonEmpty(input, StringUtil::isWhitespace);
}
/**
@@ -282,11 +279,25 @@ public class StringUtil {
throw new IllegalArgumentException("separators must not contain a
surrogate");
}
}
+ return splitNonEmpty(input, c -> c <= Character.MAX_VALUE &&
isAnyOf((char) c, separators));
+ }
+
+ /**
+ * Splits {@code input} on runs of the code points that {@code separator}
accepts and
+ * drops the empty fields that leading, trailing or adjacent separators
would produce. The
+ * one loop behind every non-empty split of this class.
+ *
+ * @param input The text to split, not {@code null}.
+ * @param separator Accepts a code point that separates fields.
+ * @return The non-empty fields in order.
+ */
+ private static String[] splitNonEmpty(CharSequence input, IntPredicate
separator) {
final List<String> fields = new ArrayList<>();
final int length = input.length();
int start = -1;
- for (int i = 0; i < length; i++) {
- if (isAnyOf(input.charAt(i), separators)) {
+ for (int i = 0; i < length; ) {
+ final int codePoint = Character.codePointAt(input, i);
+ if (separator.test(codePoint)) {
if (start >= 0) {
fields.add(input.subSequence(start, i).toString());
start = -1;
@@ -294,6 +305,7 @@ public class StringUtil {
} else if (start < 0) {
start = i;
}
+ i += Character.charCount(codePoint);
}
if (start >= 0) {
fields.add(input.subSequence(start, length).toString());
diff --git
a/opennlp-api/src/test/java/opennlp/tools/util/StringUtilSplitNonEmptyTest.java
b/opennlp-api/src/test/java/opennlp/tools/util/StringUtilSplitNonEmptyTest.java
index e8c538ffd..c9228ba16 100644
---
a/opennlp-api/src/test/java/opennlp/tools/util/StringUtilSplitNonEmptyTest.java
+++
b/opennlp-api/src/test/java/opennlp/tools/util/StringUtilSplitNonEmptyTest.java
@@ -36,6 +36,8 @@ public class StringUtilSplitNonEmptyTest {
private static final String SMILEY = "\uD83D\uDE00";
+ private static final String SUPPLEMENTARY_WITH_SPACE_LOW_BITS =
"\uD800\uDC20";
+
private static Stream<Arguments> cases() {
return Stream.of(
Arguments.of("", new char[] {':'}, new String[0]),
@@ -54,6 +56,8 @@ public class StringUtilSplitNonEmptyTest {
Arguments.of("a\\sb", new char[] {' '}, new String[] {"a\\sb"}),
// only the listed separators split, other whitespace stays in the
field
Arguments.of("a\u00A0b\u2003c", new char[] {' '}, new String[]
{"a\u00A0b\u2003c"}),
+ Arguments.of(SUPPLEMENTARY_WITH_SPACE_LOW_BITS + " x", new char[] {'
'},
+ new String[] {SUPPLEMENTARY_WITH_SPACE_LOW_BITS, "x"}),
Arguments.of(SMILEY + ":" + SMILEY, new char[] {':'}, new String[]
{SMILEY, SMILEY}));
}
@@ -82,7 +86,8 @@ public class StringUtilSplitNonEmptyTest {
*/
@Test
void testMatchesRegexSplitWithoutEmptyFields() {
- final String[] pieces = {":", ";", " ", "\t", ".", "|", "a", "b", "xy",
SMILEY, "0"};
+ final String[] pieces = {":", ";", " ", "\t", ".", "|", "a", "b", "xy",
SMILEY,
+ SUPPLEMENTARY_WITH_SPACE_LOW_BITS, "0"};
final char[][] separatorSets = {{':'}, {';'}, {' ', '\t'}, {'.', '|'},
{':', ';', ' '}};
final Pattern[] patterns = new Pattern[separatorSets.length];
for (int s = 0; s < separatorSets.length; s++) {
diff --git a/opennlp-api/src/test/java/opennlp/tools/util/StringUtilTest.java
b/opennlp-api/src/test/java/opennlp/tools/util/StringUtilTest.java
index 1aa05bb8b..0cd9256fb 100644
--- a/opennlp-api/src/test/java/opennlp/tools/util/StringUtilTest.java
+++ b/opennlp-api/src/test/java/opennlp/tools/util/StringUtilTest.java
@@ -556,6 +556,46 @@ public class StringUtilTest {
StringUtil.splitOnUnicodeWhitespace(withInfo));
}
+ @Test
+ void testSplitOnWhitespaceNullThrows() {
+ Assertions.assertThrows(IllegalArgumentException.class, () ->
StringUtil.splitOnWhitespace(null));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"", " ", " \t\n ", "\u00A0\u3000"})
+ void testSplitOnWhitespaceBlankYieldsEmpty(String input) {
+ Assertions.assertArrayEquals(new String[0],
StringUtil.splitOnWhitespace(input));
+ }
+
+ @Test
+ void testSplitOnWhitespaceCollapsesRunsAndIgnoresEnds() {
+ Assertions.assertArrayEquals(new String[] {"a", "b", "c"},
+ StringUtil.splitOnWhitespace(" a \t\u00A0b\u3000\r\nc "));
+ Assertions.assertArrayEquals(new String[] {"a\u200Bb", "\uD83D\uDE00"},
+ StringUtil.splitOnWhitespace("a\u200Bb \uD83D\uDE00"));
+ }
+
+ @Test
+ void testSplitOnWhitespaceFollowsTheWhitespaceMode() {
+ // U+0085 has the White_Space property but is not Character.isWhitespace;
U+001C is the
+ // reverse. The mode-aware split follows the active mode, the Unicode
split does not.
+ final String withNel = "a\u0085b";
+ final String withInfo = "a\u001Cb";
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
StringUtil.splitOnWhitespace(withNel));
+ Assertions.assertArrayEquals(new String[] {"a\u001Cb"},
StringUtil.splitOnWhitespace(withInfo));
+
+ WhitespaceMode.setActive(WhitespaceMode.LEGACY);
+ Assertions.assertArrayEquals(new String[] {"a\u0085b"},
StringUtil.splitOnWhitespace(withNel));
+ Assertions.assertArrayEquals(new String[] {"a", "b"},
StringUtil.splitOnWhitespace(withInfo));
+ }
+
+ @Test
+ void testSplitOnWhitespaceAgreesWithTheUnicodeSplitInCurrentMode() {
+ final String input = " t0\u2003t1\u0085t2 \u3000 t3\uD801\uDC12 ";
+ Assertions.assertArrayEquals(StringUtil.splitOnUnicodeWhitespace(input),
+ StringUtil.splitOnWhitespace(input));
+ }
+
@Test
void testSplitOnUnicodeWhitespaceAllWhiteSpaceCodePointsAsMixedSeparators() {
// Build "t0 <ws0> t1 <ws1> ... tN" using every White_Space code point.
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinder.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinder.java
index 205e4052f..5e6a66f94 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinder.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinder.java
@@ -24,6 +24,7 @@ import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import opennlp.tools.commons.SuppressForbidden;
import opennlp.tools.util.ParamChecks;
import opennlp.tools.util.Span;
@@ -32,6 +33,7 @@ import opennlp.tools.util.Span;
*/
public final class RegexNameFinder implements TokenNameFinder {
+ @SuppressForbidden("user-supplied name finder patterns")
private Pattern[] mPatterns;
private String sType;
private Map<String, Pattern[]> regexMap;
@@ -56,6 +58,7 @@ public final class RegexNameFinder implements TokenNameFinder
{
*
* @throws IllegalArgumentException Thrown if {@code patterns} were {@code
null} or empty.
*/
+ @SuppressForbidden("user-supplied name finder patterns")
public RegexNameFinder(Pattern[] patterns, String type) {
if (patterns == null || patterns.length == 0) {
throw new IllegalArgumentException("patterns must not be null or
empty!");
@@ -65,6 +68,7 @@ public final class RegexNameFinder implements TokenNameFinder
{
sType = type;
}
+ @SuppressForbidden("user-supplied name finder patterns")
@Override
public Span[] find(String[] tokens) {
Map<Integer, Integer> sentencePosTokenMap = new HashMap<>();
@@ -131,10 +135,12 @@ public final class RegexNameFinder implements
TokenNameFinder {
* @param text The text to use.
* @return An array of {@link Span} instances representing the annotations.
*/
+ @SuppressForbidden("user-supplied name finder patterns")
public Span[] find(String text) {
return getAnnotations(text);
}
+ @SuppressForbidden("user-supplied name finder patterns")
private Span[] getAnnotations(String text) {
Collection<Span> annotations = new LinkedList<>();
if (regexMap != null) {
@@ -168,6 +174,7 @@ public final class RegexNameFinder implements
TokenNameFinder {
/**
* @return Retrieves the {@link Pattern matching patterns} used.
*/
+ @SuppressForbidden("user-supplied name finder patterns")
public Pattern[] getMatchingPatterns() {
return mPatterns;
}
@@ -175,6 +182,7 @@ public final class RegexNameFinder implements
TokenNameFinder {
/**
* @param mPatterns The {@link Pattern matching patterns} to be set.
*/
+ @SuppressForbidden("user-supplied name finder patterns")
public void setMatchingPatterns(Pattern[] mPatterns) {
this.mPatterns = mPatterns;
}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
index 7a6097a5a..3f8e798ec 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
@@ -21,6 +21,7 @@ import java.util.HashMap;
import java.util.Map;
import java.util.regex.Pattern;
+import opennlp.tools.commons.SuppressForbidden;
import opennlp.tools.util.ParamChecks;
/**
@@ -89,6 +90,7 @@ public class RegexNameFinderFactory {
USA_PHONE_NUM {
@Override
+ @SuppressForbidden("default patterns of the regex name finder")
public Map<String, Pattern[]> getRegexMap() {
Pattern[] p = new Pattern[1];
// p[0] = Pattern.compile("([\\+(]?(\\d){2,}[)]?[- \\.]?(\\d){2,}[-
\\.]?(\\d){2,}[- \\.]?
@@ -108,6 +110,7 @@ public class RegexNameFinderFactory {
},
EMAIL {
@Override
+ @SuppressForbidden("default patterns of the regex name finder")
public Map<String, Pattern[]> getRegexMap() {
Pattern[] p = new Pattern[1];
// Every quantifier is bounded by a constant, which removes both the
exponential
@@ -134,6 +137,7 @@ public class RegexNameFinderFactory {
},
URL {
@Override
+ @SuppressForbidden("default patterns of the regex name finder")
public Map<String, Pattern[]> getRegexMap() {
Pattern[] p = new Pattern[1];
// Flattened to single-level groups and every quantifier bounded by a
constant.
@@ -161,6 +165,7 @@ public class RegexNameFinderFactory {
},
MGRS {
@Override
+ @SuppressForbidden("default patterns of the regex name finder")
public Map<String, Pattern[]> getRegexMap() {
Pattern[] p = new Pattern[1];
p[0] =
Pattern.compile("\\d{1,2}[A-Za-z]\\s*[A-Za-z]{2}\\s*\\d{1,5}\\s*\\d{1,5}",
@@ -177,6 +182,7 @@ public class RegexNameFinderFactory {
},
DEGREES_MIN_SEC_LAT_LON {
@Override
+ @SuppressForbidden("default patterns of the regex name finder")
public Map<String, Pattern[]> getRegexMap() {
Pattern[] p = new Pattern[1];
p[0] =
Pattern.compile("([-|\\+]?\\d{1,3}[d|D|\\u00B0|\\s](\\s*\\d{1,2}['|\\u2019|\\s])"
+
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/lang/es/AncoraSpanishHeadRules.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/lang/es/AncoraSpanishHeadRules.java
index 61e38595e..68c246af1 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/lang/es/AncoraSpanishHeadRules.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/lang/es/AncoraSpanishHeadRules.java
@@ -37,6 +37,7 @@ import java.util.Stack;
import java.util.StringTokenizer;
import java.util.regex.Pattern;
+import opennlp.tools.commons.SuppressForbidden;
import opennlp.tools.parser.Constituent;
import opennlp.tools.parser.GapLabeler;
import opennlp.tools.parser.HeadRules;
@@ -85,6 +86,7 @@ public class AncoraSpanishHeadRules implements HeadRules,
GapLabeler, Serializab
private static class HeadRule {
public final boolean leftToRight;
public final String[] tags;
+ @SuppressForbidden("head-rule tags are regular expressions from a
user-supplied rules file")
public final Pattern[] tagPatterns;
public HeadRule(boolean l2r, String[] tags) {
@@ -119,21 +121,23 @@ public class AncoraSpanishHeadRules implements HeadRules,
GapLabeler, Serializab
}
}
- // Tag patterns are regexes (e.g. "AQA.*", "GRUP\\.A"), not literals, so
they must be
- // matched with Pattern/matches() semantics - they are precompiled once here
(and once
- // per HeadRule, see HeadRule.tagPatterns) instead of via String.matches(),
which would
- // otherwise recompile the same regex on every single constituent comparison.
+ // Tags are regular expressions (e.g. "AQA.*", "GRUP\\.A") matched against
the whole
+ // constituent type; they are compiled once, here and per HeadRule.
private static final String[] TAGS1 =
{"AQA.*", "AQC.*", "GRUP\\.A", "S\\.A", "NC.*S.*", "NP.*", "NC.*P.*",
"GRUP\\.NOM"};
+ @SuppressForbidden("head-rule tags are regular expressions from a
user-supplied rules file")
private static final Pattern[] TAGS1_PATTERNS = compile(TAGS1);
private static final String[] TAGS2 = {"\\$", "GRUP\\.A", "SA"};
+ @SuppressForbidden("head-rule tags are regular expressions from a
user-supplied rules file")
private static final Pattern[] TAGS2_PATTERNS = compile(TAGS2);
private static final String[] TAGS3 =
{"AQ0.*", "AQ[AC].*", "AO.*", "GRUP\\.A", "S\\.A", "RG", "RN",
"GRUP\\.NOM"};
+ @SuppressForbidden("head-rule tags are regular expressions from a
user-supplied rules file")
private static final Pattern[] TAGS3_PATTERNS = compile(TAGS3);
+ @SuppressForbidden("head-rule tags are regular expressions from a
user-supplied rules file")
private static Pattern[] compile(String[] tags) {
Pattern[] patterns = new Pattern[tags.length];
for (int i = 0; i < tags.length; i++) {
@@ -170,6 +174,7 @@ public class AncoraSpanishHeadRules implements HeadRules,
GapLabeler, Serializab
}
@Override
+ @SuppressForbidden("head-rule tags are regular expressions from a
user-supplied rules file")
public Parse getHead(Parse[] constituents, String type) {
if (Parser.TOK_NODE.equals(constituents[0].getType())) {
return null;
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
index 294103598..493c8751a 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java
@@ -2295,7 +2295,7 @@ public final class HunspellDictionary {
final String trimmed = trim(line);
if (trimmed.startsWith(SET_PREFIX) ||
trimmed.startsWith(SET_TAB_PREFIX)) {
// the encoding is the first field after the directive; later fields
are ignored
- final String name = split(trimmed.substring(SET_PREFIX.length()))[0];
+ final String name =
StringUtil.splitOnWhitespace(trimmed.substring(SET_PREFIX.length()))[0];
try {
return Charset.forName(name);
} catch (IllegalCharsetNameException | UnsupportedCharsetException e) {
@@ -2382,7 +2382,7 @@ public final class HunspellDictionary {
final String[] lines = splitLines(StringUtil.stripByteOrderMark(content));
final String[][] fieldsByLine = new String[lines.length][];
for (int i = 0; i < lines.length; i++) {
- fieldsByLine[i] = split(lines[i]);
+ fieldsByLine[i] = StringUtil.splitOnWhitespace(lines[i]);
}
result.flagMode = readFlagMode(fieldsByLine);
result.flagAliases.addAll(readFlagAliases(fieldsByLine, result.flagMode));
@@ -2768,7 +2768,8 @@ public final class HunspellDictionary {
flags = parseAliasedFlags(flagRun, affix.flagMode, affix.flagAliases,
i + 1).clone();
}
if (morphology >= 0) {
- affix.entryMorphology.put(flags,
parseMorphology(split(line.substring(morphology)),
+ affix.entryMorphology.put(flags,
+
parseMorphology(StringUtil.splitOnWhitespace(line.substring(morphology)),
affix.morphologyAliases, i + 1));
}
entries.computeIfAbsent(removeIgnored(word.replace("\\/", "/"),
affix.ignoredCharacters),
@@ -3130,26 +3131,4 @@ public final class HunspellDictionary {
}
return lines.toArray(new String[0]);
}
-
- /**
- * Splits a line on whitespace with a single character scan.
- *
- * @param line The line to split.
- * @return The whitespace-separated fields, without empty ones. Never {@code
null}.
- */
- private static String[] split(String line) {
- final List<String> parts = new ArrayList<>();
- int start = -1;
- for (int i = 0; i <= line.length(); i++) {
- if (i == line.length() || StringUtil.isWhitespace(line.charAt(i))) {
- if (start >= 0) {
- parts.add(line.substring(start, i));
- start = -1;
- }
- } else if (start < 0) {
- start = i;
- }
- }
- return parts.toArray(new String[0]);
- }
}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokSpanEventStream.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokSpanEventStream.java
index 36f4a98de..b5e297aff 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokSpanEventStream.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokSpanEventStream.java
@@ -26,6 +26,7 @@ import java.util.regex.Pattern;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import opennlp.tools.commons.SuppressForbidden;
import opennlp.tools.ml.model.Event;
import opennlp.tools.tokenize.lang.Factory;
import opennlp.tools.util.AbstractEventStream;
@@ -44,6 +45,7 @@ public class TokSpanEventStream extends
AbstractEventStream<TokenSample> {
private final boolean skipAlphaNumerics;
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
private final Pattern alphaNumeric;
/**
@@ -56,6 +58,7 @@ public class TokSpanEventStream extends
AbstractEventStream<TokenSample> {
* {@link Factory#DEFAULT_ALPHANUMERIC}.
* @param cg A {@link TokenContextGenerator} which should be used for the
event stream {@code d}.
*/
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
public TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean
skipAlphaNumerics,
Pattern alphaNumeric, TokenContextGenerator cg) {
super(tokenSamples);
@@ -95,6 +98,7 @@ public class TokSpanEventStream extends
AbstractEventStream<TokenSample> {
* @return An {@link Iterator} for text {@link Event events} representing
the {@code tokenSample}.
*/
@Override
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
protected Iterator<Event> createEvents(TokenSample tokenSample) {
List<Event> events = new ArrayList<>(50);
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerFactory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerFactory.java
index 2cbb9461d..eabc986cb 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerFactory.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerFactory.java
@@ -22,6 +22,7 @@ import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;
+import opennlp.tools.commons.SuppressForbidden;
import opennlp.tools.dictionary.Dictionary;
import opennlp.tools.tokenize.lang.Factory;
import opennlp.tools.util.BaseToolFactory;
@@ -38,6 +39,7 @@ public class TokenizerFactory extends BaseToolFactory {
private String languageCode;
private Dictionary abbreviationDictionary;
private Boolean useAlphaNumericOptimization = false;
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
private Pattern alphaNumericPattern;
private static final String ABBREVIATIONS_ENTRY_NAME =
"abbreviations.dictionary";
@@ -62,6 +64,7 @@ public class TokenizerFactory extends BaseToolFactory {
* (default is: {@code "^[A-Za-z0-9]+$"},
provided by
* {@link Factory#DEFAULT_ALPHANUMERIC}.
*/
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
public TokenizerFactory(String languageCode, Dictionary
abbreviationDictionary,
boolean useAlphaNumericOptimization, Pattern
alphaNumericPattern) {
this.init(languageCode, abbreviationDictionary,
@@ -76,6 +79,7 @@ public class TokenizerFactory extends BaseToolFactory {
* (default is: {@code "^[A-Za-z0-9]+$"},
provided by
* {@link Factory#DEFAULT_ALPHANUMERIC}.
*/
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
protected void init(String languageCode, Dictionary abbreviationDictionary,
boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) {
this.languageCode = languageCode;
@@ -111,6 +115,7 @@ public class TokenizerFactory extends BaseToolFactory {
}
@Override
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
public Map<String, String> createManifestEntries() {
Map<String, String> manifestEntries = super.createManifestEntries();
@@ -140,6 +145,7 @@ public class TokenizerFactory extends BaseToolFactory {
*
* @throws InvalidFormatException Thrown if one of the input parameters
doesn't comply the expected format.
*/
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
public static TokenizerFactory create(String subclassName, String
languageCode,
Dictionary abbreviationDictionary,
boolean useAlphaNumericOptimization,
@@ -166,6 +172,7 @@ public class TokenizerFactory extends BaseToolFactory {
/**
* @return Retrieves the (user-)specified alphanumeric {@link Pattern} or a
default.
*/
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
public Pattern getAlphaNumericPattern() {
if (this.alphaNumericPattern == null) {
if (this.artifactProvider != null) {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerME.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerME.java
index 859d7b14a..256fea895 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerME.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerME.java
@@ -24,6 +24,7 @@ import java.util.List;
import java.util.Map;
import java.util.regex.Pattern;
+import opennlp.tools.commons.SuppressForbidden;
import opennlp.tools.commons.ThreadSafe;
import opennlp.tools.dictionary.Dictionary;
import opennlp.tools.ml.ArrayMath;
@@ -102,6 +103,7 @@ public class TokenizerME extends AbstractTokenizer
implements Probabilistic {
*/
public static final String NO_SPLIT = "F";
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
private final Pattern alphanumeric;
/*
@@ -200,6 +202,7 @@ public class TokenizerME extends AbstractTokenizer
implements Probabilistic {
* @return A {@link Span} array containing individual tokens as elements.
*/
@Override
+ @SuppressForbidden("user-supplied alphanumeric pattern stored in the
tokenizer model")
public Span[] tokenizePos(String d) {
Span[] tokens = (keepNewLines ? WHITESPACE_KEEPING_NEW_LINES :
WHITESPACE).tokenizePos(d);
List<Span> localTokens = new ArrayList<>();
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lang/Factory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lang/Factory.java
index ebb236ccc..99c53d8f3 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lang/Factory.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lang/Factory.java
@@ -20,37 +20,52 @@ package opennlp.tools.tokenize.lang;
import java.util.Set;
import java.util.regex.Pattern;
+import opennlp.tools.commons.SuppressForbidden;
import opennlp.tools.tokenize.DefaultTokenContextGenerator;
import opennlp.tools.tokenize.TokenContextGenerator;
public class Factory {
- public static final Pattern DEFAULT_ALPHANUMERIC =
Pattern.compile("^[A-Za-z0-9]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ public static final Pattern DEFAULT_ALPHANUMERIC = compile("^[A-Za-z0-9]+$");
- private static final Pattern PORTUGUESE =
Pattern.compile("^[0-9a-záãâàéêíóõôúüçA-ZÁÃÂÀÉÊÍÓÕÔÚÜÇ]+$");
- private static final Pattern FRENCH =
Pattern.compile("^[a-zA-Z0-9àâäèéêëîïôœùûüÿçÀÂÄÈÉÊËÎÏÔŒÙÛÜŸÇ]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ private static final Pattern PORTUGUESE =
compile("^[0-9a-záãâàéêíóõôúüçA-ZÁÃÂÀÉÊÍÓÕÔÚÜÇ]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ private static final Pattern FRENCH =
compile("^[a-zA-Z0-9àâäèéêëîïôœùûüÿçÀÂÄÈÉÊËÎÏÔŒÙÛÜŸÇ]+$");
// From: https://www.sttmedia.com/characterfrequency-dutch
- private static final Pattern DUTCH =
Pattern.compile("^[A-Za-z0-9äöüëèéïijÄÖÜËÉÈÏIJ]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ private static final Pattern DUTCH =
compile("^[A-Za-z0-9äöüëèéïijÄÖÜËÉÈÏIJ]+$");
// Note: The extra é and É are included to cover German "Lehnwörter" such as
"Café"
- private static final Pattern GERMAN =
Pattern.compile("^[A-Za-z0-9äéöüÄÉÖÜß]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ private static final Pattern GERMAN = compile("^[A-Za-z0-9äéöüÄÉÖÜß]+$");
// From: https://en.wikipedia.org/wiki/Polish_alphabet
// https://pl.wikipedia.org/wiki/Alfabet_polski
- private static final Pattern POLISH =
Pattern.compile("^[A-Za-z0-9żźćńółęąśŻŹĆĄŚĘŁÓŃ]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ private static final Pattern POLISH =
compile("^[A-Za-z0-9żźćńółęąśŻŹĆĄŚĘŁÓŃ]+$");
// From: https://it.wikipedia.org/wiki/Alfabeto_italiano
- private static final Pattern ITALIAN =
Pattern.compile("^[0-9a-zàèéìîíòóùüA-ZÀÈÉÌÎÍÒÓÙÜ]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ private static final Pattern ITALIAN =
compile("^[0-9a-zàèéìîíòóùüA-ZÀÈÉÌÎÍÒÓÙÜ]+$");
// From: https://en.wikiversity.org/wiki/Alphabet/Spanish_alphabet &
//
https://en.wikipedia.org/wiki/Spanish_orthography#Alphabet_in_Spanish &
// https://www.fundeu.es/consulta/tilde-en-la-y-y-griega-o-ye-24786/
- private static final Pattern SPANISH =
Pattern.compile("^[0-9a-záéíóúüýñA-ZÁÉÍÓÚÝÑ]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ private static final Pattern SPANISH =
compile("^[0-9a-záéíóúüýñA-ZÁÉÍÓÚÝÑ]+$");
// From: https://en.wikipedia.org/wiki/Catalan_orthography#Spelling_patterns
- private static final Pattern CATALAN =
Pattern.compile("^[0-9a-zàèéíïòóúüçA-ZÀÈÉÍÏÒÓÚÜÇ]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ private static final Pattern CATALAN =
compile("^[0-9a-zàèéíïòóúüçA-ZÀÈÉÍÏÒÓÚÜÇ]+$");
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
+ private static Pattern compile(String regex) {
+ return Pattern.compile(regex);
+ }
+
/**
* Gets the alphanumeric pattern for a language.
*
@@ -58,6 +73,7 @@ public class Factory {
* {@link #DEFAULT_ALPHANUMERIC} pattern will be
returned.
* @return The alphanumeric {@link Pattern} for the language, or the default
pattern.
*/
+ @SuppressForbidden("default alphanumeric patterns of the tokenizer model")
public Pattern getAlphanumeric(String languageCode) {
// For reference: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
if ("es".equals(languageCode) || "spa".equals(languageCode)) {
diff --git a/pom.xml b/pom.xml
index f7a98a098..157d0776a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -447,6 +447,25 @@ org.xml.sax.helpers.XMLReaderFactory#createXMLReader(**)
<goal>testCheck</goal>
</goals>
</execution>
+ <execution>
+ <!-- Production classes
only: no java.util.regex outside members marked
+
@opennlp.tools.commons.SuppressForbidden (OPENNLP-1935). -->
+
<id>forbidden-regex-main</id>
+
<phase>process-classes</phase>
+ <goals>
+
<goal>check</goal>
+ </goals>
+ <configuration>
+
<bundledSignatures combine.self="override" />
+ <signatures
combine.self="override" />
+
<signaturesFiles>
+
<signaturesFile>${maven.multiModuleProjectDirectory}/dev/forbidden-regex.txt</signaturesFile>
+
</signaturesFiles>
+
<suppressAnnotations>
+
<suppressAnnotation>opennlp.tools.commons.SuppressForbidden</suppressAnnotation>
+
</suppressAnnotations>
+ </configuration>
+ </execution>
</executions>
</plugin>
</plugins>
