This is an automated email from the ASF dual-hosted git repository. rzo1 pushed a commit to branch feat/GH-1479-tokenizer-character-policy in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 44327b054bacc809c3d62c8640ed8bc3402f72d8 Author: Kristian Rickert <[email protected]> AuthorDate: Fri Oct 23 23:25:31 2026 -0400 GH-1479: Configure the tokenizer alphanumeric optimization with TokenizerCharacterPolicy instead of a regex TokenizerME, TokSpanEventStream, TokenizerFactory and the language Factory take a TokenizerCharacterPolicy for the alphanumeric optimization. The Pattern variants of the TokenizerFactory constructor, TokenizerFactory.create and the TokSpanEventStream constructor, getAlphaNumericPattern(), Factory.getAlphanumeric(String) and Factory.DEFAULT_ALPHANUMERIC stay as deprecated for removal in 4.0, and only they keep @SuppressForbidden. New overloads without the last argument use the policy of the language. The factory writes the policy to the model manifest as characterPolicy, in the text form of TokenizerCharacterPolicy.toString(), which TokenizerCharacterPolicy.parse(String) reads; for a language policy it also writes the alphaNumericPattern of earlier releases. A model with one of the alphaNumericPattern values OpenNLP wrote loads with the matching language policy, and a model with a custom pattern loads and tests candidate tokens with that regular expression. A manifest without either property uses the policy for its language code. The language policies are tested against their old regular expressions, TokenizerME output is pinned with and without the optimization, and the manifest tests cover the policy, every legacy pattern, a custom and an invalid pattern, a malformed policy and a manifest without either, plus the deprecated constructors, create and getters. TokenizerCharacterPolicyBenchmark compares the policy with the regular expressions. The tokenizer chapter and the README describe the policy, the deprecation and the migration. Fixes #1479 Signed-off-by: Kristian Rickert <[email protected]> Co-authored-by: Claude Code 2.1.296 (Opus 5.5) <[email protected]> --- README.md | 1 + dev/forbidden-regex.txt | 2 +- .../tokenizer/TokenizerCrossValidatorTool.java | 2 +- .../cmdline/tokenizer/TokenizerTrainerTool.java | 2 +- .../formats/masc/MascTokenSampleStreamTest.java | 2 +- .../TokenizerCharacterPolicyBenchmark.java | 133 +++++++ .../tools/tokenize/TokenizerMEBenchmark.java | 2 +- .../opennlp/tools/tokenize/TokSpanEventStream.java | 59 +++- .../tools/tokenize/TokenizerCharacterPolicy.java | 162 +++++++++ .../opennlp/tools/tokenize/TokenizerFactory.java | 341 +++++++++++++++--- .../java/opennlp/tools/tokenize/TokenizerME.java | 16 +- .../java/opennlp/tools/tokenize/lang/Factory.java | 133 +++++-- .../opennlp/tools/ThreadSafetyBenchmarkIT.java | 2 +- .../tools/tokenize/DummyTokenizerFactory.java | 9 +- .../tools/tokenize/LegacyAlphaNumericPatterns.java | 55 +++ .../tools/tokenize/TokSpanEventStreamTest.java | 114 ++++++ .../tokenize/TokenizerCharacterPolicyTest.java | 133 +++++++ .../tools/tokenize/TokenizerFactoryTest.java | 386 +++++++++++++++++---- .../opennlp/tools/tokenize/TokenizerMETest.java | 63 +++- .../opennlp/tools/tokenize/TokenizerTestUtil.java | 14 +- .../opennlp/tools/tokenize/lang/FactoryTest.java | 164 +++++++++ opennlp-docs/src/docbkx/tokenizer.xml | 46 ++- .../opennlp/tools/eval/ArvoresDeitadasEval.java | 2 +- .../tools/eval/EnglishTokenSampleStreamEval.java | 2 +- 24 files changed, 1668 insertions(+), 177 deletions(-) diff --git a/README.md b/README.md index 0a9f83f2c..88454edbd 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,7 @@ The list below covers the changes most likely to affect existing code. It is not - `RealBasicEventStream`: a line that contains only an outcome is read as an event without contexts and no longer ends the stream. - Parser: `ParserFactory.create` and the chunking and tree insert `Parser` constructors throw `IllegalArgumentException` for a beam size below 1 or an advance percentage outside (0, 1] ([GH-1476](https://github.com/apache/opennlp/issues/1476)). - Model resolver: `AbstractClassPathModelFinder.findModels` returns an unmodifiable set, so the classpath finders and `DefaultClassPathModelProvider` can be shared between threads ([GH-1506](https://github.com/apache/opennlp/issues/1506)). +- Tokenizer: the alphanumeric optimization is configured with a `TokenizerCharacterPolicy`. The `Pattern` variants of the `TokenizerFactory` constructor, `TokenizerFactory.create` and the `TokSpanEventStream` constructor, `TokenizerFactory.getAlphaNumericPattern()`, and `opennlp.tools.tokenize.lang.Factory.getAlphanumeric(String)` and `Factory.DEFAULT_ALPHANUMERIC` are deprecated for removal in 4.0. A call that passes a `null` literal as the pattern no longer compiles because it matches [...] ### Thread safety diff --git a/dev/forbidden-regex.txt b/dev/forbidden-regex.txt index 3eecc2937..2ae65b772 100644 --- a/dev/forbidden-regex.txt +++ b/dev/forbidden-regex.txt @@ -16,7 +16,7 @@ # forbiddenapis signatures for the forbidden-regex-main execution in the root pom.xml. # Production code scans text with code-point loops and the opennlp.tools.util.StringUtil # helpers instead of java.util.regex (OPENNLP-1935). A member with a public contract that is -# a user-supplied pattern (RegexNameFinder, the tokenizer factories) is marked with +# a user-supplied pattern (for example RegexNameFinder) is marked with # @opennlp.tools.commons.SuppressForbidden and is exempt. Only production classes are # checked; tests may use regular expressions. diff --git a/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/tokenizer/TokenizerCrossValidatorTool.java b/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/tokenizer/TokenizerCrossValidatorTool.java index 5dcfc9ffa..77d557773 100644 --- a/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/tokenizer/TokenizerCrossValidatorTool.java +++ b/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/tokenizer/TokenizerCrossValidatorTool.java @@ -72,7 +72,7 @@ public final class TokenizerCrossValidatorTool TokenizerFactory tokFactory = TokenizerFactory.create( params.getFactory(), params.getLang(), dict, - params.getAlphaNumOpt(), null); + params.getAlphaNumOpt()); validator = new opennlp.tools.tokenize.TokenizerCrossValidator(mlParams, tokFactory, listener); diff --git a/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/tokenizer/TokenizerTrainerTool.java b/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/tokenizer/TokenizerTrainerTool.java index 4f5389ab2..a9c2ba12b 100644 --- a/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/tokenizer/TokenizerTrainerTool.java +++ b/opennlp-core/opennlp-cli/src/main/java/opennlp/tools/cmdline/tokenizer/TokenizerTrainerTool.java @@ -100,7 +100,7 @@ public final class TokenizerTrainerTool TokenizerFactory tokFactory = TokenizerFactory.create( params.getFactory(), params.getLang(), dict, - params.getAlphaNumOpt(), null); + params.getAlphaNumOpt()); model = opennlp.tools.tokenize.TokenizerME.train(sampleStream, tokFactory, mlParams); diff --git a/opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/masc/MascTokenSampleStreamTest.java b/opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/masc/MascTokenSampleStreamTest.java index d5a925eff..0af7c5c83 100644 --- a/opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/masc/MascTokenSampleStreamTest.java +++ b/opennlp-core/opennlp-formats/src/test/java/opennlp/tools/formats/masc/MascTokenSampleStreamTest.java @@ -134,7 +134,7 @@ public class MascTokenSampleStreamTest extends AbstractMascSampleStreamTest { trainingParameters.put(Parameters.ITERATIONS_PARAM, 20); TokenizerModel model = TokenizerME.train(trainTokens, new TokenizerFactory( - "en", null, false, null), trainingParameters); + "en", null, false), trainingParameters); ObjectStream<TokenSample> testTokens = new MascTokenSampleStream( new MascDocumentStream(directory, true, fileFilter)); diff --git a/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/tokenize/TokenizerCharacterPolicyBenchmark.java b/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/tokenize/TokenizerCharacterPolicyBenchmark.java new file mode 100644 index 000000000..4a78a0f5d --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/tokenize/TokenizerCharacterPolicyBenchmark.java @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.tokenize; + +import java.util.concurrent.TimeUnit; +import java.util.regex.Pattern; + +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.BenchmarkMode; +import org.openjdk.jmh.annotations.Fork; +import org.openjdk.jmh.annotations.Level; +import org.openjdk.jmh.annotations.Measurement; +import org.openjdk.jmh.annotations.Mode; +import org.openjdk.jmh.annotations.OutputTimeUnit; +import org.openjdk.jmh.annotations.Param; +import org.openjdk.jmh.annotations.Scope; +import org.openjdk.jmh.annotations.Setup; +import org.openjdk.jmh.annotations.State; +import org.openjdk.jmh.annotations.Warmup; +import org.openjdk.jmh.infra.Blackhole; +import org.openjdk.jmh.runner.Runner; +import org.openjdk.jmh.runner.options.Options; +import org.openjdk.jmh.runner.options.OptionsBuilder; + +import opennlp.tools.tokenize.lang.Factory; + +/** + * JMH benchmark for the check of the alphanumeric optimization in {@link TokenizerME}: whether a + * whitespace-separated candidate token is accepted whole. It compares + * {@link TokenizerCharacterPolicy#test(CharSequence)} with the regular expression that releases + * before 3.0 stored in the model manifest for the same language, on the same candidates. + */ +@BenchmarkMode(Mode.AverageTime) +@OutputTimeUnit(TimeUnit.NANOSECONDS) +@Warmup(iterations = 5, time = 1) +@Measurement(iterations = 10, time = 1) +@Fork(2) +public class TokenizerCharacterPolicyBenchmark { + + /** Candidate tokens of two or more characters, as the tokenizer checks them. */ + private static final String[] CANDIDATES = { + "The", "driver", "injured", "accident.", "Edinburgh", "Edinburgh.", "letter", "right", + "OpenNLP", "provides", "tools", "NLP.", "Pierre", "Vinken,", "61", "years", "old,", + "nonexecutive", "Nov.", "29.", "555-1234", "(anytime).", "3.5\"", "don't", "it's", + "N.V.,", "e-mail", "2026", "acción", "niño", "señor", "árbol", "Málaga,", "über", + "Straße", "Ärger", "garçon", "Œuvre", "żółć", "Łódź", "città", "perché", "naïve", + "coöperate", "café", "x86_64", "C++", "@user", "#tag", "https://opennlp.apache.org"; + }; + + /** The language whose policy and released expression are compared. */ + @State(Scope.Benchmark) + public static class LanguageState { + + @Param({"en", "es", "de", "pl"}) + String language; + + TokenizerCharacterPolicy policy; + + Pattern pattern; + + @Setup(Level.Trial) + public void setUp() { + policy = new Factory().getCharacterPolicy(language); + pattern = Pattern.compile(switch (language) { + case "es" -> "^[0-9a-záéíóúüýñA-ZÁÉÍÓÚÝÑ]+$"; + case "de" -> "^[A-Za-z0-9äéöüÄÉÖÜß]+$"; + case "pl" -> "^[A-Za-z0-9żźćńółęąśŻŹĆĄŚĘŁÓŃ]+$"; + default -> "^[A-Za-z0-9]+$"; + }); + for (String candidate : CANDIDATES) { + if (policy.test(candidate) != pattern.matcher(candidate).matches()) { + throw new IllegalStateException("policy and expression disagree on " + candidate); + } + } + } + } + + /** + * Checks every candidate with the character policy. + * + * @param state The language state. + * @param bh The sink for the results. + */ + @Benchmark + public void characterPolicy(LanguageState state, Blackhole bh) { + for (String candidate : CANDIDATES) { + bh.consume(state.policy.test(candidate)); + } + } + + /** + * Checks every candidate with the regular expression of earlier releases. + * + * @param state The language state. + * @param bh The sink for the results. + */ + @Benchmark + public void regularExpression(LanguageState state, Blackhole bh) { + for (String candidate : CANDIDATES) { + bh.consume(state.pattern.matcher(candidate).matches()); + } + } + + /** + * Quick local iteration only: {@code forks(0)} disables JVM fork isolation + * (unlike {@code mvn} with the {@code jmh} profile). + * Use the Maven-invoked configuration for publishable numbers. + */ + public static void main(String[] args) throws Exception { + Options opt = new OptionsBuilder() + .include(TokenizerCharacterPolicyBenchmark.class.getSimpleName()) + .forks(0) + .warmupIterations(3) + .measurementIterations(5) + .build(); + new Runner(opt).run(); + } +} diff --git a/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/tokenize/TokenizerMEBenchmark.java b/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/tokenize/TokenizerMEBenchmark.java index 9442483fa..a8a92c7bd 100644 --- a/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/tokenize/TokenizerMEBenchmark.java +++ b/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/tokenize/TokenizerMEBenchmark.java @@ -82,7 +82,7 @@ public class TokenizerMEBenchmark { p.put(Parameters.CUTOFF_PARAM, 0); model = TokenizerME.train(samples, TokenizerFactory.create( - null, "eng", null, true, null), p); + null, "eng", null, true), p); } } diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokSpanEventStream.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokSpanEventStream.java index b5e297aff..4758fadbb 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokSpanEventStream.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokSpanEventStream.java @@ -21,6 +21,7 @@ package opennlp.tools.tokenize; import java.util.ArrayList; import java.util.Iterator; import java.util.List; +import java.util.function.Predicate; import java.util.regex.Pattern; import org.slf4j.Logger; @@ -31,6 +32,7 @@ import opennlp.tools.ml.model.Event; import opennlp.tools.tokenize.lang.Factory; import opennlp.tools.util.AbstractEventStream; import opennlp.tools.util.ObjectStream; +import opennlp.tools.util.ParamChecks; import opennlp.tools.util.Span; /** @@ -45,30 +47,66 @@ public class TokSpanEventStream extends AbstractEventStream<TokenSample> { private final boolean skipAlphaNumerics; - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") - private final Pattern alphaNumeric; + private final Predicate<CharSequence> alphanumeric; /** * Initializes a new event stream based on the data stream using a {@link TokenContextGenerator}. * * @param tokenSamples The {@link ObjectStream data stream} for this event stream. * @param skipAlphaNumerics Whether alphanumerics are skipped, or not. - * @param alphaNumeric A custom alphanumeric {@link Pattern} or {@code null}. - * Default is: {@code "^[A-Za-z0-9]+$"}, provided by - * {@link Factory#DEFAULT_ALPHANUMERIC}. + * @param characterPolicy The {@link TokenizerCharacterPolicy} that decides which candidate + * tokens count as alphanumeric. Must not be {@code null}. * @param cg A {@link TokenContextGenerator} which should be used for the event stream {@code d}. + * @throws IllegalArgumentException Thrown if {@code characterPolicy} is {@code null}. */ - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") + public TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, + TokenizerCharacterPolicy characterPolicy, TokenContextGenerator cg) { + this(tokenSamples, skipAlphaNumerics, + (Predicate<CharSequence>) ParamChecks.requireNonNullArg(characterPolicy, "characterPolicy"), + cg); + } + + /** + * Initializes a new event stream that decides which candidate tokens count as alphanumeric + * with a regular expression. + * + * @param tokenSamples The {@link ObjectStream data stream} for this event stream. + * @param skipAlphaNumerics Whether alphanumerics are skipped, or not. + * @param alphaNumeric A custom alphanumeric {@link Pattern}, or {@code null} for + * {@link Factory#DEFAULT_CHARACTER_POLICY}. + * @param cg A {@link TokenContextGenerator} which should be used for the event stream {@code d}. + * @deprecated Use + * {@link #TokSpanEventStream(ObjectStream, boolean, TokenizerCharacterPolicy, TokenContextGenerator)}. + * This constructor will be removed in 4.0. + */ + @Deprecated(since = "3.0.0", forRemoval = true) + @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, TokenContextGenerator cg) { + this(tokenSamples, skipAlphaNumerics, alphaNumeric == null + ? Factory.DEFAULT_CHARACTER_POLICY : new TokenizerFactory.PatternTest(alphaNumeric), cg); + } + + /** + * Initializes a new event stream with the test that decides which candidate tokens count as + * alphanumeric. + * + * @param tokenSamples The {@link ObjectStream data stream} for this event stream. + * @param skipAlphaNumerics Whether alphanumerics are skipped, or not. + * @param alphanumeric The test for alphanumeric candidate tokens. Must not be {@code null}. + * @param cg A {@link TokenContextGenerator} which should be used for the event stream {@code d}. + */ + TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, + Predicate<CharSequence> alphanumeric, TokenContextGenerator cg) { super(tokenSamples); - this.alphaNumeric = alphaNumeric; + this.alphanumeric = alphanumeric; this.skipAlphaNumerics = skipAlphaNumerics; this.cg = cg; } /** - * Initializes a new event stream based on the data stream using a {@link TokenContextGenerator}. + * Initializes a new event stream based on the data stream using a {@link TokenContextGenerator} + * and {@link Factory#DEFAULT_CHARACTER_POLICY}. * * @param tokenSamples The {@link ObjectStream data stream} for this event stream. * @param skipAlphaNumerics Whether alphanumerics are skipped, or not. @@ -76,7 +114,7 @@ public class TokSpanEventStream extends AbstractEventStream<TokenSample> { */ public TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenContextGenerator cg) { - this(tokenSamples, skipAlphaNumerics, new Factory().getAlphanumeric(null), cg ); + this(tokenSamples, skipAlphaNumerics, Factory.DEFAULT_CHARACTER_POLICY, cg); } /** @@ -98,7 +136,6 @@ public class TokSpanEventStream extends AbstractEventStream<TokenSample> { * @return An {@link Iterator} for text {@link Event events} representing the {@code tokenSample}. */ @Override - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") protected Iterator<Event> createEvents(TokenSample tokenSample) { List<Event> events = new ArrayList<>(50); @@ -123,7 +160,7 @@ public class TokSpanEventStream extends AbstractEventStream<TokenSample> { //adjust cSpan to text offsets cSpan = new Span(cSpan.getStart() + start, cSpan.getEnd() + start); //should we skip this token - if (ctok.length() > 1 && (!skipAlphaNumerics || !alphaNumeric.matcher(ctok).matches())) { + if (ctok.length() > 1 && (!skipAlphaNumerics || !alphanumeric.test(ctok))) { //find offsets of annotated tokens inside of candidate tokens boolean foundTrainingTokens = false; diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerCharacterPolicy.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerCharacterPolicy.java index d2ff20b5a..37a4f3cef 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerCharacterPolicy.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerCharacterPolicy.java @@ -16,6 +16,8 @@ */ package opennlp.tools.tokenize; +import java.util.HexFormat; +import java.util.Objects; import java.util.function.Predicate; import opennlp.tools.util.ParamChecks; @@ -32,10 +34,20 @@ import opennlp.tools.util.normalizer.UnicodeWhitespace; * * <p>This predicate tests a complete candidate without finding token boundaries or changing * its text.</p> + * + * @since 3.0.0 */ public final class TokenizerCharacterPolicy implements Predicate<CharSequence> { private static final String CODE_POINT_FORMAT = "U+%04X"; + private static final String LETTERS_FIELD = "letters="; + private static final String DIGITS_FIELD = ";digits="; + private static final String MARKS_FIELD = ";marks="; + private static final char RANGE_SEPARATOR = ','; + private static final char RANGE_DASH = '-'; + private static final HexFormat UPPER_HEX = HexFormat.of().withUpperCase(); + private static final int MIN_HEX_DIGITS = 4; + private static final int BITS_PER_HEX_DIGIT = 4; private static final TokenizerCharacterPolicy ASCII = new TokenizerCharacterPolicy( CodePointSet.ofRange('A', 'Z').union(CodePointSet.ofRange('a', 'z')), @@ -86,6 +98,30 @@ public final class TokenizerCharacterPolicy implements Predicate<CharSequence> { return ASCII; } + /** + * Parses the text form written by {@link #toString()}: the three fields + * {@code letters=}, {@code ;digits=} and {@code ;marks=} in that order, each followed by + * zero or more comma separated hex code points ({@code 0041}) or inclusive ranges + * ({@code 0041-005A}). Hex digits may use either case; no prefix or whitespace is accepted. + * + * @param text The text form to read. + * @return The policy the text describes. + * @throws IllegalArgumentException Thrown if {@code text} is {@code null} or not of that + * form, or if the sets it describes are rejected by + * {@link #of(CodePointSet, CodePointSet, CodePointSet)}. + */ + public static TokenizerCharacterPolicy parse(String text) { + ParamChecks.requireNonNullArg(text, "text"); + final int digitsAt = text.indexOf(DIGITS_FIELD); + final int marksAt = digitsAt < 0 ? -1 : text.indexOf(MARKS_FIELD, digitsAt + DIGITS_FIELD.length()); + if (!text.startsWith(LETTERS_FIELD) || digitsAt < 0 || marksAt < 0) { + throw malformed(text, "expected the fields " + LETTERS_FIELD + DIGITS_FIELD + MARKS_FIELD); + } + return of(parseRanges(text, LETTERS_FIELD.length(), digitsAt), + parseRanges(text, digitsAt + DIGITS_FIELD.length(), marksAt), + parseRanges(text, marksAt + MARKS_FIELD.length(), text.length())); + } + /** * {@inheritDoc} * @@ -141,6 +177,132 @@ public final class TokenizerCharacterPolicy implements Predicate<CharSequence> { return marks; } + /** + * {@inheritDoc} + * + * <p>Two policies are equal if their letter, digit and mark sets are equal.</p> + */ + @Override + public boolean equals(Object o) { + return o instanceof TokenizerCharacterPolicy other && letters.equals(other.letters) + && digits.equals(other.digits) && marks.equals(other.marks); + } + + /** + * {@inheritDoc} + * + * <p>Consistent with {@link #equals(Object)}: computed from the letter, digit and mark sets.</p> + */ + @Override + public int hashCode() { + return Objects.hash(letters, digits, marks); + } + + /** + * {@inheritDoc} + * + * <p>Writes the text form read by {@link #parse(String)}, for example + * {@code letters=0041-005A,0061-007A;digits=0030-0039;marks=} for {@link #ascii()}: runs of + * adjacent code points become ranges, and each code point is written with at least four + * upper-case hex digits.</p> + */ + @Override + public String toString() { + final StringBuilder text = new StringBuilder(); + appendRanges(text.append(LETTERS_FIELD), letters); + appendRanges(text.append(DIGITS_FIELD), digits); + appendRanges(text.append(MARKS_FIELD), marks); + return text.toString(); + } + + /** Appends the members of {@code set} as comma separated code points and ranges. */ + private static void appendRanges(StringBuilder text, CodePointSet set) { + final int[] members = set.toArray(); + int first = 0; + while (first < members.length) { + int last = first; + while (last + 1 < members.length && members[last + 1] == members[last] + 1) { + last++; + } + if (first > 0) { + text.append(RANGE_SEPARATOR); + } + appendHex(text, members[first]); + if (last > first) { + appendHex(text.append(RANGE_DASH), members[last]); + } + first = last + 1; + } + } + + /** Appends {@code codePoint} as upper-case hex with at least {@link #MIN_HEX_DIGITS} digits. */ + private static void appendHex(StringBuilder text, int codePoint) { + final String hex = UPPER_HEX.toHexDigits(codePoint); + int start = 0; + while (start < hex.length() - MIN_HEX_DIGITS && hex.charAt(start) == '0') { + start++; + } + text.append(hex, start, hex.length()); + } + + /** Reads the comma separated code points and ranges in {@code text[from, to)}. */ + private static CodePointSet parseRanges(String text, int from, int to) { + CodePointSet members = CodePointSet.of(); + if (from == to) { + return members; + } + int offset = from; + while (true) { + int separator = text.indexOf(RANGE_SEPARATOR, offset); + if (separator < 0 || separator > to) { + separator = to; + } + members = members.union(parseRange(text, offset, separator)); + if (separator == to) { + return members; + } + offset = separator + 1; + } + } + + /** Reads one code point or one inclusive range in {@code text[from, to)}. */ + private static CodePointSet parseRange(String text, int from, int to) { + final int dash = text.indexOf(RANGE_DASH, from); + if (dash < 0 || dash >= to) { + return CodePointSet.of(parseHex(text, from, to)); + } + final int low = parseHex(text, from, dash); + final int high = parseHex(text, dash + 1, to); + if (low > high) { + throw malformed(text, "descending range '" + text.substring(from, to) + "'"); + } + return CodePointSet.ofRange(low, high); + } + + /** Reads the hex code point in {@code text[from, to)}. */ + private static int parseHex(String text, int from, int to) { + if (from >= to) { + throw malformed(text, "empty code point at offset " + from); + } + int value = 0; + for (int i = from; i < to; i++) { + final char c = text.charAt(i); + if (!HexFormat.isHexDigit(c)) { + throw malformed(text, "'" + text.substring(from, to) + "' is not a hex code point"); + } + value = value << BITS_PER_HEX_DIGIT | HexFormat.fromHexDigit(c); + if (value > Character.MAX_CODE_POINT) { + throw malformed(text, "'" + text.substring(from, to) + "' is beyond U+10FFFF"); + } + } + return value; + } + + /** {@return the exception for a text form that cannot be read} */ + private static IllegalArgumentException malformed(String text, String detail) { + return new IllegalArgumentException("Malformed character policy '" + text + "': " + detail); + } + /** Throws if {@code set} contains a surrogate code point or {@link UnicodeWhitespace} member. */ private static void validateSet(String name, CodePointSet set) { for (int codePoint = Character.MIN_SURROGATE; codePoint <= Character.MAX_SURROGATE; diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerFactory.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerFactory.java index eabc986cb..44d9378bc 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerFactory.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerFactory.java @@ -18,9 +18,12 @@ package opennlp.tools.tokenize; import java.util.Collections; +import java.util.List; import java.util.Map; import java.util.Set; +import java.util.function.Predicate; import java.util.regex.Pattern; +import java.util.regex.PatternSyntaxException; import opennlp.tools.commons.SuppressForbidden; import opennlp.tools.dictionary.Dictionary; @@ -39,20 +42,43 @@ public class TokenizerFactory extends BaseToolFactory { private String languageCode; private Dictionary abbreviationDictionary; private Boolean useAlphaNumericOptimization = false; - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") + private TokenizerCharacterPolicy characterPolicy; + @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") private Pattern alphaNumericPattern; private static final String ABBREVIATIONS_ENTRY_NAME = "abbreviations.dictionary"; private static final String USE_ALPHA_NUMERIC_OPTIMIZATION = "useAlphaNumericOptimization"; + private static final String CHARACTER_POLICY = "characterPolicy"; private static final String ALPHA_NUMERIC_PATTERN = "alphaNumericPattern"; + /** + * The language codes with a dedicated character policy, whose alphanumeric expressions releases + * before 3.0 wrote to the model manifest; {@code en} stands for the ASCII default. + */ + private static final List<String> LEGACY_LANGUAGES = + List.of("en", "pt", "fr", "nl", "de", "pl", "it", "es", "ca"); + /** * Instantiates a {@link TokenizerFactory} that provides the default implementation * of the resources. */ public TokenizerFactory() { } - + + /** + * Instantiates a {@link TokenizerFactory} with the {@link TokenizerCharacterPolicy} of its + * language. Use this constructor to programmatically create a factory. + * + * @param languageCode The ISO language code to be used for this factory. + * @param abbreviationDictionary The {@link Dictionary} which holds abbreviations. + * @param useAlphaNumericOptimization Whether alphanumerics are skipped, or not. + */ + public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, + boolean useAlphaNumericOptimization) { + this(languageCode, abbreviationDictionary, useAlphaNumericOptimization, + (TokenizerCharacterPolicy) null); + } + /** * Instantiates a {@link TokenizerFactory}. Use this constructor to * programmatically create a factory. @@ -60,31 +86,51 @@ public class TokenizerFactory extends BaseToolFactory { * @param languageCode The ISO language code to be used for this factory. * @param abbreviationDictionary The {@link Dictionary} which holds abbreviations. * @param useAlphaNumericOptimization Whether alphanumerics are skipped, or not. - * @param alphaNumericPattern {@code null} or a custom alphanumeric {@link Pattern} - * (default is: {@code "^[A-Za-z0-9]+$"}, provided by - * {@link Factory#DEFAULT_ALPHANUMERIC}. + * @param characterPolicy {@code null} or a custom {@link TokenizerCharacterPolicy} + * (default is the policy for {@code languageCode}, provided by + * {@link Factory#getCharacterPolicy(String)}). */ - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, - boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) { + boolean useAlphaNumericOptimization, + TokenizerCharacterPolicy characterPolicy) { this.init(languageCode, abbreviationDictionary, - useAlphaNumericOptimization, alphaNumericPattern); + useAlphaNumericOptimization, characterPolicy); } /** + * Instantiates a {@link TokenizerFactory} whose alphanumeric optimization tests candidate + * tokens with a regular expression. + * * @param languageCode The ISO language code to be used for this factory. * @param abbreviationDictionary The {@link Dictionary} which holds abbreviations. * @param useAlphaNumericOptimization Whether alphanumerics are skipped, or not. - * @param alphaNumericPattern {@code null} or a custom alphanumeric {@link Pattern} - * (default is: {@code "^[A-Za-z0-9]+$"}, provided by - * {@link Factory#DEFAULT_ALPHANUMERIC}. + * @param alphaNumericPattern {@code null} for the policy of {@code languageCode}, or a custom + * alphanumeric {@link Pattern}. + * @deprecated Use + * {@link #TokenizerFactory(String, Dictionary, boolean, TokenizerCharacterPolicy)}. This + * constructor will be removed in 4.0. + */ + @Deprecated(since = "3.0.0", forRemoval = true) + @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") + public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, + boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) { + this(languageCode, abbreviationDictionary, useAlphaNumericOptimization); + this.alphaNumericPattern = alphaNumericPattern; + } + + /** + * @param languageCode The ISO language code to be used for this factory. + * @param abbreviationDictionary The {@link Dictionary} which holds abbreviations. + * @param useAlphaNumericOptimization Whether alphanumerics are skipped, or not. + * @param characterPolicy {@code null} or a custom {@link TokenizerCharacterPolicy} + * (default is the policy for {@code languageCode}, provided by + * {@link Factory#getCharacterPolicy(String)}). */ - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") protected void init(String languageCode, Dictionary abbreviationDictionary, - boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) { + boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) { this.languageCode = languageCode; this.useAlphaNumericOptimization = useAlphaNumericOptimization; - this.alphaNumericPattern = alphaNumericPattern; + this.characterPolicy = characterPolicy; this.abbreviationDictionary = abbreviationDictionary; } @@ -100,6 +146,14 @@ public class TokenizerFactory extends BaseToolFactory { throw new InvalidFormatException("Abbreviations dictionary '" + abbreviationsEntry + "' has wrong type, needs to be of type Dictionary!"); } + + try { + if (this.characterPolicy == null && this.alphaNumericPattern == null) { + readManifest(); + } + } catch (IllegalArgumentException e) { + throw new InvalidFormatException(e.getMessage(), e); + } } @Override @@ -115,16 +169,21 @@ public class TokenizerFactory extends BaseToolFactory { } @Override - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") public Map<String, String> createManifestEntries() { Map<String, String> manifestEntries = super.createManifestEntries(); manifestEntries.put(USE_ALPHA_NUMERIC_OPTIMIZATION, Boolean.toString(isUseAlphaNumericOptimization())); - - // alphanumeric pattern is optional - if (getAlphaNumericPattern() != null) { - manifestEntries.put(ALPHA_NUMERIC_PATTERN, getAlphaNumericPattern().pattern()); + resolve(); + if (this.alphaNumericPattern != null) { + manifestEntries.put(ALPHA_NUMERIC_PATTERN, patternText(this.alphaNumericPattern)); + } else { + manifestEntries.put(CHARACTER_POLICY, this.characterPolicy.toString()); + // Releases before 3.0 read only the pattern; give them the one with the same characters. + final String legacy = legacyExpression(this.characterPolicy); + if (legacy != null) { + manifestEntries.put(ALPHA_NUMERIC_PATTERN, legacy); + } } return manifestEntries; @@ -138,29 +197,28 @@ public class TokenizerFactory extends BaseToolFactory { * @param abbreviationDictionary An optional {@link Dictionary} containing abbreviations, * or {@code null} if not present. * @param useAlphaNumericOptimization Whether the alphanumeric optimization is be enabled or not. - * @param alphaNumericPattern The {@link Pattern} the alphanumeric optimization should use, - * if enabled. + * @param characterPolicy The {@link TokenizerCharacterPolicy} the alphanumeric optimization + * should use, if enabled, or {@code null} for the language default. * * @return A valid {@link TokenizerFactory} instance. * * @throws InvalidFormatException Thrown if one of the input parameters doesn't comply the expected format. */ - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") public static TokenizerFactory create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, - Pattern alphaNumericPattern) + TokenizerCharacterPolicy characterPolicy) throws InvalidFormatException { if (subclassName == null) { // will create the default factory return new TokenizerFactory(languageCode, abbreviationDictionary, - useAlphaNumericOptimization, alphaNumericPattern); + useAlphaNumericOptimization, characterPolicy); } try { TokenizerFactory theFactory = ExtensionLoader.instantiateExtension( TokenizerFactory.class, subclassName); theFactory.init(languageCode, abbreviationDictionary, - useAlphaNumericOptimization, alphaNumericPattern); + useAlphaNumericOptimization, characterPolicy); return theFactory; } catch (Exception e) { String msg = "Could not instantiate the " + subclassName @@ -170,24 +228,101 @@ public class TokenizerFactory extends BaseToolFactory { } /** - * @return Retrieves the (user-)specified alphanumeric {@link Pattern} or a default. + * Factory method the framework uses instantiate a new {@link TokenizerFactory} with the + * {@link TokenizerCharacterPolicy} of its language. + * + * @param subclassName The name of the class implementing the {@link TokenizerFactory}. + * @param languageCode The ISO language code the {@link Tokenizer} should use. + * @param abbreviationDictionary An optional {@link Dictionary} containing abbreviations, + * or {@code null} if not present. + * @param useAlphaNumericOptimization Whether the alphanumeric optimization is be enabled or not. + * + * @return A valid {@link TokenizerFactory} instance. + * + * @throws InvalidFormatException Thrown if one of the input parameters doesn't comply the expected format. */ - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") + public static TokenizerFactory create(String subclassName, String languageCode, + Dictionary abbreviationDictionary, + boolean useAlphaNumericOptimization) + throws InvalidFormatException { + return create(subclassName, languageCode, abbreviationDictionary, + useAlphaNumericOptimization, (TokenizerCharacterPolicy) null); + } + + /** + * Factory method the framework uses instantiate a new {@link TokenizerFactory} whose + * alphanumeric optimization tests candidate tokens with a regular expression. + * + * @param subclassName The name of the class implementing the {@link TokenizerFactory}. + * @param languageCode The ISO language code the {@link Tokenizer} should use. + * @param abbreviationDictionary An optional {@link Dictionary} containing abbreviations, + * or {@code null} if not present. + * @param useAlphaNumericOptimization Whether the alphanumeric optimization is be enabled or not. + * @param alphaNumericPattern {@code null} for the policy of {@code languageCode}, or a custom + * alphanumeric {@link Pattern}. + * + * @return A valid {@link TokenizerFactory} instance. + * + * @throws InvalidFormatException Thrown if one of the input parameters doesn't comply the expected format. + * @deprecated Use + * {@link #create(String, String, Dictionary, boolean, TokenizerCharacterPolicy)}. This + * method will be removed in 4.0. + */ + @Deprecated(since = "3.0.0", forRemoval = true) + @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") + public static TokenizerFactory create(String subclassName, String languageCode, + Dictionary abbreviationDictionary, + boolean useAlphaNumericOptimization, + Pattern alphaNumericPattern) + throws InvalidFormatException { + TokenizerFactory factory = create(subclassName, languageCode, abbreviationDictionary, + useAlphaNumericOptimization); + factory.alphaNumericPattern = alphaNumericPattern; + return factory; + } + + /** + * @return Retrieves the (user-)specified {@link TokenizerCharacterPolicy}, the one read from + * the model manifest, or the default for the language in use; {@code null} if this + * factory tests candidate tokens with a custom alphanumeric {@link Pattern}. + * @throws IllegalArgumentException Thrown if the manifest has a character policy or an + * alphanumeric pattern that cannot be read. + */ + public TokenizerCharacterPolicy getCharacterPolicy() { + resolve(); + return this.characterPolicy; + } + + /** + * @return Retrieves the custom alphanumeric {@link Pattern} of this factory, or else the pattern + * of earlier releases that accepts the same characters as its + * {@link TokenizerCharacterPolicy}; {@code null} for a custom policy that no such + * pattern matches. + * @throws IllegalArgumentException Thrown if the manifest has a character policy or an + * alphanumeric pattern that cannot be read. + * @deprecated Use {@link #getCharacterPolicy()}. This method will be removed in 4.0. + */ + @Deprecated(since = "3.0.0", forRemoval = true) + @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public Pattern getAlphaNumericPattern() { - if (this.alphaNumericPattern == null) { - if (this.artifactProvider != null) { - String prop = this.artifactProvider.getManifestProperty(ALPHA_NUMERIC_PATTERN); - if (prop != null) { - this.alphaNumericPattern = Pattern.compile(prop); - } - } - // could not load from manifest, will get from language dependent factory - if (this.alphaNumericPattern == null) { - Factory f = new Factory(); - this.alphaNumericPattern = f.getAlphanumeric(languageCode); - } + resolve(); + if (this.alphaNumericPattern != null) { + return this.alphaNumericPattern; } - return this.alphaNumericPattern; + final String legacy = legacyExpression(this.characterPolicy); + return legacy == null ? null : Pattern.compile(legacy); + } + + /** + * {@return the test the alphanumeric optimization applies to a candidate token} + * + * @throws IllegalArgumentException Thrown if the manifest has a character policy or an + * alphanumeric pattern that cannot be read. + */ + Predicate<CharSequence> alphanumericTest() { + resolve(); + return this.alphaNumericPattern != null + ? new PatternTest(this.alphaNumericPattern) : this.characterPolicy; } /** @@ -235,4 +370,130 @@ public class TokenizerFactory extends BaseToolFactory { } return f.createTokenContextGenerator(getLanguageCode(), abbs); } + + /** + * Sets the character policy or the alphanumeric pattern from the model manifest if neither is + * set yet, and falls back to the policy of the language. + * + * @throws IllegalArgumentException Thrown if the manifest has a character policy or an + * alphanumeric pattern that cannot be read. + */ + private void resolve() { + if (this.characterPolicy == null && this.alphaNumericPattern == null) { + if (this.artifactProvider != null) { + readManifest(); + } + if (this.characterPolicy == null && this.alphaNumericPattern == null) { + this.characterPolicy = new Factory().getCharacterPolicy(getLanguageCode()); + } + } + } + + /** + * Reads the character policy, or else the alphanumeric pattern, from the model manifest. A + * pattern that releases before 3.0 wrote for a language is replaced by the policy that accepts + * the same characters; any other pattern is kept and evaluated as a regular expression. + * + * @throws IllegalArgumentException Thrown if the character policy text cannot be read, or if + * the alphanumeric pattern is not a valid regular expression. + */ + @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") + private void readManifest() { + String text = this.artifactProvider.getManifestProperty(CHARACTER_POLICY); + if (text != null) { + try { + this.characterPolicy = TokenizerCharacterPolicy.parse(text); + return; + } catch (IllegalArgumentException e) { + throw new IllegalArgumentException(describeModel() + " has an unreadable " + + CHARACTER_POLICY + ": " + e.getMessage(), e); + } + } + String pattern = this.artifactProvider.getManifestProperty(ALPHA_NUMERIC_PATTERN); + if (pattern == null) { + return; + } + this.characterPolicy = legacyPolicy(pattern); + if (this.characterPolicy == null) { + try { + this.alphaNumericPattern = Pattern.compile(pattern); + } catch (PatternSyntaxException e) { + throw new IllegalArgumentException(describeModel() + " has an invalid " + + ALPHA_NUMERIC_PATTERN + " '" + pattern + "'", e); + } + } + } + + /** + * {@return the policy of the language whose expression of an earlier release is + * {@code expression}, or {@code null} if there is none} + * + * @param expression An alphanumeric expression from a model manifest. + */ + @SuppressWarnings("removal") + @SuppressForbidden("compares with the alphanumeric patterns of earlier releases") + private static TokenizerCharacterPolicy legacyPolicy(String expression) { + final Factory factory = new Factory(); + for (String language : LEGACY_LANGUAGES) { + if (factory.getAlphanumeric(language).pattern().equals(expression)) { + return factory.getCharacterPolicy(language); + } + } + return null; + } + + /** + * {@return the expression of an earlier release that accepts the same characters as + * {@code policy}, or {@code null} if there is none} + * + * @param policy A character policy. + */ + @SuppressWarnings("removal") + @SuppressForbidden("compares with the alphanumeric patterns of earlier releases") + private static String legacyExpression(TokenizerCharacterPolicy policy) { + final Factory factory = new Factory(); + for (String language : LEGACY_LANGUAGES) { + if (factory.getCharacterPolicy(language).equals(policy)) { + return factory.getAlphanumeric(language).pattern(); + } + } + return null; + } + + /** + * {@return the text of {@code pattern}} + * + * @param pattern An alphanumeric pattern. + */ + @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") + private static String patternText(Pattern pattern) { + return pattern.pattern(); + } + + /** + * Tests a candidate token against a whole-match regular expression, for factories and event + * streams configured with a custom alphanumeric {@link Pattern}. + */ + @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") + static final class PatternTest implements Predicate<CharSequence> { + + private final Pattern pattern; + + /** + * @param pattern The alphanumeric pattern. + */ + PatternTest(Pattern pattern) { + this.pattern = pattern; + } + + @Override + public boolean test(CharSequence candidate) { + return pattern.matcher(candidate).matches(); + } + } + + /** {@return a description of the loaded model for error messages} */ + private String describeModel() { + return "The tokenizer model for language '" + this.artifactProvider.getLanguage() + "'"; + } } diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerME.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerME.java index 256fea895..8aa9fe8fc 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerME.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/TokenizerME.java @@ -22,9 +22,8 @@ import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; -import java.util.regex.Pattern; +import java.util.function.Predicate; -import opennlp.tools.commons.SuppressForbidden; import opennlp.tools.commons.ThreadSafe; import opennlp.tools.dictionary.Dictionary; import opennlp.tools.ml.ArrayMath; @@ -103,8 +102,10 @@ public class TokenizerME extends AbstractTokenizer implements Probabilistic { */ public static final String NO_SPLIT = "F"; - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") - private final Pattern alphanumeric; + /* + * The test that decides which tokens the alphanumeric optimization accepts. + */ + private final Predicate<CharSequence> alphanumeric; /* * The maximum entropy model to use to evaluate contexts. @@ -167,7 +168,7 @@ public class TokenizerME extends AbstractTokenizer implements Probabilistic { this.abbDict = abbDict; TokenizerFactory factory = model.getFactory(); this.cg = factory.getContextGenerator(); - this.alphanumeric = factory.getAlphaNumericPattern(); + this.alphanumeric = factory.alphanumericTest(); this.useAlphaNumericOptimization = factory.isUseAlphaNumericOptimization(); } @@ -202,7 +203,6 @@ public class TokenizerME extends AbstractTokenizer implements Probabilistic { * @return A {@link Span} array containing individual tokens as elements. */ @Override - @SuppressForbidden("user-supplied alphanumeric pattern stored in the tokenizer model") public Span[] tokenizePos(String d) { Span[] tokens = (keepNewLines ? WHITESPACE_KEEPING_NEW_LINES : WHITESPACE).tokenizePos(d); List<Span> localTokens = new ArrayList<>(); @@ -213,7 +213,7 @@ public class TokenizerME extends AbstractTokenizer implements Probabilistic { if (tok.length() < 2) { localTokens.add(s); localProbs.add(1d); - } else if (useAlphaNumericOptimization() && alphanumeric.matcher(tok).matches()) { + } else if (useAlphaNumericOptimization() && alphanumeric.test(tok)) { localTokens.add(s); localProbs.add(1d); } else { @@ -283,7 +283,7 @@ public class TokenizerME extends AbstractTokenizer implements Probabilistic { ObjectStream<Event> eventStream = new TokSpanEventStream(samples, factory.isUseAlphaNumericOptimization(), - factory.getAlphaNumericPattern(), factory.getContextGenerator()); + factory.alphanumericTest(), factory.getContextGenerator()); EventTrainer<TrainingParameters> trainer = TrainerFactory.getEventTrainer( mlParams, manifestInfoEntries); diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lang/Factory.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lang/Factory.java index 99c53d8f3..1faf59ee3 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lang/Factory.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lang/Factory.java @@ -17,64 +17,94 @@ package opennlp.tools.tokenize.lang; +import java.util.Map; import java.util.Set; import java.util.regex.Pattern; import opennlp.tools.commons.SuppressForbidden; import opennlp.tools.tokenize.DefaultTokenContextGenerator; import opennlp.tools.tokenize.TokenContextGenerator; +import opennlp.tools.tokenize.TokenizerCharacterPolicy; +import opennlp.tools.util.normalizer.CodePointSet; +/** + * Provides the language dependent tokenizer resources: the {@link TokenizerCharacterPolicy} + * that decides which tokens the alphanumeric optimization accepts without further splitting, + * and the {@link TokenContextGenerator}. + */ public class Factory { - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - public static final Pattern DEFAULT_ALPHANUMERIC = compile("^[A-Za-z0-9]+$"); + /** + * The policy for languages without a dedicated one: ASCII letters and digits, no marks. + */ + public static final TokenizerCharacterPolicy DEFAULT_CHARACTER_POLICY = + TokenizerCharacterPolicy.ascii(); + + private static final String ASCII_EXPRESSION = "^[A-Za-z0-9]+$"; + + /** + * The alphanumeric pattern of earlier releases for languages without a dedicated one. + * + * @deprecated Use {@link #DEFAULT_CHARACTER_POLICY}, which accepts the same characters. This + * constant will be removed in 4.0. + */ + @Deprecated(since = "3.0.0", forRemoval = true) + @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API") + public static final Pattern DEFAULT_ALPHANUMERIC = compile(ASCII_EXPRESSION); + + private static final CodePointSet ASCII_LETTERS = + CodePointSet.ofRange('A', 'Z').union(CodePointSet.ofRange('a', 'z')); + private static final CodePointSet ASCII_DIGITS = CodePointSet.ofRange('0', '9'); + private static final CodePointSet NO_MARKS = CodePointSet.of(); - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - private static final Pattern PORTUGUESE = compile("^[0-9a-záãâàéêíóõôúüçA-ZÁÃÂÀÉÊÍÓÕÔÚÜÇ]+$"); - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - private static final Pattern FRENCH = compile("^[a-zA-Z0-9àâäèéêëîïôœùûüÿçÀÂÄÈÉÊËÎÏÔŒÙÛÜŸÇ]+$"); + private static final TokenizerCharacterPolicy PORTUGUESE = latinPolicy("áãâàéêíóõôúüçÁÃÂÀÉÊÍÓÕÔÚÜÇ"); + private static final TokenizerCharacterPolicy FRENCH = latinPolicy("àâäèéêëîïôœùûüÿçÀÂÄÈÉÊËÎÏÔŒÙÛÜŸÇ"); // From: https://www.sttmedia.com/characterfrequency-dutch - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - private static final Pattern DUTCH = compile("^[A-Za-z0-9äöüëèéïijÄÖÜËÉÈÏIJ]+$"); + private static final TokenizerCharacterPolicy DUTCH = latinPolicy("äöüëèéïijÄÖÜËÉÈÏIJ"); // Note: The extra é and É are included to cover German "Lehnwörter" such as "Café" - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - private static final Pattern GERMAN = compile("^[A-Za-z0-9äéöüÄÉÖÜß]+$"); + private static final TokenizerCharacterPolicy GERMAN = latinPolicy("äéöüÄÉÖÜß"); // From: https://en.wikipedia.org/wiki/Polish_alphabet // https://pl.wikipedia.org/wiki/Alfabet_polski - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - private static final Pattern POLISH = compile("^[A-Za-z0-9żźćńółęąśŻŹĆĄŚĘŁÓŃ]+$"); + private static final TokenizerCharacterPolicy POLISH = latinPolicy("żźćńółęąśŻŹĆĄŚĘŁÓŃ"); // From: https://it.wikipedia.org/wiki/Alfabeto_italiano - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - private static final Pattern ITALIAN = compile("^[0-9a-zàèéìîíòóùüA-ZÀÈÉÌÎÍÒÓÙÜ]+$"); + private static final TokenizerCharacterPolicy ITALIAN = latinPolicy("àèéìîíòóùüÀÈÉÌÎÍÒÓÙÜ"); // From: https://en.wikiversity.org/wiki/Alphabet/Spanish_alphabet & // https://en.wikipedia.org/wiki/Spanish_orthography#Alphabet_in_Spanish & // https://www.fundeu.es/consulta/tilde-en-la-y-y-griega-o-ye-24786/ - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - private static final Pattern SPANISH = compile("^[0-9a-záéíóúüýñA-ZÁÉÍÓÚÝÑ]+$"); + private static final TokenizerCharacterPolicy SPANISH = latinPolicy("áéíóúüýñÁÉÍÓÚÝÑ"); // From: https://en.wikipedia.org/wiki/Catalan_orthography#Spelling_patterns - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - private static final Pattern CATALAN = compile("^[0-9a-zàèéíïòóúüçA-ZÀÈÉÍÏÒÓÚÜÇ]+$"); - - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - private static Pattern compile(String regex) { - return Pattern.compile(regex); - } + private static final TokenizerCharacterPolicy CATALAN = latinPolicy("àèéíïòóúüçÀÈÉÍÏÒÓÚÜÇ"); + + /** + * The alphanumeric expressions of releases before 3.0, keyed by the policy that accepts the same + * characters. + */ + private static final Map<TokenizerCharacterPolicy, String> LEGACY_EXPRESSIONS = Map.of( + DEFAULT_CHARACTER_POLICY, ASCII_EXPRESSION, + PORTUGUESE, "^[0-9a-záãâàéêíóõôúüçA-ZÁÃÂÀÉÊÍÓÕÔÚÜÇ]+$", + FRENCH, "^[a-zA-Z0-9àâäèéêëîïôœùûüÿçÀÂÄÈÉÊËÎÏÔŒÙÛÜŸÇ]+$", + DUTCH, "^[A-Za-z0-9äöüëèéïijÄÖÜËÉÈÏIJ]+$", + GERMAN, "^[A-Za-z0-9äéöüÄÉÖÜß]+$", + POLISH, "^[A-Za-z0-9żźćńółęąśŻŹĆĄŚĘŁÓŃ]+$", + ITALIAN, "^[0-9a-zàèéìîíòóùüA-ZÀÈÉÌÎÍÒÓÙÜ]+$", + SPANISH, "^[0-9a-záéíóúüýñA-ZÁÉÍÓÚÝÑ]+$", + CATALAN, "^[0-9a-zàèéíïòóúüçA-ZÀÈÉÍÏÒÓÚÜÇ]+$"); /** - * Gets the alphanumeric pattern for a language. + * Gets the character policy for a language. * - * @param languageCode The ISO_639-1 code. If {@code null}, or unknown, the - * {@link #DEFAULT_ALPHANUMERIC} pattern will be returned. - * @return The alphanumeric {@link Pattern} for the language, or the default pattern. + * @param languageCode An ISO 639-1 code, or an ISO 639-2 code in its terminology or + * bibliographic form ({@code deu} or {@code ger}). If {@code null}, or unknown, the + * {@link #DEFAULT_CHARACTER_POLICY} will be returned. + * @return The {@link TokenizerCharacterPolicy} for the language, or the default policy. */ - @SuppressForbidden("default alphanumeric patterns of the tokenizer model") - public Pattern getAlphanumeric(String languageCode) { + public TokenizerCharacterPolicy getCharacterPolicy(String languageCode) { // For reference: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes if ("es".equals(languageCode) || "spa".equals(languageCode)) { return SPANISH; @@ -101,12 +131,38 @@ public class Factory { return DUTCH; } - return DEFAULT_ALPHANUMERIC; + return DEFAULT_CHARACTER_POLICY; + } + + /** + * Gets the alphanumeric pattern that releases before 3.0 used for a language. + * + * @param languageCode An ISO 639-1 code, or an ISO 639-2 code in its terminology or + * bibliographic form. If {@code null}, or unknown, the + * {@link #DEFAULT_ALPHANUMERIC} pattern will be returned. + * @return The {@link Pattern} that accepts the same characters as + * {@link #getCharacterPolicy(String)} for the language. + * @deprecated Use {@link #getCharacterPolicy(String)}. This method will be removed in 4.0. + */ + @Deprecated(since = "3.0.0", forRemoval = true) + @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API") + public Pattern getAlphanumeric(String languageCode) { + return compile(LEGACY_EXPRESSIONS.get(getCharacterPolicy(languageCode))); + } + + /** + * {@return the compiled alphanumeric expression of an earlier release} + * + * @param expression The expression. + */ + @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API") + private static Pattern compile(String expression) { + return Pattern.compile(expression); } /** * Initializes a customized {@link TokenContextGenerator} via a set of {@code abbreviations}. - * + * * @param languageCode The ISO_639-1 code to be used. * @param abbreviations The abbreviations to be used for new instance. */ @@ -114,4 +170,19 @@ public class Factory { return new DefaultTokenContextGenerator(abbreviations); } + /** + * Builds a policy of the ASCII letters plus {@code extraLetters}, the ASCII digits, and no marks. + * + * @param extraLetters The letters of a Latin script language beyond ASCII. + * @return The policy. + */ + private static TokenizerCharacterPolicy latinPolicy(String extraLetters) { + CodePointSet letters = ASCII_LETTERS; + for (int offset = 0; offset < extraLetters.length();) { + final int codePoint = extraLetters.codePointAt(offset); + letters = letters.union(CodePointSet.of(codePoint)); + offset += Character.charCount(codePoint); + } + return TokenizerCharacterPolicy.of(letters, ASCII_DIGITS, NO_MARKS); + } } diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/ThreadSafetyBenchmarkIT.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/ThreadSafetyBenchmarkIT.java index 1077204e2..72d78ab36 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/ThreadSafetyBenchmarkIT.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/ThreadSafetyBenchmarkIT.java @@ -386,7 +386,7 @@ public class ThreadSafetyBenchmarkIT { TrainingParameters p = new TrainingParameters(); p.put(Parameters.ITERATIONS_PARAM, 100); p.put(Parameters.CUTOFF_PARAM, 0); - return TokenizerME.train(samples, TokenizerFactory.create(null, "eng", null, true, null), p); + return TokenizerME.train(samples, TokenizerFactory.create(null, "eng", null, true), p); } /** Trains a sentence model from bundled {@code Sentences.txt}. */ diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/DummyTokenizerFactory.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/DummyTokenizerFactory.java index dab0577ba..773d18743 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/DummyTokenizerFactory.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/DummyTokenizerFactory.java @@ -22,7 +22,6 @@ import java.io.InputStream; import java.io.OutputStream; import java.util.Map; import java.util.Set; -import java.util.regex.Pattern; import opennlp.tools.dictionary.Dictionary; import opennlp.tools.util.model.ArtifactSerializer; @@ -37,16 +36,16 @@ public class DummyTokenizerFactory extends TokenizerFactory { public DummyTokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, - Pattern alphaNumericPattern) { + TokenizerCharacterPolicy characterPolicy) { super(languageCode, abbreviationDictionary, useAlphaNumericOptimization, - alphaNumericPattern); + characterPolicy); } @Override protected void init(String languageCode, Dictionary abbreviationDictionary, - boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) { + boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) { super.init(languageCode, abbreviationDictionary, useAlphaNumericOptimization, - alphaNumericPattern); + characterPolicy); this.dict = new DummyDictionary(abbreviationDictionary); } diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/LegacyAlphaNumericPatterns.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/LegacyAlphaNumericPatterns.java new file mode 100644 index 000000000..cad3dd093 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/LegacyAlphaNumericPatterns.java @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.tokenize; + +import java.util.Map; + +/** + * The alphanumeric regular expressions that OpenNLP wrote into tokenizer model manifests up + * to 2.x, keyed by the language code of the policy that replaces each one. Tests use them as + * the oracle for the policies and as legacy manifest input. + */ +public final class LegacyAlphaNumericPatterns { + + public static final String DEFAULT = "^[A-Za-z0-9]+$"; + public static final String PORTUGUESE = "^[0-9a-záãâàéêíóõôúüçA-ZÁÃÂÀÉÊÍÓÕÔÚÜÇ]+$"; + public static final String FRENCH = "^[a-zA-Z0-9àâäèéêëîïôœùûüÿçÀÂÄÈÉÊËÎÏÔŒÙÛÜŸÇ]+$"; + public static final String DUTCH = "^[A-Za-z0-9äöüëèéïijÄÖÜËÉÈÏIJ]+$"; + public static final String GERMAN = "^[A-Za-z0-9äéöüÄÉÖÜß]+$"; + public static final String POLISH = "^[A-Za-z0-9żźćńółęąśŻŹĆĄŚĘŁÓŃ]+$"; + public static final String ITALIAN = "^[0-9a-zàèéìîíòóùüA-ZÀÈÉÌÎÍÒÓÙÜ]+$"; + public static final String SPANISH = "^[0-9a-záéíóúüýñA-ZÁÉÍÓÚÝÑ]+$"; + public static final String CATALAN = "^[0-9a-zàèéíïòóúüçA-ZÀÈÉÍÏÒÓÚÜÇ]+$"; + + /** + * Each language code with the regular expression its policy replaces. + */ + public static final Map<String, String> BY_LANGUAGE = Map.of( + "en", DEFAULT, + "pt", PORTUGUESE, + "fr", FRENCH, + "nl", DUTCH, + "de", GERMAN, + "pl", POLISH, + "it", ITALIAN, + "es", SPANISH, + "ca", CATALAN); + + private LegacyAlphaNumericPatterns() { + } +} diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokSpanEventStreamTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokSpanEventStreamTest.java index c9dd6ea90..2442a259b 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokSpanEventStreamTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokSpanEventStreamTest.java @@ -18,19 +18,32 @@ package opennlp.tools.tokenize; import java.io.IOException; +import java.util.ArrayList; +import java.util.List; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; import opennlp.tools.ml.model.Event; +import opennlp.tools.tokenize.lang.Factory; import opennlp.tools.util.ObjectStream; import opennlp.tools.util.ObjectStreamUtils; +import opennlp.tools.util.normalizer.CodePointSet; /** * Tests for the {@link TokSpanEventStream} class. */ public class TokSpanEventStreamTest { + private static final String SPLIT_SAMPLE = "ab<SPLIT>cd"; + private static final String NON_ASCII_SPLIT_SAMPLE = "\u00E4b<SPLIT>cd"; + private static final List<String> SPLIT_SAMPLE_OUTCOMES = + List.of(TokenizerME.NO_SPLIT, TokenizerME.SPLIT, TokenizerME.NO_SPLIT); + private static final TokenizerCharacterPolicy X_ONLY = TokenizerCharacterPolicy.of( + CodePointSet.of('x'), CodePointSet.of(), CodePointSet.of()); + /** * Tests the event stream for correctly generated outcomes. */ @@ -54,4 +67,105 @@ public class TokSpanEventStreamTest { Assertions.assertNull(eventStream.read()); } } + + @Test + void testPolicyConstructorSkipsCandidatesThePolicyAccepts() throws IOException { + Assertions.assertEquals(List.of(), outcomes(SPLIT_SAMPLE, true, TokenizerCharacterPolicy.ascii())); + } + + @Test + void testPolicyConstructorKeepsCandidatesThePolicyRejects() throws IOException { + Assertions.assertEquals(SPLIT_SAMPLE_OUTCOMES, outcomes(SPLIT_SAMPLE, true, X_ONLY)); + } + + @Test + void testPolicyIsIgnoredWhenAlphanumericsAreNotSkipped() throws IOException { + Assertions.assertEquals(SPLIT_SAMPLE_OUTCOMES, + outcomes(SPLIT_SAMPLE, false, TokenizerCharacterPolicy.ascii())); + } + + @ParameterizedTest + @CsvSource({ + SPLIT_SAMPLE + ", 0", + NON_ASCII_SPLIT_SAMPLE + ", 3"}) + void testContextGeneratorConstructorUsesTheDefaultPolicy(String sample, int expectedEvents) + throws IOException { + ObjectStream<TokenSample> samples = new TokenSampleStream( + ObjectStreamUtils.createObjectStream(sample)); + + try (ObjectStream<Event> eventStream = new TokSpanEventStream(samples, true, + new DefaultTokenContextGenerator())) { + Assertions.assertEquals(expectedEvents, drain(eventStream).size()); + } + } + + @Test + void testDefaultPolicyIsTheFactoryDefault() throws IOException { + Assertions.assertEquals(outcomes(NON_ASCII_SPLIT_SAMPLE, true, Factory.DEFAULT_CHARACTER_POLICY), + outcomes(NON_ASCII_SPLIT_SAMPLE, true, TokenizerCharacterPolicy.ascii())); + } + + @Test + void testNullPolicyIsRejected() { + ObjectStream<TokenSample> samples = new TokenSampleStream( + ObjectStreamUtils.createObjectStream(SPLIT_SAMPLE)); + + IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class, + () -> new TokSpanEventStream(samples, true, (TokenizerCharacterPolicy) null, + new DefaultTokenContextGenerator())); + Assertions.assertEquals("characterPolicy must not be null", e.getMessage()); + } + + /** + * The deprecated constructor with the ASCII pattern of earlier releases produces the events of + * the ASCII policy. + */ + @Test + @SuppressWarnings("removal") + void testDeprecatedPatternConstructorMatchesTheAsciiPolicy() throws IOException { + ObjectStream<TokenSample> patternSamples = new TokenSampleStream( + ObjectStreamUtils.createObjectStream(SPLIT_SAMPLE, NON_ASCII_SPLIT_SAMPLE)); + ObjectStream<TokenSample> policySamples = new TokenSampleStream( + ObjectStreamUtils.createObjectStream(SPLIT_SAMPLE, NON_ASCII_SPLIT_SAMPLE)); + try (ObjectStream<Event> byPattern = new TokSpanEventStream(patternSamples, true, + java.util.regex.Pattern.compile("^[A-Za-z0-9]+$"), new DefaultTokenContextGenerator()); + ObjectStream<Event> byPolicy = new TokSpanEventStream(policySamples, true, + TokenizerCharacterPolicy.ascii(), new DefaultTokenContextGenerator())) { + int events = 0; + Event expected; + while ((expected = byPolicy.read()) != null) { + events++; + Event actual = byPattern.read(); + Assertions.assertNotNull(actual); + Assertions.assertEquals(expected.getOutcome(), actual.getOutcome()); + Assertions.assertArrayEquals(expected.getContext(), actual.getContext()); + } + Assertions.assertNull(byPattern.read()); + Assertions.assertTrue(events > 0, "the non-ASCII sample must produce events"); + } + } + + private static List<String> outcomes(String sample, boolean skipAlphaNumerics, + TokenizerCharacterPolicy policy) throws IOException { + ObjectStream<TokenSample> samples = new TokenSampleStream( + ObjectStreamUtils.createObjectStream(sample)); + + try (ObjectStream<Event> eventStream = new TokSpanEventStream(samples, skipAlphaNumerics, + policy, new DefaultTokenContextGenerator())) { + List<String> outcomes = new ArrayList<>(); + for (Event event : drain(eventStream)) { + outcomes.add(event.getOutcome()); + } + return outcomes; + } + } + + private static List<Event> drain(ObjectStream<Event> eventStream) throws IOException { + List<Event> events = new ArrayList<>(); + Event event; + while ((event = eventStream.read()) != null) { + events.add(event); + } + return events; + } } diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerCharacterPolicyTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerCharacterPolicyTest.java index 853a35860..ff225b5d1 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerCharacterPolicyTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerCharacterPolicyTest.java @@ -24,6 +24,7 @@ import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.Arguments; import org.junit.jupiter.params.provider.CsvSource; import org.junit.jupiter.params.provider.MethodSource; +import org.junit.jupiter.params.provider.NullSource; import org.junit.jupiter.params.provider.ValueSource; import opennlp.tools.util.normalizer.CodePointSet; @@ -32,6 +33,7 @@ import opennlp.tools.util.normalizer.UnicodeWhitespace; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotEquals; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -41,6 +43,7 @@ public class TokenizerCharacterPolicyTest { private static final int MUSICAL_MARK = 0x1D165; private static final int MATHEMATICAL_DIGIT = 0x1D7D8; private static final int ACUTE_ACCENT = 0x0301; + private static final String ASCII_TEXT = "letters=0041-005A,0061-007A;digits=0030-0039;marks="; private static final CodePointSet A = CodePointSet.of('a'); private static final CodePointSet SEVEN = CodePointSet.of('7'); @@ -246,6 +249,136 @@ public class TokenizerCharacterPolicyTest { } } + @Test + void testAsciiPresetHasAFixedTextForm() { + assertEquals(ASCII_TEXT, TokenizerCharacterPolicy.ascii().toString()); + } + + @Test + void testTextFormWritesSupplementaryCodePointsWithFiveDigits() { + TokenizerCharacterPolicy policy = TokenizerCharacterPolicy.of( + CodePointSet.of(DESERET_LETTER), EMPTY, CodePointSet.of(MUSICAL_MARK)); + + assertEquals("letters=10400;digits=;marks=1D165", policy.toString()); + } + + @Test + void testTextFormJoinsAdjacentCodePointsIntoRanges() { + TokenizerCharacterPolicy policy = TokenizerCharacterPolicy.of( + CodePointSet.of('a', 'b', 'c', 'e', 'g', 'h'), CodePointSet.of('7'), EMPTY); + + assertEquals("letters=0061-0063,0065,0067-0068;digits=0037;marks=", policy.toString()); + } + + /** + * Supplies policies with different set shapes. + * + * @return Policies to write and read back. + */ + static Stream<TokenizerCharacterPolicy> roundTripPolicies() { + return Stream.of( + TokenizerCharacterPolicy.ascii(), + GRAMMAR, + TokenizerCharacterPolicy.of(A, EMPTY, EMPTY), + TokenizerCharacterPolicy.of(EMPTY, SEVEN, EMPTY), + TokenizerCharacterPolicy.of(CodePointSet.of(DESERET_LETTER), + CodePointSet.of(MATHEMATICAL_DIGIT), CodePointSet.of(MUSICAL_MARK)), + TokenizerCharacterPolicy.of(CodePointSet.ofRange(0x21, 0x2F).union(CodePointSet.of('a')), + CodePointSet.ofRange(Character.MAX_SURROGATE + 1, Character.MAX_SURROGATE + 3), + CodePointSet.of(Character.MAX_CODE_POINT))); + } + + @ParameterizedTest + @MethodSource("roundTripPolicies") + void testParseReadsTheTextFormBack(TokenizerCharacterPolicy policy) { + TokenizerCharacterPolicy parsed = TokenizerCharacterPolicy.parse(policy.toString()); + + assertEquals(policy, parsed); + assertEquals(policy.hashCode(), parsed.hashCode()); + assertEquals(policy.toString(), parsed.toString()); + } + + @ParameterizedTest + @ValueSource(strings = { + "letters=0061;digits=;marks=0301", + "letters=61;digits=;marks=301", + "letters=0061-0061;digits=;marks=0301-0301", + "letters=0000061;digits=;marks=0301"}) + void testParseAcceptsShortLongAndSingletonRangeSpellings(String text) { + TokenizerCharacterPolicy expected = TokenizerCharacterPolicy.of( + A, EMPTY, CodePointSet.of(ACUTE_ACCENT)); + + assertEquals(expected, TokenizerCharacterPolicy.parse(text)); + } + + @Test + void testParseAcceptsLowerCaseHexDigits() { + TokenizerCharacterPolicy expected = TokenizerCharacterPolicy.of( + CodePointSet.of(DESERET_LETTER), EMPTY, CodePointSet.of(MUSICAL_MARK)); + + assertEquals(expected, TokenizerCharacterPolicy.parse("letters=10400;digits=;marks=1d165")); + } + + @ParameterizedTest + @NullSource + @ValueSource(strings = { + "", + "letters=;digits=;marks=", + "letters=0041-005A", + "letters=0041;digits=0030", + "digits=0030;letters=0041;marks=", + "letters=zz;digits=;marks=", + "letters=005A-0041;digits=;marks=", + "letters=0041;digits=0041;marks=", + "letters=D800;digits=;marks=", + "letters=0020;digits=;marks=", + "letters=0041,;digits=;marks=", + "letters=,0041;digits=;marks=", + "letters=0041-;digits=;marks=", + "letters=-0041;digits=;marks=", + "letters=0041-005A-0061;digits=;marks=", + "letters=110000;digits=;marks=", + "letters=0041;digits=;marks=;extra=", + "letters=0041;digits=;marks=0301;", + "letters 0041;digits=;marks=", + "LETTERS=0041;DIGITS=;MARKS=", + "letters=0x41;digits=;marks=", + "letters=U+0041;digits=;marks=", + "letters= 0041;digits=;marks=", + "letters=0041;digits=;marks=\n", + "^[A-Za-z0-9]+$"}) + void testParseRejectsMalformedText(String text) { + assertThrows(IllegalArgumentException.class, () -> TokenizerCharacterPolicy.parse(text)); + } + + @Test + void testParseErrorNamesTheOffendingText() { + IllegalArgumentException e = assertThrows(IllegalArgumentException.class, + () -> TokenizerCharacterPolicy.parse("letters=zz;digits=;marks=")); + + assertTrue(e.getMessage().contains("zz"), e.getMessage()); + } + + @Test + void testEqualsComparesTheThreeSets() { + TokenizerCharacterPolicy ascii = TokenizerCharacterPolicy.of( + CodePointSet.ofRange('A', 'Z').union(CodePointSet.ofRange('a', 'z')), + CodePointSet.ofRange('0', '9'), EMPTY); + TokenizerCharacterPolicy swapped = TokenizerCharacterPolicy.of( + CodePointSet.ofRange('0', '9'), + CodePointSet.ofRange('A', 'Z').union(CodePointSet.ofRange('a', 'z')), EMPTY); + TokenizerCharacterPolicy marked = TokenizerCharacterPolicy.of( + CodePointSet.ofRange('A', 'Z').union(CodePointSet.ofRange('a', 'z')), + CodePointSet.ofRange('0', '9'), CodePointSet.of(ACUTE_ACCENT)); + + assertEquals(TokenizerCharacterPolicy.ascii(), ascii); + assertEquals(TokenizerCharacterPolicy.ascii().hashCode(), ascii.hashCode()); + assertNotEquals(TokenizerCharacterPolicy.ascii(), swapped); + assertNotEquals(TokenizerCharacterPolicy.ascii(), marked); + assertNotEquals(TokenizerCharacterPolicy.ascii(), null); + assertNotEquals(TokenizerCharacterPolicy.ascii(), ASCII_TEXT); + } + private static String codePoints(int... codePoints) { StringBuilder value = new StringBuilder(); for (int codePoint : codePoints) { diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerFactoryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerFactoryTest.java index de478083c..98bef1ca0 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerFactoryTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerFactoryTest.java @@ -22,10 +22,20 @@ import java.io.ByteArrayOutputStream; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.util.Locale; -import java.util.regex.Pattern; +import java.util.Map; +import java.util.Properties; +import java.util.stream.Stream; +import java.util.zip.ZipEntry; +import java.util.zip.ZipInputStream; +import java.util.zip.ZipOutputStream; import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.CsvSource; +import org.junit.jupiter.params.provider.MethodSource; import opennlp.tools.dictionary.Dictionary; import opennlp.tools.formats.ResourceAsStreamFactory; @@ -33,9 +43,11 @@ import opennlp.tools.tokenize.DummyTokenizerFactory.DummyContextGenerator; import opennlp.tools.tokenize.DummyTokenizerFactory.DummyDictionary; import opennlp.tools.tokenize.lang.Factory; import opennlp.tools.util.InputStreamFactory; +import opennlp.tools.util.InvalidFormatException; import opennlp.tools.util.ObjectStream; import opennlp.tools.util.PlainTextByLineStream; import opennlp.tools.util.TrainingParameters; +import opennlp.tools.util.normalizer.CodePointSet; /** * Tests for the {@link TokenizerFactory} class. @@ -47,6 +59,12 @@ public class TokenizerFactoryTest { private static final Locale LOCALE_PORTUGUESE = Locale.of("pt"); private static final Locale LOCALE_SPANISH = Locale.of("es"); + private static final String MANIFEST_ENTRY = "manifest.properties"; + private static final String CHARACTER_POLICY_KEY = "characterPolicy"; + private static final String LEGACY_PATTERN_KEY = "alphaNumericPattern"; + private static final String CUSTOM_LEGACY_PATTERN = "^[A-Za-z0-9_]+$"; + private static final Factory LANGUAGE_FACTORY = new Factory(); + private static ObjectStream<TokenSample> createSampleStream() throws IOException { InputStreamFactory in = new ResourceAsStreamFactory( TokenizerFactoryTest.class, "/opennlp/tools/tokenize/token.train"); @@ -59,6 +77,14 @@ public class TokenizerFactoryTest { return TokenizerME.train(createSampleStream(), factory, TrainingParameters.defaultParams()); } + /* One English model with the alphanumeric optimization, rewritten by the manifest tests. */ + private static TokenizerModel engModel; + + @BeforeAll + static void trainEnglishModel() throws IOException { + engModel = train(new TokenizerFactory("eng", null, true)); + } + private static Dictionary loadAbbDictionary(Locale loc) throws IOException { final String abbrevDict; if (loc.equals(LOCALE_DUTCH)) { @@ -88,14 +114,13 @@ public class TokenizerFactoryTest { Dictionary dic = loadAbbDictionary(Locale.ENGLISH); final String lang = "eng"; - TokenizerModel model = train(new TokenizerFactory(lang, dic, false, null)); + TokenizerModel model = train(new TokenizerFactory(lang, dic, false)); TokenizerFactory factory = model.getFactory(); Assertions.assertNotNull(factory.getAbbreviationDictionary()); Assertions.assertInstanceOf(DefaultTokenContextGenerator.class, factory.getContextGenerator()); - String defaultPattern = Factory.DEFAULT_ALPHANUMERIC.pattern(); - Assertions.assertEquals(defaultPattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(Factory.DEFAULT_CHARACTER_POLICY, factory.getCharacterPolicy()); Assertions.assertEquals(lang, factory.getLanguageCode()); Assertions.assertEquals(lang, model.getLanguage()); Assertions.assertFalse(factory.isUseAlphaNumericOptimization()); @@ -110,7 +135,7 @@ public class TokenizerFactoryTest { Assertions.assertNotNull(factory.getAbbreviationDictionary()); Assertions.assertInstanceOf(DefaultTokenContextGenerator.class, factory.getContextGenerator()); - Assertions.assertEquals(defaultPattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(Factory.DEFAULT_CHARACTER_POLICY, factory.getCharacterPolicy()); Assertions.assertEquals(lang, factory.getLanguageCode()); Assertions.assertEquals(lang, model.getLanguage()); Assertions.assertFalse(factory.isUseAlphaNumericOptimization()); @@ -122,14 +147,13 @@ public class TokenizerFactoryTest { Dictionary dic = null; final String lang = "eng"; - TokenizerModel model = train(new TokenizerFactory(lang, dic, false, null)); + TokenizerModel model = train(new TokenizerFactory(lang, dic, false)); TokenizerFactory factory = model.getFactory(); Assertions.assertNull(factory.getAbbreviationDictionary()); Assertions.assertInstanceOf(DefaultTokenContextGenerator.class, factory.getContextGenerator()); - String defaultPattern = Factory.DEFAULT_ALPHANUMERIC.pattern(); - Assertions.assertEquals(defaultPattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(Factory.DEFAULT_CHARACTER_POLICY, factory.getCharacterPolicy()); Assertions.assertEquals(lang, factory.getLanguageCode()); Assertions.assertEquals(lang, model.getLanguage()); Assertions.assertFalse(factory.isUseAlphaNumericOptimization()); @@ -144,27 +168,29 @@ public class TokenizerFactoryTest { Assertions.assertNull(factory.getAbbreviationDictionary()); Assertions.assertInstanceOf(DefaultTokenContextGenerator.class, factory.getContextGenerator()); - Assertions.assertEquals(defaultPattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(Factory.DEFAULT_CHARACTER_POLICY, factory.getCharacterPolicy()); Assertions.assertEquals(lang, factory.getLanguageCode()); Assertions.assertEquals(lang, model.getLanguage()); Assertions.assertFalse(factory.isUseAlphaNumericOptimization()); } @Test - void testCustomPatternAndAlphaOpt() throws IOException { + void testCustomPolicyAndAlphaOpt() throws IOException { Dictionary dic = null; final String lang = "spa"; - String pattern = "^[0-9a-záéíóúüýñA-ZÁÉÍÓÚÝÑ]+$"; + TokenizerCharacterPolicy ascii = TokenizerCharacterPolicy.ascii(); + TokenizerCharacterPolicy policy = TokenizerCharacterPolicy.of( + ascii.getLetters().union(CodePointSet.of('ñ', 'Ñ')), ascii.getDigits(), + CodePointSet.of(0x0301)); - TokenizerModel model = train(new TokenizerFactory(lang, dic, true, - Pattern.compile(pattern))); + TokenizerModel model = train(new TokenizerFactory(lang, dic, true, policy)); TokenizerFactory factory = model.getFactory(); Assertions.assertNull(factory.getAbbreviationDictionary()); Assertions.assertInstanceOf(DefaultTokenContextGenerator.class, factory.getContextGenerator()); - Assertions.assertEquals(pattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(policy, factory.getCharacterPolicy()); Assertions.assertEquals(lang, factory.getLanguageCode()); Assertions.assertEquals(lang, model.getLanguage()); Assertions.assertTrue(factory.isUseAlphaNumericOptimization()); @@ -178,13 +204,250 @@ public class TokenizerFactoryTest { factory = fromSerialized.getFactory(); Assertions.assertNull(factory.getAbbreviationDictionary()); Assertions.assertInstanceOf(DefaultTokenContextGenerator.class, factory.getContextGenerator()); - Assertions.assertEquals(pattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(policy, factory.getCharacterPolicy()); Assertions.assertEquals(lang, factory.getLanguageCode()); Assertions.assertEquals(lang, model.getLanguage()); Assertions.assertTrue(factory.isUseAlphaNumericOptimization()); } - void checkCustomPatternForTokenizerME(String lang, String pattern, String sentence, + /** + * A language policy is written with the pattern of earlier releases that accepts the same + * characters, so those releases apply the same characters to the model. + */ + @Test + void testManifestCarriesTheCharacterPolicyAndItsLegacyPattern() throws IOException { + final String lang = "spa"; + TokenizerModel model = train(new TokenizerFactory(lang, null, true)); + + Properties manifest = readManifest(serialize(model)); + + Assertions.assertEquals(LANGUAGE_FACTORY.getCharacterPolicy(lang).toString(), + manifest.getProperty(CHARACTER_POLICY_KEY)); + Assertions.assertEquals(LegacyAlphaNumericPatterns.BY_LANGUAGE.get("es"), + manifest.getProperty(LEGACY_PATTERN_KEY)); + } + + /** A custom policy has no pattern of an earlier release, so only the policy is written. */ + @Test + void testManifestOfACustomPolicyHasNoLegacyPattern() throws IOException { + TokenizerCharacterPolicy policy = TokenizerCharacterPolicy.of( + CodePointSet.ofRange('a', 'z'), CodePointSet.ofRange('0', '9'), CodePointSet.of()); + TokenizerModel model = train(new TokenizerFactory("eng", null, true, policy)); + + Properties manifest = readManifest(serialize(model)); + + Assertions.assertEquals(policy.toString(), manifest.getProperty(CHARACTER_POLICY_KEY)); + Assertions.assertNull(manifest.getProperty(LEGACY_PATTERN_KEY)); + } + + /** + * Supplies each legacy regex with the language code of its policy. + * + * @return Regex and language code pairs. + */ + static Stream<Arguments> legacyPatterns() { + return LegacyAlphaNumericPatterns.BY_LANGUAGE.entrySet().stream() + .map(entry -> Arguments.of(entry.getValue(), entry.getKey())); + } + + @ParameterizedTest + @MethodSource("legacyPatterns") + void testLegacyPatternInTheManifestMapsToItsPolicy(String pattern, String lang) + throws IOException { + byte[] legacy = rewriteManifest(engModel, Map.of(LEGACY_PATTERN_KEY, pattern), CHARACTER_POLICY_KEY); + + TokenizerModel loaded = new TokenizerModel(new ByteArrayInputStream(legacy)); + + Assertions.assertEquals(LANGUAGE_FACTORY.getCharacterPolicy(lang), + loaded.getFactory().getCharacterPolicy()); + Assertions.assertTrue(loaded.getFactory().isUseAlphaNumericOptimization()); + Assertions.assertArrayEquals(new String[] {"test", ","}, new TokenizerME(loaded).tokenize("test,")); + } + + /** + * A model whose manifest has a custom pattern loads, and its alphanumeric optimization tests + * candidate tokens with that pattern: "it's" stays one token only because the pattern accepts + * the apostrophe; the English model splits it otherwise. + */ + @Test + @SuppressWarnings("removal") + void testCustomLegacyPatternInTheManifestIsEvaluatedAsARegularExpression() throws IOException { + final String withApostrophe = "^[A-Za-z0-9']+$"; + byte[] legacy = rewriteManifest(engModel, Map.of(LEGACY_PATTERN_KEY, withApostrophe), + CHARACTER_POLICY_KEY); + + TokenizerModel loaded = new TokenizerModel(new ByteArrayInputStream(legacy)); + + Assertions.assertNull(loaded.getFactory().getCharacterPolicy()); + Assertions.assertEquals(withApostrophe, + loaded.getFactory().getAlphaNumericPattern().pattern()); + Assertions.assertArrayEquals(new String[] {"it's"}, new TokenizerME(loaded).tokenize("it's")); + Assertions.assertArrayEquals(new String[] {"it", "'s"}, + new TokenizerME(engModel).tokenize("it's")); + } + + @Test + void testInvalidLegacyPatternInTheManifestIsRejected() throws IOException { + byte[] invalid = rewriteManifest(engModel, Map.of(LEGACY_PATTERN_KEY, "[a-z"), + CHARACTER_POLICY_KEY); + + InvalidFormatException e = assertRejectedAtLoad(invalid); + + Assertions.assertEquals("The tokenizer model for language 'eng' has an invalid " + + LEGACY_PATTERN_KEY + " '[a-z'", e.getMessage()); + } + + /** + * The deprecated constructor with a custom pattern writes only that pattern to the manifest, + * and the model tokenizes with it after loading. + */ + @Test + @SuppressWarnings("removal") + void testDeprecatedPatternConstructorRoundTrips() throws IOException { + TokenizerModel model = train(new TokenizerFactory("eng", null, true, + java.util.regex.Pattern.compile(CUSTOM_LEGACY_PATTERN))); + byte[] bytes = serialize(model); + + Properties manifest = readManifest(bytes); + TokenizerModel loaded = new TokenizerModel(new ByteArrayInputStream(bytes)); + + Assertions.assertEquals(CUSTOM_LEGACY_PATTERN, manifest.getProperty(LEGACY_PATTERN_KEY)); + Assertions.assertNull(manifest.getProperty(CHARACTER_POLICY_KEY)); + Assertions.assertNull(loaded.getFactory().getCharacterPolicy()); + Assertions.assertEquals(CUSTOM_LEGACY_PATTERN, + loaded.getFactory().getAlphaNumericPattern().pattern()); + } + + /** + * The deprecated pattern getter returns the pattern of earlier releases for a language policy. + * + * @param pattern The pattern of the language. + * @param lang The language code. + */ + @ParameterizedTest + @MethodSource("legacyPatterns") + @SuppressWarnings("removal") + void testDeprecatedPatternGetterReturnsTheLegacyPatternOfTheLanguage(String pattern, + String lang) { + TokenizerFactory factory = new TokenizerFactory(lang, null, true); + + Assertions.assertEquals(pattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(pattern, LANGUAGE_FACTORY.getAlphanumeric(lang).pattern()); + } + + /** A deprecated create call with a {@code null} pattern uses the policy of the language. */ + @Test + @SuppressWarnings("removal") + void testDeprecatedCreateWithoutPatternUsesTheLanguagePolicy() throws InvalidFormatException { + TokenizerFactory factory = TokenizerFactory.create(null, "spa", null, true, + (java.util.regex.Pattern) null); + + Assertions.assertEquals(LANGUAGE_FACTORY.getCharacterPolicy("spa"), + factory.getCharacterPolicy()); + } + + @ParameterizedTest + @CsvSource({"spa, es", "eng, en", "x-unspecified, en"}) + void testManifestWithoutPolicyOrPatternUsesTheLanguageDefault(String lang, String policyLang) + throws IOException { + TokenizerModel model = train(new TokenizerFactory(lang, null, true)); + byte[] bare = rewriteManifest(model, Map.of(), CHARACTER_POLICY_KEY, LEGACY_PATTERN_KEY); + + TokenizerModel loaded = new TokenizerModel(new ByteArrayInputStream(bare)); + + Assertions.assertEquals(LANGUAGE_FACTORY.getCharacterPolicy(policyLang), + loaded.getFactory().getCharacterPolicy()); + } + + @Test + void testMalformedCharacterPolicyInTheManifestIsRejected() throws IOException { + byte[] malformed = rewriteManifest(engModel, + Map.of(CHARACTER_POLICY_KEY, "letters=zz;digits=;marks=")); + + InvalidFormatException e = assertRejectedAtLoad(malformed); + + Assertions.assertTrue(e.getMessage().contains(CHARACTER_POLICY_KEY), e.getMessage()); + Assertions.assertTrue(e.getMessage().contains("zz"), e.getMessage()); + } + + @Test + void testCharacterPolicyInTheManifestWinsOverALegacyPattern() throws IOException { + byte[] both = rewriteManifest(engModel, Map.of(LEGACY_PATTERN_KEY, CUSTOM_LEGACY_PATTERN)); + + TokenizerModel loaded = new TokenizerModel(new ByteArrayInputStream(both)); + + Assertions.assertEquals(Factory.DEFAULT_CHARACTER_POLICY, loaded.getFactory().getCharacterPolicy()); + } + + /** + * Loads {@code model} and expects the {@link InvalidFormatException} of the factory, which + * {@link opennlp.tools.util.model.BaseModel} surfaces as the cause of an + * {@link IllegalArgumentException}. + * + * @param model The model bytes. + * @return The exception the factory raised. + */ + private static InvalidFormatException assertRejectedAtLoad(byte[] model) { + IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class, + () -> new TokenizerModel(new ByteArrayInputStream(model))); + return Assertions.assertInstanceOf(InvalidFormatException.class, e.getCause()); + } + + private static byte[] serialize(TokenizerModel model) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + model.serialize(out); + return out.toByteArray(); + } + + private static Properties readManifest(byte[] model) throws IOException { + try (ZipInputStream zip = new ZipInputStream(new ByteArrayInputStream(model))) { + ZipEntry entry; + while ((entry = zip.getNextEntry()) != null) { + if (MANIFEST_ENTRY.equals(entry.getName())) { + Properties manifest = new Properties(); + manifest.load(zip); + return manifest; + } + } + } + throw new IOException("No " + MANIFEST_ENTRY + " in the model"); + } + + /** + * Serializes {@code model} with its manifest changed, the way a model written by an + * earlier release would read. + * + * @param model The model to serialize. + * @param put The manifest properties to set. + * @param remove The manifest properties to drop. + * @return The model bytes. + */ + private static byte[] rewriteManifest(TokenizerModel model, Map<String, String> put, + String... remove) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + try (ZipInputStream zip = new ZipInputStream(new ByteArrayInputStream(serialize(model))); + ZipOutputStream rewritten = new ZipOutputStream(out)) { + ZipEntry entry; + while ((entry = zip.getNextEntry()) != null) { + rewritten.putNextEntry(new ZipEntry(entry.getName())); + if (MANIFEST_ENTRY.equals(entry.getName())) { + Properties manifest = new Properties(); + manifest.load(zip); + for (String key : remove) { + manifest.remove(key); + } + manifest.putAll(put); + manifest.store(rewritten, ""); + } else { + zip.transferTo(rewritten); + } + rewritten.closeEntry(); + } + } + return out.toByteArray(); + } + + void checkLanguagePolicyForTokenizerME(String lang, String sentence, int expectedNumTokens) throws IOException { Locale loc = Locale.ENGLISH; if ("dut".equals(lang) || "nld".equals(lang)) { @@ -203,7 +466,7 @@ public class TokenizerFactoryTest { loc = LOCALE_SPANISH; } TokenizerModel model = train(new TokenizerFactory(lang, loadAbbDictionary(loc), true, - Pattern.compile(pattern))); + LANGUAGE_FACTORY.getCharacterPolicy(lang))); TokenizerME tokenizer = new TokenizerME(model); String[] tokens = tokenizer.tokenize(sentence); @@ -222,120 +485,107 @@ public class TokenizerFactoryTest { } } - // For language specific patterns see: opennlp.tools.tokenize.lang.Factory + // For language specific policies see: opennlp.tools.tokenize.lang.Factory @Test - void testCustomPatternForTokenizerMEWithAbbreviationsDeu() throws IOException { + void testLanguagePolicyForTokenizerMEWithAbbreviationsDeu() throws IOException { String lang = "deu"; - String pattern = "^[A-Za-z0-9äéöüÄÉÖÜß]+$"; String sentence = "Ich wähle den auf S. 183 ff. mitgeteilten " + "Traum von der botanischen Monographie."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 14); + checkLanguagePolicyForTokenizerME(lang, sentence, 14); } @Test // to verify OPENNLP-1555 - void testCustomPatternForTokenizerMEWithMultiDotAbbreviationsDeu() throws IOException { + void testLanguagePolicyForTokenizerMEWithMultiDotAbbreviationsDeu() throws IOException { String lang = "deu"; - String pattern = "^[A-Za-z0-9äéöüÄÉÖÜß]+$"; // Adds an extra "z.B.", the compact form of "z. B." (zum Beispiel => for example) String sentence = "Ich wähle z.B. den auf S. 183 ff. mitgeteilten " + "Traum von der botanischen Monographie."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 15); + checkLanguagePolicyForTokenizerME(lang, sentence, 15); } @Test - void testCustomPatternForTokenizerMEWithAbbreviationsDut() throws IOException { + void testLanguagePolicyForTokenizerMEWithAbbreviationsDut() throws IOException { String lang = "dut"; - String pattern = "^[A-Za-z0-9äöüëèéïijÄÖÜËÉÈÏIJ]+$"; String sentence = "Ik kies voor de droom van de botanische monografie die " + "op p. 183 en volgende wordt beschreven."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 18); + checkLanguagePolicyForTokenizerME(lang, sentence, 18); } @Test // to verify OPENNLP-1555 - void testCustomPatternForTokenizerMEWithMultiDotAbbreviationsDut() throws IOException { + void testLanguagePolicyForTokenizerMEWithMultiDotAbbreviationsDut() throws IOException { String lang = "dut"; - String pattern = "^[A-Za-z0-9äöüëèéïijÄÖÜËÉÈÏIJ]+$"; String sentence = "Ik kies voor de droom van de botanische monografie die " + "op p. 183 e.v. wordt beschreven."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 17); + checkLanguagePolicyForTokenizerME(lang, sentence, 17); } @Test - void testCustomPatternForTokenizerMEWithAbbreviationsFra() throws IOException { + void testLanguagePolicyForTokenizerMEWithAbbreviationsFra() throws IOException { String lang = "fra"; - String pattern = "^[a-zA-Z0-9àâäèéêëîïôœùûüÿçÀÂÄÈÉÊËÎÏÔŒÙÛÜŸÇ]+$"; String sentence = "Je choisis le rêve de la monographie botanique communiqué à la p. 205."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 14); + checkLanguagePolicyForTokenizerME(lang, sentence, 14); } @Test - void testCustomPatternForTokenizerMEWithAbbreviationsPol() throws IOException { + void testLanguagePolicyForTokenizerMEWithAbbreviationsPol() throws IOException { String lang = "pol"; - String pattern = "^[A-Za-z0-9żźćńółęąśŻŹĆĄŚĘŁÓŃ]+$"; String sentence = "W szkicu autobiograficznym pt. moje życie i psychoanaliza Freud pisze, że " + "jego przodkowie żyli przez wiele lat w Kolonii."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 21); + checkLanguagePolicyForTokenizerME(lang, sentence, 21); } @Test - void testCustomPatternForTokenizerMEWithAbbreviationsPor() throws IOException { + void testLanguagePolicyForTokenizerMEWithAbbreviationsPor() throws IOException { String lang = "por"; - String pattern = "^[0-9a-záãâàéêíóõôúüçA-ZÁÃÂÀÉÊÍÓÕÔÚÜÇ]+$"; String sentence = "O povo pernambucano, tradicionalmente inimigo dos imperadores, " + "lembrava-se do tempo em que o Sr. D. Pedro de Alcantara dava-se ao luxo " + "de visitar o norte."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 28); + checkLanguagePolicyForTokenizerME(lang, sentence, 28); } @Test - void testCustomPatternForTokenizerMEWithAbbreviationsSpa() throws IOException { + void testLanguagePolicyForTokenizerMEWithAbbreviationsSpa() throws IOException { String lang = "spa"; - String pattern = "^[0-9a-záéíóúüýñA-ZÁÉÍÓÚÝÑ]+$"; String sentence = "Elegiremos el de la monografía botánica expuesto antes del " + "capítulo V en pág. 448 del presente volumen."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 18); + checkLanguagePolicyForTokenizerME(lang, sentence, 18); } @Test - void testCustomPatternForTokenizerMEPor() throws IOException { + void testLanguagePolicyForTokenizerMEPor() throws IOException { String lang = "por"; - String pattern = "^[0-9a-záãâàéêíóõôúüçA-ZÁÃÂÀÉÊÍÓÕÔÚÜÇ]+$"; String sentence = "Na floresta mágica a raposa dança com unicórnios felizes."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 10); + checkLanguagePolicyForTokenizerME(lang, sentence, 10); } @Test - void testCustomPatternForTokenizerMESpa() throws IOException { + void testLanguagePolicyForTokenizerMESpa() throws IOException { String lang = "spa"; - String pattern = "^[0-9a-záéíóúüýñA-ZÁÉÍÓÚÝÑ]+$"; String sentence = "En el verano los niños juegan en el parque y sus risas crean alegría."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 15); + checkLanguagePolicyForTokenizerME(lang, sentence, 15); } @Test - void testCustomPatternForTokenizerMECat() throws IOException { + void testLanguagePolicyForTokenizerMECat() throws IOException { String lang = "cat"; - String pattern = "^[0-9a-zàèéíïòóúüçA-ZÀÈÉÍÏÒÓÚÜÇ]+$"; String sentence = "Als xiuxiuejants avets l'os blau neda amb cignes i s'ho passen bé."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 15); + checkLanguagePolicyForTokenizerME(lang, sentence, 15); } @Test - void testCustomPatternForTokenizerMEIta() throws IOException { + void testLanguagePolicyForTokenizerMEIta() throws IOException { String lang = "ita"; - String pattern = "^[0-9a-zàèéìîíòóùüA-ZÀÈÉÌÎÍÒÓÙÜ]+$"; String sentence = "Cosa fare di domenica per migliorare il tuo lunedì."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 10); + checkLanguagePolicyForTokenizerME(lang, sentence, 10); } @Test - void testCustomPatternForTokenizerMEWithAbbreviationsIta() throws IOException { + void testLanguagePolicyForTokenizerMEWithAbbreviationsIta() throws IOException { String lang = "ita"; - String pattern = "^[0-9a-zàèéìîíòóùüA-ZÀÈÉÌÎÍÒÓÙÜ]+$"; String sentence = "La chiesa fu costruita fra il 1258 ed il 1308 ca. come chiesa " + "del convento degli Agostiniani."; - checkCustomPatternForTokenizerME(lang, pattern, sentence, 18); + checkLanguagePolicyForTokenizerME(lang, sentence, 18); } @Test @@ -343,10 +593,9 @@ public class TokenizerFactoryTest { Dictionary dic = null; String lang = "ita"; - String pattern = "^[0-9a-zàèéìîíòóùüA-ZÀÈÉÌÎÍÒÓÙÜ]+$"; TokenizerModel model = train(new TokenizerFactory(lang, dic, true, - Pattern.compile(pattern))); + LANGUAGE_FACTORY.getCharacterPolicy(lang))); TokenizerME tokenizer = new TokenizerME(model); String sentence = "La contrazione di \"dove è\" è \"dov'è\"."; @@ -365,10 +614,9 @@ public class TokenizerFactoryTest { Dictionary dic = null; String lang = "eng"; - String pattern = "^[A-Za-z0-9]+$"; TokenizerModel model = train(new TokenizerFactory(lang, dic, true, - Pattern.compile(pattern))); + LANGUAGE_FACTORY.getCharacterPolicy(lang))); TokenizerME tokenizer = new TokenizerME(model); String sentence = "The cat wasn't in the house and the dog wasn't either."; @@ -387,15 +635,14 @@ public class TokenizerFactoryTest { Dictionary dic = loadAbbDictionary(Locale.ENGLISH); final String lang = "eng"; - String pattern = "^[0-9A-Za-z]+$"; + TokenizerCharacterPolicy policy = TokenizerCharacterPolicy.ascii(); - TokenizerModel model = train(new DummyTokenizerFactory(lang, dic, true, - Pattern.compile(pattern))); + TokenizerModel model = train(new DummyTokenizerFactory(lang, dic, true, policy)); TokenizerFactory factory = model.getFactory(); Assertions.assertInstanceOf(DummyDictionary.class, factory.getAbbreviationDictionary()); Assertions.assertInstanceOf(DummyContextGenerator.class, factory.getContextGenerator()); - Assertions.assertEquals(pattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(policy, factory.getCharacterPolicy()); Assertions.assertEquals(lang, factory.getLanguageCode()); Assertions.assertEquals(lang, model.getLanguage()); Assertions.assertTrue(factory.isUseAlphaNumericOptimization()); @@ -409,7 +656,7 @@ public class TokenizerFactoryTest { factory = fromSerialized.getFactory(); Assertions.assertInstanceOf(DummyDictionary.class, factory.getAbbreviationDictionary()); Assertions.assertInstanceOf(DummyContextGenerator.class, factory.getContextGenerator()); - Assertions.assertEquals(pattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(policy, factory.getCharacterPolicy()); Assertions.assertEquals(lang, factory.getLanguageCode()); Assertions.assertEquals(lang, model.getLanguage()); Assertions.assertTrue(factory.isUseAlphaNumericOptimization()); @@ -419,15 +666,14 @@ public class TokenizerFactoryTest { void testCreateDummyFactory() throws IOException { Dictionary dic = loadAbbDictionary(Locale.ENGLISH); final String lang = "eng"; - String pattern = "^[0-9A-Za-z]+$"; + TokenizerCharacterPolicy policy = TokenizerCharacterPolicy.ascii(); TokenizerFactory factory = TokenizerFactory.create( - DummyTokenizerFactory.class.getCanonicalName(), lang, dic, true, - Pattern.compile(pattern)); + DummyTokenizerFactory.class.getCanonicalName(), lang, dic, true, policy); Assertions.assertInstanceOf(DummyDictionary.class, factory.getAbbreviationDictionary()); Assertions.assertInstanceOf(DummyContextGenerator.class, factory.getContextGenerator()); - Assertions.assertEquals(pattern, factory.getAlphaNumericPattern().pattern()); + Assertions.assertEquals(policy, factory.getCharacterPolicy()); Assertions.assertEquals(lang, factory.getLanguageCode()); Assertions.assertTrue(factory.isUseAlphaNumericOptimization()); } diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerMETest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerMETest.java index 6c2e4381a..b943b1876 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerMETest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerMETest.java @@ -19,9 +19,13 @@ package opennlp.tools.tokenize; import java.io.IOException; import java.nio.charset.StandardCharsets; +import java.util.stream.Stream; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; import opennlp.tools.formats.ResourceAsStreamFactory; import opennlp.tools.util.InputStreamFactory; @@ -30,6 +34,7 @@ import opennlp.tools.util.ObjectStream; import opennlp.tools.util.Parameters; import opennlp.tools.util.PlainTextByLineStream; import opennlp.tools.util.TrainingParameters; +import opennlp.tools.util.normalizer.CodePointSet; /** * Tests for the {@link TokenizerME} class. @@ -42,6 +47,8 @@ import opennlp.tools.util.TrainingParameters; */ public class TokenizerMETest { + private static final String CONTRACTION = "it's"; + @Test void testTokenizerSimpleModel() throws IOException { @@ -75,6 +82,60 @@ public class TokenizerMETest { Assertions.assertEquals("!", tokens[8]); } + /** + * Supplies sentences with the tokens the model trained from token.train produces for them. + * + * @return Sentence and token array pairs. + */ + static Stream<Arguments> pinnedSentences() { + return Stream.of( + Arguments.of("Sounds like it's not properly thought through!", + new String[] {"Sounds", "like", "it", "'s", "not", "properly", "thought", "through", "!"}), + Arguments.of("Mr. Vinken is chairman of Elsevier N.V., the Dutch publishing group.", + new String[] {"Mr", ".", "Vinken", "is", "chairman", "of", "Elsevier", "N", ".V", ".", ",", + "the", "Dutch", "publishing", "group", "."}), + Arguments.of("The cat wasn't in the house, and the dog wasn't either.", + new String[] {"The", "cat", "wasn", "'t", "in", "the", "house", ",", "and", "the", "dog", + "wasn", "'t", "either", "."}), + Arguments.of("Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29.", + new String[] {"Pierre", "Vinken", ",", "61", "years", "old", ",", "will", "join", "the", + "board", "as", "a", "nonexecutive", "director", "Nov", ".", "29", "."}), + Arguments.of("Call me at 555-1234 (anytime).", + new String[] {"Call", "me", "at", "555-1234", "(anytime)", "."}), + Arguments.of("It's a 3.5\" disk; don't lose it!", + new String[] {"It", "'s", "a", "3.5", "\"", "disk;", "don", "'t", "lose", "it!"})); + } + + /** + * Pins the output of a model trained with and one trained without the alphanumeric + * optimization: on these sentences the policy only skips tokens the model would not split. + */ + @ParameterizedTest + @MethodSource("pinnedSentences") + void testTokenizationWithAndWithoutAlphaNumericOptimization(String sentence, String[] expected) + throws IOException { + TokenizerME optimized = new TokenizerME(TokenizerTestUtil.createMaxentTokenModel(true)); + TokenizerME plain = new TokenizerME(TokenizerTestUtil.createMaxentTokenModel(false)); + + Assertions.assertTrue(optimized.useAlphaNumericOptimization()); + Assertions.assertFalse(plain.useAlphaNumericOptimization()); + Assertions.assertArrayEquals(expected, optimized.tokenize(sentence)); + Assertions.assertArrayEquals(expected, plain.tokenize(sentence)); + } + + @Test + void testCharacterPolicyDecidesWhichTokensStayWhole() throws IOException { + TokenizerCharacterPolicy ascii = TokenizerCharacterPolicy.ascii(); + TokenizerCharacterPolicy withApostrophe = TokenizerCharacterPolicy.of( + ascii.getLetters().union(CodePointSet.of('\'')), ascii.getDigits(), ascii.getMarks()); + TokenizerME tokenizer = new TokenizerME(TokenizerTestUtil.createMaxentTokenModel( + new TokenizerFactory("eng", null, true, withApostrophe))); + + Assertions.assertArrayEquals(new String[] {CONTRACTION}, tokenizer.tokenize(CONTRACTION)); + Assertions.assertArrayEquals(new String[] {"it", "'s"}, + new TokenizerME(TokenizerTestUtil.createMaxentTokenModel(true)).tokenize(CONTRACTION)); + } + @Test void testInsufficientData() { @@ -90,7 +151,7 @@ public class TokenizerMETest { mlParams.put(Parameters.ITERATIONS_PARAM, 100); mlParams.put(Parameters.CUTOFF_PARAM, 5); - TokenizerME.train(samples, TokenizerFactory.create(null, "eng", null, true, null), mlParams); + TokenizerME.train(samples, TokenizerFactory.create(null, "eng", null, true), mlParams); }); diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerTestUtil.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerTestUtil.java index a085f306e..b6cf38332 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerTestUtil.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/TokenizerTestUtil.java @@ -59,10 +59,20 @@ public class TokenizerTestUtil { mlParams.put(Parameters.CUTOFF_PARAM, 0); return TokenizerME.train(new CollectionObjectStream<>(samples), - TokenizerFactory.create(null, "eng", null, true, null), mlParams); + TokenizerFactory.create(null, "eng", null, true), mlParams); } static TokenizerModel createMaxentTokenModel() throws IOException { + return createMaxentTokenModel(true); + } + + static TokenizerModel createMaxentTokenModel(boolean useAlphaNumericOptimization) + throws IOException { + return createMaxentTokenModel( + TokenizerFactory.create(null, "eng", null, useAlphaNumericOptimization)); + } + + static TokenizerModel createMaxentTokenModel(TokenizerFactory factory) throws IOException { InputStreamFactory trainDataIn = new ResourceAsStreamFactory( TokenizerModel.class, "/opennlp/tools/tokenize/token.train"); @@ -74,7 +84,7 @@ public class TokenizerTestUtil { mlParams.put(Parameters.ITERATIONS_PARAM, 100); mlParams.put(Parameters.CUTOFF_PARAM, 0); - return TokenizerME.train(samples, TokenizerFactory.create(null, "eng", null, true, null), mlParams); + return TokenizerME.train(samples, factory, mlParams); } } diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lang/FactoryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lang/FactoryTest.java new file mode 100644 index 000000000..69845fcb4 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/lang/FactoryTest.java @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.tokenize.lang; + +import java.util.List; +import java.util.regex.Pattern; +import java.util.stream.Stream; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.CsvSource; +import org.junit.jupiter.params.provider.MethodSource; +import org.junit.jupiter.params.provider.NullSource; +import org.junit.jupiter.params.provider.ValueSource; + +import opennlp.tools.tokenize.LegacyAlphaNumericPatterns; +import opennlp.tools.tokenize.TokenizerCharacterPolicy; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Tests for the {@link Factory} class. The regular expressions in + * {@link LegacyAlphaNumericPatterns} are the oracle for the character policies that replace them. + */ +public class FactoryTest { + + private static final int DIGIT_COUNT = 10; + + private static final List<String> MULTI_CHARACTER_TOKENS = List.of( + "abc", "ABC", "0123456789", "a1b2", "Café", "CAFÉ", "straße", "naïve", "ijs", "IJs", + "żółw", "año", "Niño", "coração", "ÀÉÎÕÜ", "œuvre", "Ÿ", "maçã", "a-b", "a b", "a_b", + "ab\n", "ab\r\n", "\u00E9\u0301", "e\u0301", "a\u00A0b", "\u3042", "\uD801\uDC00", + "ab\uD801\uDC00", "a\uD800", "\uDC00a"); + + private static final List<String> SUPPLEMENTARY_TOKENS = List.of( + "\uD801\uDC00", "\uD835\uDFD8", "\uD83D\uDE00", "a\uD801\uDC00", "\uD801\uDC00a"); + + private final Factory factory = new Factory(); + + /** + * Supplies one representative language code per policy with its legacy regex. + * + * @return Language code and regex pairs. + */ + static Stream<Arguments> shippedPolicies() { + return LegacyAlphaNumericPatterns.BY_LANGUAGE.entrySet().stream() + .map(entry -> Arguments.of(entry.getKey(), entry.getValue())); + } + + @ParameterizedTest + @MethodSource("shippedPolicies") + void testPolicyMatchesLegacyRegexOnEveryBmpCodePoint(String languageCode, String regex) { + Pattern oracle = Pattern.compile(regex); + TokenizerCharacterPolicy policy = factory.getCharacterPolicy(languageCode); + + for (int codePoint = 0; codePoint <= Character.MAX_VALUE; codePoint++) { + String token = String.valueOf((char) codePoint); + assertEquals(oracle.matcher(token).matches(), policy.test(token), + languageCode + " U+" + Integer.toHexString(codePoint)); + } + } + + @ParameterizedTest + @MethodSource("shippedPolicies") + void testPolicyMatchesLegacyRegexOnMultiCharacterTokens(String languageCode, String regex) { + Pattern oracle = Pattern.compile(regex); + TokenizerCharacterPolicy policy = factory.getCharacterPolicy(languageCode); + + for (String token : MULTI_CHARACTER_TOKENS) { + assertEquals(oracle.matcher(token).matches(), policy.test(token), + languageCode + " '" + token + "'"); + } + } + + @ParameterizedTest + @MethodSource("shippedPolicies") + void testPolicyRejectsSupplementaryCharactersLikeLegacyRegex(String languageCode, String regex) { + Pattern oracle = Pattern.compile(regex); + TokenizerCharacterPolicy policy = factory.getCharacterPolicy(languageCode); + + for (String token : SUPPLEMENTARY_TOKENS) { + assertFalse(oracle.matcher(token).matches(), languageCode + " oracle '" + token + "'"); + assertFalse(policy.test(token), languageCode + " '" + token + "'"); + } + } + + @ParameterizedTest + @MethodSource("shippedPolicies") + void testShippedPoliciesHaveAsciiDigitsAndNoMarks(String languageCode, String regex) { + TokenizerCharacterPolicy policy = factory.getCharacterPolicy(languageCode); + + assertEquals(DIGIT_COUNT, policy.getDigits().size(), languageCode); + for (int digit = '0'; digit <= '9'; digit++) { + assertTrue(policy.getDigits().contains(digit), languageCode); + } + assertTrue(policy.getMarks().isEmpty(), languageCode); + } + + @ParameterizedTest + @MethodSource("shippedPolicies") + void testLetterSetIsExactlyTheLegacyClassMinusDigits(String languageCode, String regex) { + Pattern oracle = Pattern.compile(regex); + TokenizerCharacterPolicy policy = factory.getCharacterPolicy(languageCode); + + int expectedLetters = 0; + for (int codePoint = 0; codePoint <= Character.MAX_VALUE; codePoint++) { + if (oracle.matcher(String.valueOf((char) codePoint)).matches() + && !policy.getDigits().contains(codePoint)) { + expectedLetters++; + assertTrue(policy.getLetters().contains(codePoint), + languageCode + " U+" + Integer.toHexString(codePoint)); + } + } + assertEquals(expectedLetters, policy.getLetters().size(), languageCode); + } + + @ParameterizedTest + @CsvSource({ + "es, spa", + "it, ita", + "pt, por", + "ca, cat", + "pl, pol", + "de, deu", + "de, ger", + "fr, fre", + "fr, fra", + "nl, nld", + "nl, dut"}) + void testTwoAndThreeLetterCodesShareOnePolicy(String twoLetter, String threeLetter) { + assertSame(factory.getCharacterPolicy(twoLetter), factory.getCharacterPolicy(threeLetter)); + } + + @ParameterizedTest + @NullSource + @ValueSource(strings = {"en", "eng", "xx", "", "ES", "x-unspecified"}) + void testUnknownOrAbsentLanguageUsesTheDefaultPolicy(String languageCode) { + assertSame(Factory.DEFAULT_CHARACTER_POLICY, factory.getCharacterPolicy(languageCode)); + } + + @Test + void testDefaultPolicyIsTheAsciiPreset() { + assertSame(TokenizerCharacterPolicy.ascii(), Factory.DEFAULT_CHARACTER_POLICY); + } +} diff --git a/opennlp-docs/src/docbkx/tokenizer.xml b/opennlp-docs/src/docbkx/tokenizer.xml index 2eb69fad2..7e9a6e476 100644 --- a/opennlp-docs/src/docbkx/tokenizer.xml +++ b/opennlp-docs/src/docbkx/tokenizer.xml @@ -213,7 +213,8 @@ double[] tokenProbs = tokenizer.getTokenProbabilities();]]> not infer Unicode character categories, so callers choose the exact script coverage. The predicate checks a complete candidate without finding token boundaries or changing its text. - This policy does not yet configure tokenizer training or inference. + <code>TokenizerME</code> applies the policy of its model for the alphanumeric + optimization; the training section explains how a model selects one. </para> <programlisting language="java"> <![CDATA[TokenizerCharacterPolicy policy = TokenizerCharacterPolicy.of( @@ -374,6 +375,49 @@ try { }]]> </programlisting> </para> + <para> + The last argument of <code>TokenizerFactory.create</code> is the + <code>TokenizerCharacterPolicy</code> that the alphanumeric optimization uses to + accept a candidate token without consulting the model. Leaving it out selects + the policy for the language code: ASCII letters and digits for most languages, + and ASCII plus the letters of the language for Catalan (ca), Dutch (nl), + French (fr), German (de), Italian (it), Polish (pl), Portuguese (pt) and + Spanish (es); the ISO 639-2 codes are accepted too. None of these presets has + marks, so a decomposed character such as <code>e</code> followed by U+0301 is + not accepted by them; supply a policy with the combining marks of your text to + accept it. The policy is written to the model manifest under the property + <code>characterPolicy</code> as comma separated hex code points and ranges, for + example <code>letters=0041-005A,0061-007A;digits=0030-0039;marks=</code> for the + ASCII preset, and read back from there when the model is loaded. + <code>TokenizerCharacterPolicy.toString()</code> produces this form and + <code>TokenizerCharacterPolicy.parse(String)</code> reads it. For a language + preset the manifest also gets the <code>alphaNumericPattern</code> that earlier + releases wrote for that language, so they tokenize the model the same way. + </para> + <para> + Models written by earlier releases carry the property + <code>alphaNumericPattern</code> with a regular expression instead. A model with + one of the expressions OpenNLP itself wrote for the languages above loads + unchanged and uses the policy with the same characters, so the released models + need no conversion. A model trained with a custom pattern loads too and tests + candidate tokens with that regular expression. This regular expression support, + with the <code>Pattern</code> arguments of the <code>TokenizerFactory</code> + constructor, <code>TokenizerFactory.create</code> and the + <code>TokSpanEventStream</code> constructor and + <code>TokenizerFactory.getAlphaNumericPattern()</code>, is deprecated and will be + removed in 4.0. To migrate such a model, express the letters, digits and marks of + its pattern as a <code>TokenizerCharacterPolicy</code>, then either train the + model again with that policy, or replace <code>alphaNumericPattern</code> in the + <code>manifest.properties</code> entry of the model file with + <code>characterPolicy</code> set to the text form of the policy and zip the model + again. + </para> + <para> + A model with no policy and no pattern in its manifest uses the policy for its language + code, as a new factory does. OpenNLP 2.x reads only + <code>alphaNumericPattern</code>, so it applies its default pattern to a model + written with a custom policy. + </para> </section> </section> diff --git a/opennlp-eval-tests/src/test/java/opennlp/tools/eval/ArvoresDeitadasEval.java b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/ArvoresDeitadasEval.java index 1478a95af..e7e3b9ddb 100644 --- a/opennlp-eval-tests/src/test/java/opennlp/tools/eval/ArvoresDeitadasEval.java +++ b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/ArvoresDeitadasEval.java @@ -117,7 +117,7 @@ public class ArvoresDeitadasEval extends AbstractEvalTest { TokenizerCrossValidator validator; TokenizerFactory tokFactory = TokenizerFactory.create(null, LANG, null, - true, null); + true); validator = new opennlp.tools.tokenize.TokenizerCrossValidator(params, tokFactory); diff --git a/opennlp-eval-tests/src/test/java/opennlp/tools/eval/EnglishTokenSampleStreamEval.java b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/EnglishTokenSampleStreamEval.java index 512cc49b6..425bdc978 100644 --- a/opennlp-eval-tests/src/test/java/opennlp/tools/eval/EnglishTokenSampleStreamEval.java +++ b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/EnglishTokenSampleStreamEval.java @@ -110,7 +110,7 @@ public class EnglishTokenSampleStreamEval extends AbstractEvalTest { final TrainingParameters params = ModelUtil.createDefaultTrainingParameters(); final TokenizerModel model = TokenizerME.train( new CollectionObjectStream<>(samples(trainLines)), - TokenizerFactory.create(null, "eng", null, true, null), params); + TokenizerFactory.create(null, "eng", null, true), params); final TokenizerEvaluator evaluator = new TokenizerEvaluator(new TokenizerME(model)); evaluator.evaluate(new CollectionObjectStream<>(samples(testLines))); final double fMeasure = evaluator.getFMeasure().getFMeasure();
