This is an automated email from the ASF dual-hosted git repository.
mawiesne pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 021b731d5 GH-1476: Let the parsers take a caller-supplied POSTagger
and Chunker (#1478)
021b731d5 is described below
commit 021b731d53a08f593790d056fcbdf7f9a3ecbccd
Author: Kristian Rickert <[email protected]>
AuthorDate: Sun Oct 18 14:48:43 2026 -0400
GH-1476: Let the parsers take a caller-supplied POSTagger and Chunker
(#1478)
* GH-1476: Let the parsers take a caller-supplied POSTagger and Chunker
Both bottom-up parsers get a public constructor that takes a POSTagger
and a Chunker next to the ParserModel. The build, attach and check
models and the head rules still come from the model; its tagger and
chunker models are not used, so an application can plug in its own
tagger without loading and discarding the model's. The model-only
constructors delegate to the new one.
ParserFactory.create(model, tagger, chunker, beamSize,
advancePercentage) picks the parser for the model's ParserType with the
supplied components. The existing create methods build the model's
POSTaggerME and ChunkerME and call it, so the type dispatch exists once.
Arguments are checked at the public boundary with an
IllegalArgumentException: null model, tagger, chunker or head rules, a
beam size below 1, and an advance percentage outside (0, 1]. The range
checks are ParamChecks.requirePositive and the new
ParamChecks.requirePositiveFraction, and the README lists the stricter
checks as a breaking change. Tests pin that supplied components are
used, that the model's own tagger and chunker models are not read, that
the model's components give the parses of the model-only factory
method, and the accept and reject cases of each check. The manual shows
the new factory method in the parsing API section.
---------
Signed-off-by: Kristian Rickert <[email protected]>
Co-authored-by: Claude Code 2.1.296 (Opus 5.5) <[email protected]>
---
README.md | 1 +
.../main/java/opennlp/tools/util/ParamChecks.java | 19 ++
.../java/opennlp/tools/util/ParamChecksTest.java | 15 ++
.../tools/parser/AbstractBottomUpParser.java | 22 ++-
.../java/opennlp/tools/parser/ParserFactory.java | 61 +++++-
.../java/opennlp/tools/parser/chunking/Parser.java | 41 +++-
.../opennlp/tools/parser/treeinsert/Parser.java | 44 ++++-
.../tools/parser/AbstractParserModelTest.java | 111 +++++++++++
.../opennlp/tools/parser/ParserFactoryTest.java | 207 +++++++++++++++++++++
.../java/opennlp/tools/parser/ParserTestUtil.java | 144 ++++++++++++++
.../opennlp/tools/parser/chunking/ParserTest.java | 8 +
.../tools/parser/treeinsert/ParserTest.java | 8 +
opennlp-docs/src/docbkx/parser.xml | 11 ++
13 files changed, 666 insertions(+), 26 deletions(-)
diff --git a/README.md b/README.md
index 1ba0d1acc..42ec642ec 100644
--- a/README.md
+++ b/README.md
@@ -141,6 +141,7 @@ The list below covers the changes most likely to affect
existing code. It is not
- CoNLL-U reader: a multiword token range whose word lines are missing from
the sentence throws `InvalidFormatException`.
- `PlainTextByLineStream.reset()` closes the previous reader, so its
`InputStreamFactory` must return a new stream for each pass.
- `RealBasicEventStream`: a line that contains only an outcome is read as an
event without contexts and no longer ends the stream.
+- Parser: `ParserFactory.create` and the chunking and tree insert `Parser`
constructors throw `IllegalArgumentException` for a beam size below 1 or an
advance percentage outside (0, 1]
([GH-1476](https://github.com/apache/opennlp/issues/1476)).
### Thread safety
diff --git a/opennlp-api/src/main/java/opennlp/tools/util/ParamChecks.java
b/opennlp-api/src/main/java/opennlp/tools/util/ParamChecks.java
index cda645186..895db2971 100644
--- a/opennlp-api/src/main/java/opennlp/tools/util/ParamChecks.java
+++ b/opennlp-api/src/main/java/opennlp/tools/util/ParamChecks.java
@@ -39,6 +39,7 @@ public final class ParamChecks {
private static final String NOT_NEGATIVE_SUFFIX = " must not be negative";
private static final String FINITE_SUFFIX = " must be finite";
private static final String POSITIVE_SUFFIX = " must be positive";
+ private static final String POSITIVE_FRACTION_SUFFIX = " must be in (0, 1]";
private ParamChecks() {
// utility class
@@ -225,6 +226,24 @@ public final class ParamChecks {
return value;
}
+ /**
+ * Returns {@code value} if it is in (0, 1].
+ * <p>
+ * The message of a thrown exception is {@code "<name> must be in (0, 1]"}.
+ *
+ * @param value The argument to check.
+ * @param name The parameter name used in the exception message.
+ * @return {@code value}, always in {@code (0, 1]}.
+ * @throws IllegalArgumentException Thrown if {@code value} is not in {@code
(0, 1]},
+ * including {@code NaN}.
+ */
+ public static double requirePositiveFraction(double value, String name) {
+ if (!(value > 0 && value <= 1)) {
+ throw new IllegalArgumentException(name + POSITIVE_FRACTION_SUFFIX);
+ }
+ return value;
+ }
+
/**
* Checks two arrays that hold corresponding values. Both arrays must be
non-empty,
* have the same length, and contain no {@code null} entries.
diff --git a/opennlp-api/src/test/java/opennlp/tools/util/ParamChecksTest.java
b/opennlp-api/src/test/java/opennlp/tools/util/ParamChecksTest.java
index 92c0af06b..c64649e6b 100644
--- a/opennlp-api/src/test/java/opennlp/tools/util/ParamChecksTest.java
+++ b/opennlp-api/src/test/java/opennlp/tools/util/ParamChecksTest.java
@@ -219,6 +219,21 @@ public class ParamChecksTest {
assertEquals("weight must be finite", e.getMessage());
}
+ @ParameterizedTest
+ @ValueSource(doubles = {Double.MIN_VALUE, 0.5, 1.0})
+ void testPositiveFractionIsReturnedUnchanged(double value) {
+ assertEquals(value, ParamChecks.requirePositiveFraction(value, "share"));
+ }
+
+ @ParameterizedTest
+ @ValueSource(doubles = {0.0, -0.0, -0.5, 1.0000000000000002,
Double.POSITIVE_INFINITY,
+ Double.NaN})
+ void testNonPositiveFractionIsRejected(double value) {
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> ParamChecks.requirePositiveFraction(value, "share"));
+ assertEquals("share must be in (0, 1]", e.getMessage());
+ }
+
/**
* Supplies invalid parallel arrays, including inputs that violate several
checks.
*
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/AbstractBottomUpParser.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/AbstractBottomUpParser.java
index 34c13d9b7..dc7d2fa30 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/AbstractBottomUpParser.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/AbstractBottomUpParser.java
@@ -34,6 +34,7 @@ import opennlp.tools.ngram.NGramModel;
import opennlp.tools.parser.chunking.ParserEventStream;
import opennlp.tools.postag.POSTagger;
import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
import opennlp.tools.util.Parameters;
import opennlp.tools.util.Sequence;
import opennlp.tools.util.Span;
@@ -153,15 +154,26 @@ public abstract class AbstractBottomUpParser implements
Parser {
*/
protected boolean createDerivationString = false;
+ /**
+ * Initializes a bottom-up parser with the components every subclass needs.
+ *
+ * @param tagger The {@link POSTagger} used to tag. Must not be {@code null}.
+ * @param chunker The {@link Chunker} used to chunk. Must not be {@code
null}.
+ * @param headRules The {@link HeadRules} for head word percolation. Must
not be {@code null}.
+ * @param beamSize The number of different parses kept during parsing. Must
be at least {@code 1}.
+ * @param advancePercentage The minimal amount of probability mass which
advanced outcomes
+ * must represent. Must be greater than {@code 0}
and at most {@code 1}.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null} or
out of range.
+ */
public AbstractBottomUpParser(POSTagger tagger, Chunker chunker, HeadRules
headRules,
int beamSize, double advancePercentage) {
- this.tagger = tagger;
- this.chunker = chunker;
- this.M = beamSize;
+ this.tagger = ParamChecks.requireNonNullArg(tagger, "tagger");
+ this.chunker = ParamChecks.requireNonNullArg(chunker, "chunker");
+ this.headRules = ParamChecks.requireNonNullArg(headRules, "headRules");
+ this.M = ParamChecks.requirePositive(beamSize, "beamSize");
this.K = beamSize;
- this.Q = advancePercentage;
+ this.Q = ParamChecks.requirePositiveFraction(advancePercentage,
"advancePercentage");
reportFailedParse = true;
- this.headRules = headRules;
this.punctSet = headRules.getPunctuationTags();
odh = new TreeSet<>();
ndh = new TreeSet<>();
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/ParserFactory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/ParserFactory.java
index 91ed09553..2f9886253 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/ParserFactory.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/ParserFactory.java
@@ -17,6 +17,12 @@
package opennlp.tools.parser;
+import opennlp.tools.chunker.Chunker;
+import opennlp.tools.chunker.ChunkerME;
+import opennlp.tools.postag.POSTagger;
+import opennlp.tools.postag.POSTaggerME;
+import opennlp.tools.util.ParamChecks;
+
public class ParserFactory {
private ParserFactory() {
@@ -26,29 +32,65 @@ public class ParserFactory {
* Instantiates a {@link Parser} via a given {@code model} and
* other configuration parameters.
*
- * @param model The {@link ParserModel} to use.
- * @param beamSize The number of different parses kept during parsing.
+ * @param model The {@link ParserModel} to use. Must not be {@code null}.
+ * @param beamSize The number of different parses kept during parsing. Must
be at least
+ * {@code 1}.
* @param advancePercentage The minimal amount of probability mass which
advanced outcomes
* must represent. Only outcomes which contribute
to the top
- * {@code advancePercentage} will be explored.
+ * {@code advancePercentage} will be explored. Must
be greater
+ * than {@code 0} and at most {@code 1}.
*
* @return A valid {@link Parser} instance.
+ * @throws IllegalArgumentException Thrown if {@code model} is {@code null}
or
+ * a parameter is out of range.
* @throws IllegalStateException Thrown if the {@link ParserType} is not
supported.
*
* @see Parser
* @see ParserModel
*/
public static Parser create(ParserModel model, int beamSize, double
advancePercentage) {
+ ParamChecks.requireNonNullArg(model, "model");
+ return create(model, new POSTaggerME(model.getParserTaggerModel()),
+ new ChunkerME(model.getParserChunkerModel()), beamSize,
advancePercentage);
+ }
- if (ParserType.CHUNKING.equals(model.getParserType())) {
- return new opennlp.tools.parser.chunking.Parser(model, beamSize,
advancePercentage);
+ /**
+ * Instantiates a {@link Parser} for the {@link ParserType} of a given
{@code model} that tags
+ * and chunks with a caller-supplied {@link POSTagger} and {@link Chunker}.
The build, attach
+ * and check models and the head rules are taken from {@code model}; its
tagger and chunker
+ * models are not used.
+ *
+ * @param model The {@link ParserModel} to use. Must not be {@code null}.
+ * @param tagger The {@link POSTagger} used to tag. Must not be {@code null}.
+ * @param chunker The {@link Chunker} used to chunk. Must not be {@code
null}.
+ * @param beamSize The number of different parses kept during parsing. Must
be at least
+ * {@code 1}.
+ * @param advancePercentage The minimal amount of probability mass which
advanced outcomes
+ * must represent. Only outcomes which contribute
to the top
+ * {@code advancePercentage} will be explored. Must
be greater
+ * than {@code 0} and at most {@code 1}.
+ *
+ * @return A valid {@link Parser} instance.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null} or
out of range.
+ * @throws IllegalStateException Thrown if the {@link ParserType} is not
supported.
+ *
+ * @see Parser
+ * @see ParserModel
+ * @since 3.0.0
+ */
+ public static Parser create(ParserModel model, POSTagger tagger, Chunker
chunker,
+ int beamSize, double advancePercentage) {
+ ParserType type = ParamChecks.requireNonNullArg(model,
"model").getParserType();
+ if (ParserType.CHUNKING.equals(type)) {
+ return new opennlp.tools.parser.chunking.Parser(model, tagger, chunker,
+ beamSize, advancePercentage);
}
- else if (ParserType.TREEINSERT.equals(model.getParserType())) {
- return new opennlp.tools.parser.treeinsert.Parser(model, beamSize,
advancePercentage);
+ else if (ParserType.TREEINSERT.equals(type)) {
+ return new opennlp.tools.parser.treeinsert.Parser(model, tagger, chunker,
+ beamSize, advancePercentage);
}
else {
- throw new IllegalStateException("Unexpected ParserType: " +
- model.getParserType().name());
+ throw new IllegalStateException("Unexpected ParserType: " + type);
}
}
@@ -59,6 +101,7 @@ public class ParserFactory {
* @param model The {@link ParserModel} to use.
*
* @return A valid {@link Parser} instance.
+ * @throws IllegalArgumentException Thrown if {@code model} is {@code null}.
* @throws IllegalStateException Thrown if the {@link ParserType} is not
supported.
*
* @see Parser
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/chunking/Parser.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/chunking/Parser.java
index c9f79fa64..863949ec7 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/chunking/Parser.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/chunking/Parser.java
@@ -50,6 +50,7 @@ import opennlp.tools.postag.POSTagger;
import opennlp.tools.postag.POSTaggerFactory;
import opennlp.tools.postag.POSTaggerME;
import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
import opennlp.tools.util.Span;
import opennlp.tools.util.TrainingParameters;
@@ -87,19 +88,47 @@ public class Parser extends AbstractBottomUpParser {
*
* @param model The {@link ParserModel} to use.
* @param beamSize The number of different parses kept during
parsing.
+ * Must be at least {@code 1}.
* @param advancePercentage The minimal amount of probability mass which
advanced outcomes
* must represent. Only outcomes which contribute
to the top
* {@code advancePercentage} will be explored.
- * @throws IllegalStateException Thrown if the {@link ParserType} is not
supported.
+ * Must be greater than {@code 0} and at most
{@code 1}.
+ * @throws IllegalArgumentException Thrown if {@code model} is {@code null}
or
+ * a parameter is out of range.
* @see ParserModel
* @see POSTaggerME
* @see ChunkerME
*/
public Parser(ParserModel model, int beamSize, double advancePercentage) {
- this(model.getBuildModel(), model.getCheckModel(),
+ this(ParamChecks.requireNonNullArg(model, "model"),
new POSTaggerME(model.getParserTaggerModel()),
- new ChunkerME(model.getParserChunkerModel()),
- model.getHeadRules(), beamSize, advancePercentage);
+ new ChunkerME(model.getParserChunkerModel()), beamSize,
advancePercentage);
+ }
+
+ /**
+ * Instantiates a {@link Parser} via a given {@code model} and a
caller-supplied
+ * {@link POSTagger} and {@link Chunker}. The build and check models and the
head rules
+ * are taken from {@code model}; its tagger and chunker models are not used.
+ *
+ * @param model The {@link ParserModel} to use. Must not be
{@code null}.
+ * @param tagger The {@link POSTagger} used to tag. Must not be
{@code null}.
+ * @param chunker The {@link Chunker} used to chunk. Must not be
{@code null}.
+ * @param beamSize The number of different parses kept during
parsing.
+ * Must be at least {@code 1}.
+ * @param advancePercentage The minimal amount of probability mass which
advanced outcomes
+ * must represent. Only outcomes which contribute
to the top
+ * {@code advancePercentage} will be explored.
+ * Must be greater than {@code 0} and at most
{@code 1}.
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null} or
out of range.
+ * @see ParserModel
+ * @see POSTagger
+ * @see Chunker
+ * @since 3.0.0
+ */
+ public Parser(ParserModel model, POSTagger tagger, Chunker chunker,
+ int beamSize, double advancePercentage) {
+ this(ParamChecks.requireNonNullArg(model, "model").getBuildModel(),
model.getCheckModel(),
+ tagger, chunker, model.getHeadRules(), beamSize, advancePercentage);
}
/**
@@ -107,8 +136,8 @@ public class Parser extends AbstractBottomUpParser {
* Uses the default implementations of {@link POSTaggerME} and {@link
ChunkerME}
* and default values for {@code beamSize} and {@code advancePercentage}.
*
- * @param model The {@link ParserModel} to use.
- * @throws IllegalStateException Thrown if the {@link ParserType} is not
supported.
+ * @param model The {@link ParserModel} to use. Must not be {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code model} is {@code null}.
* @see ParserModel
* @see POSTaggerME
* @see ChunkerME
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/treeinsert/Parser.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/treeinsert/Parser.java
index f4f28c6ac..6fc17b95c 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/treeinsert/Parser.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/parser/treeinsert/Parser.java
@@ -50,6 +50,7 @@ import opennlp.tools.postag.POSTagger;
import opennlp.tools.postag.POSTaggerFactory;
import opennlp.tools.postag.POSTaggerME;
import opennlp.tools.util.ObjectStream;
+import opennlp.tools.util.ParamChecks;
import opennlp.tools.util.Parameters;
import opennlp.tools.util.TrainingParameters;
@@ -116,20 +117,51 @@ public class Parser extends AbstractBottomUpParser {
*
* @param model The {@link ParserModel} to use.
* @param beamSize The number of different parses kept during parsing.
+ * Must be at least {@code 1}.
* @param advancePercentage The minimal amount of probability mass which
advanced outcomes
* must represent. Only outcomes which contribute
to the top
* {@code advancePercentage} will be explored.
+ * Must be greater than {@code 0} and at most
{@code 1}.
*
- * @throws IllegalStateException Thrown if the {@link ParserType} is not
supported.
+ * @throws IllegalArgumentException Thrown if {@code model} is {@code null}
or
+ * a parameter is out of range.
*
* @see ParserModel
* @see POSTaggerME
* @see ChunkerME
*/
public Parser(ParserModel model, int beamSize, double advancePercentage) {
- this(model.getBuildModel(), model.getAttachModel(), model.getCheckModel(),
- new POSTaggerME(model.getParserTaggerModel()), new
ChunkerME(model.getParserChunkerModel()),
- model.getHeadRules(), beamSize, advancePercentage);
+ this(ParamChecks.requireNonNullArg(model, "model"),
+ new POSTaggerME(model.getParserTaggerModel()),
+ new ChunkerME(model.getParserChunkerModel()), beamSize,
advancePercentage);
+ }
+
+ /**
+ * Instantiates a {@link Parser} via a given {@code model} and a
caller-supplied
+ * {@link POSTagger} and {@link Chunker}. The build, attach and check models
and the
+ * head rules are taken from {@code model}; its tagger and chunker models
are not used.
+ *
+ * @param model The {@link ParserModel} to use. Must not be {@code null}.
+ * @param tagger The {@link POSTagger} used to tag. Must not be {@code null}.
+ * @param chunker The {@link Chunker} used to chunk. Must not be {@code
null}.
+ * @param beamSize The number of different parses kept during parsing.
+ * Must be at least {@code 1}.
+ * @param advancePercentage The minimal amount of probability mass which
advanced outcomes
+ * must represent. Only outcomes which contribute
to the top
+ * {@code advancePercentage} will be explored.
+ * Must be greater than {@code 0} and at most
{@code 1}.
+ *
+ * @throws IllegalArgumentException Thrown if a parameter is {@code null} or
out of range.
+ *
+ * @see ParserModel
+ * @see POSTagger
+ * @see Chunker
+ * @since 3.0.0
+ */
+ public Parser(ParserModel model, POSTagger tagger, Chunker chunker,
+ int beamSize, double advancePercentage) {
+ this(ParamChecks.requireNonNullArg(model, "model").getBuildModel(),
model.getAttachModel(),
+ model.getCheckModel(), tagger, chunker, model.getHeadRules(),
beamSize, advancePercentage);
}
/**
@@ -137,9 +169,9 @@ public class Parser extends AbstractBottomUpParser {
* Uses the default implementations of {@link POSTaggerME} and {@link
ChunkerME}
* and default values for {@code beamSize} and {@code advancePercentage}.
*
- * @param model The {@link ParserModel} to use.
+ * @param model The {@link ParserModel} to use. Must not be {@code null}.
*
- * @throws IllegalStateException Thrown if the {@link ParserType} is not
supported.
+ * @throws IllegalArgumentException Thrown if {@code model} is {@code null}.
*
* @see ParserModel
* @see POSTaggerME
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/AbstractParserModelTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/AbstractParserModelTest.java
index 67296aacc..75de12a97 100644
---
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/AbstractParserModelTest.java
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/AbstractParserModelTest.java
@@ -26,9 +26,16 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.MethodSource;
import org.junit.jupiter.params.provider.ValueSource;
+import opennlp.tools.chunker.Chunker;
+import opennlp.tools.chunker.ChunkerME;
+import opennlp.tools.parser.ParserTestUtil.CountingChunker;
+import opennlp.tools.parser.ParserTestUtil.CountingTagger;
+import opennlp.tools.postag.POSTagger;
+import opennlp.tools.postag.POSTaggerME;
import opennlp.tools.tokenize.WhitespaceTokenizer;
import opennlp.tools.util.Span;
@@ -42,6 +49,110 @@ public abstract class AbstractParserModelTest {
*/
protected abstract ParserModel getModel();
+ /**
+ * Creates the parser under test through its constructor that takes a
caller-supplied
+ * tagger and chunker.
+ *
+ * @param model The {@link ParserModel} to take the build, check and head
rule artifacts from.
+ * @param tagger The {@link POSTagger} the parser shall use.
+ * @param chunker The {@link Chunker} the parser shall use.
+ * @param beamSize The number of different parses kept during parsing.
+ * @param advancePercentage The minimal amount of probability mass advanced
outcomes must represent.
+ * @return A parser that tags with {@code tagger} and chunks with {@code
chunker}.
+ */
+ protected abstract Parser createParser(ParserModel model, POSTagger tagger,
Chunker chunker,
+ int beamSize, double
advancePercentage);
+
+ /**
+ * Verifies that a caller-supplied tagger and chunker drive the parse and
that the
+ * model's own tagger and chunker are never constructed.
+ */
+ @Test
+ void testSuppliedTaggerAndChunkerAreUsed() {
+ CountingTagger tagger = new CountingTagger(new
POSTaggerME(getModel().getParserTaggerModel()));
+ CountingChunker chunker = new CountingChunker(new
ChunkerME(getModel().getParserChunkerModel()));
+ ParserModel guarded =
ParserTestUtil.withInaccessibleComponentModels(getModel());
+
+ Parser parser = createParser(guarded, tagger, chunker,
+ AbstractBottomUpParser.defaultBeamSize,
AbstractBottomUpParser.defaultAdvancePercentage);
+ Parse parsed = parser.parse(ParserTestUtil.createTestSentence());
+
+ Assertions.assertTrue(tagger.calls() > 0, "the supplied tagger was not
used");
+ Assertions.assertTrue(chunker.calls() > 0, "the supplied chunker was not
used");
+ Parse reference =
ParserFactory.create(getModel()).parse(ParserTestUtil.createTestSentence());
+ Assertions.assertEquals(reference.toStringPennTreebank(),
parsed.toStringPennTreebank());
+ }
+
+ /**
+ * Verifies that the model's own tagger and chunker, passed explicitly with
the default
+ * beam size and advance percentage, give the same parse as the model-only
constructor.
+ */
+ @Test
+ void testSuppliedComponentsMatchModelConstructor() {
+ Parser parser = createParser(getModel(), new
POSTaggerME(getModel().getParserTaggerModel()),
+ new ChunkerME(getModel().getParserChunkerModel()),
+ AbstractBottomUpParser.defaultBeamSize,
AbstractBottomUpParser.defaultAdvancePercentage);
+ Parse[] parses = parser.parse(ParserTestUtil.createTestSentence(), 2);
+ Parse[] reference =
ParserFactory.create(getModel()).parse(ParserTestUtil.createTestSentence(), 2);
+ ParserTestUtil.assertSameParses(reference, parses);
+ }
+
+ /**
+ * Verifies that the constructor with a caller-supplied tagger and chunker
rejects a
+ * {@code null} model, tagger or chunker.
+ *
+ * @param nullArgument The argument passed as {@code null}.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"model", "tagger", "chunker"})
+ void testConstructorRejectsNullArgument(String nullArgument) {
+ ParserModel model = "model".equals(nullArgument) ? null : getModel();
+ POSTagger tagger = "tagger".equals(nullArgument) ? null
+ : new POSTaggerME(getModel().getParserTaggerModel());
+ Chunker chunker = "chunker".equals(nullArgument) ? null
+ : new ChunkerME(getModel().getParserChunkerModel());
+ IllegalArgumentException e =
Assertions.assertThrows(IllegalArgumentException.class,
+ () -> createParser(model, tagger, chunker,
AbstractBottomUpParser.defaultBeamSize,
+ AbstractBottomUpParser.defaultAdvancePercentage));
+ Assertions.assertEquals(nullArgument + " must not be null",
e.getMessage());
+ }
+
+ /**
+ * Verifies that the constructor rejects a beam size or an advance
percentage outside its
+ * range. The ranges themselves are tested in {@code ParamChecksTest}.
+ *
+ * @param beamSize The beam size to pass.
+ * @param advancePercentage The advance percentage to pass.
+ * @param expected The expected exception message.
+ */
+ @ParameterizedTest
+ @CsvSource({
+ "0, 0.95, beamSize must be positive",
+ "20, 0.0, 'advancePercentage must be in (0, 1]'"
+ })
+ void testConstructorRejectsOutOfRangeArgument(int beamSize, double
advancePercentage,
+ String expected) {
+ POSTagger tagger = new POSTaggerME(getModel().getParserTaggerModel());
+ Chunker chunker = new ChunkerME(getModel().getParserChunkerModel());
+ IllegalArgumentException e =
Assertions.assertThrows(IllegalArgumentException.class,
+ () -> createParser(getModel(), tagger, chunker, beamSize,
advancePercentage));
+ Assertions.assertEquals(expected, e.getMessage());
+ }
+
+ /**
+ * Verifies that the constructor accepts the boundaries of both ranges and
parses with them.
+ *
+ * @param beamSize The beam size to pass.
+ * @param advancePercentage The advance percentage to pass.
+ */
+ @ParameterizedTest
+ @CsvSource({"1, 1.0", "1, 0.01", "20, 1.0"})
+ void testConstructorAcceptsRangeBoundaries(int beamSize, double
advancePercentage) {
+ Parser parser = createParser(getModel(), new
POSTaggerME(getModel().getParserTaggerModel()),
+ new ChunkerME(getModel().getParserChunkerModel()), beamSize,
advancePercentage);
+
Assertions.assertNotNull(parser.parse(ParserTestUtil.createTestSentence()));
+ }
+
/**
* Verifies that serialization of {@link ParserModel} equals trained state.
* <p>
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/ParserFactoryTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/ParserFactoryTest.java
new file mode 100644
index 000000000..4ed9f75e4
--- /dev/null
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/ParserFactoryTest.java
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parser;
+
+import java.io.IOException;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.chunker.Chunker;
+import opennlp.tools.chunker.ChunkerME;
+import opennlp.tools.parser.ParserTestUtil.CountingChunker;
+import opennlp.tools.parser.ParserTestUtil.CountingTagger;
+import opennlp.tools.postag.POSTagger;
+import opennlp.tools.postag.POSTaggerME;
+import opennlp.tools.util.TrainingParameters;
+
+public class ParserFactoryTest {
+
+ /* Trained dynamically before test */
+ private static ParserModel chunkingModel;
+ private static ParserModel treeInsertModel;
+
+ @BeforeAll
+ public static void setupEnvironment() throws IOException {
+ HeadRules headRules = ParserTestUtil.createTestHeadRules();
+ chunkingModel = opennlp.tools.parser.chunking.Parser.train("eng",
+ ParserTestUtil.openTestTrainingData(), headRules,
TrainingParameters.defaultParams());
+ treeInsertModel = opennlp.tools.parser.treeinsert.Parser.train("eng",
+ ParserTestUtil.openTestTrainingData(), headRules,
TrainingParameters.defaultParams());
+ Assertions.assertNotNull(chunkingModel);
+ Assertions.assertNotNull(treeInsertModel);
+ }
+
+ private static Stream<Arguments> provideModels() {
+ return Stream.of(
+ Arguments.of(chunkingModel,
opennlp.tools.parser.chunking.Parser.class),
+ Arguments.of(treeInsertModel,
opennlp.tools.parser.treeinsert.Parser.class));
+ }
+
+ /**
+ * Verifies that the model's own tagger and chunker, passed to the
five-argument overload with
+ * the default beam size and advance percentage, give the parses of the
model-only factory method.
+ *
+ * @param model The model of the parser type under test.
+ * @param expectedType The parser class of that type.
+ */
+ @ParameterizedTest(name = "{1}")
+ @MethodSource("provideModels")
+ void testCreateWithTheModelComponentsMatchesTheModelOnlyFactory(ParserModel
model,
+ Class<?>
expectedType) {
+ Parser parser = ParserFactory.create(model, new
POSTaggerME(model.getParserTaggerModel()),
+ new ChunkerME(model.getParserChunkerModel()),
AbstractBottomUpParser.defaultBeamSize,
+ AbstractBottomUpParser.defaultAdvancePercentage);
+ Assertions.assertInstanceOf(expectedType, parser);
+
+ Parse[] parses = parser.parse(ParserTestUtil.createTestSentence(), 2);
+ Parse[] reference =
ParserFactory.create(model).parse(ParserTestUtil.createTestSentence(), 2);
+ ParserTestUtil.assertSameParses(reference, parses);
+ }
+
+ /**
+ * Verifies that the five-argument overload parses with the supplied tagger
and chunker and
+ * never reads the model's tagger and chunker models.
+ *
+ * @param model The model of the parser type under test.
+ * @param expectedType The parser class of that type.
+ */
+ @ParameterizedTest(name = "{1}")
+ @MethodSource("provideModels")
+ void testCreateUsesSuppliedTaggerAndChunker(ParserModel model, Class<?>
expectedType) {
+ CountingTagger tagger = new CountingTagger(new
POSTaggerME(model.getParserTaggerModel()));
+ CountingChunker chunker = new CountingChunker(new
ChunkerME(model.getParserChunkerModel()));
+
+ Parser parser =
ParserFactory.create(ParserTestUtil.withInaccessibleComponentModels(model),
+ tagger, chunker, AbstractBottomUpParser.defaultBeamSize,
+ AbstractBottomUpParser.defaultAdvancePercentage);
+ Assertions.assertInstanceOf(expectedType, parser);
+ Parse parsed = parser.parse(ParserTestUtil.createTestSentence());
+
+ Assertions.assertTrue(tagger.calls() > 0, "the supplied tagger was not
used");
+ Assertions.assertTrue(chunker.calls() > 0, "the supplied chunker was not
used");
+ Parse reference =
ParserFactory.create(model).parse(ParserTestUtil.createTestSentence());
+ Assertions.assertEquals(reference.toStringPennTreebank(),
parsed.toStringPennTreebank());
+ }
+
+ /**
+ * Verifies that the five-argument overload applies the beam size and
advance percentage.
+ *
+ * @param model The model of the parser type under test.
+ * @param expectedType The parser class of that type.
+ */
+ @ParameterizedTest(name = "{1}")
+ @MethodSource("provideModels")
+ void testCreateAppliesBeamSizeAndAdvancePercentage(ParserModel model,
Class<?> expectedType) {
+ Parser parser = ParserFactory.create(model, new
POSTaggerME(model.getParserTaggerModel()),
+ new ChunkerME(model.getParserChunkerModel()), 1, 1.0);
+ Assertions.assertInstanceOf(expectedType, parser);
+
+ Parse[] parses = parser.parse(ParserTestUtil.createTestSentence(), 3);
+ Parse[] reference = ParserFactory.create(model, 1,
1.0).parse(ParserTestUtil.createTestSentence(), 3);
+ Parse[] wide =
ParserFactory.create(model).parse(ParserTestUtil.createTestSentence(), 3);
+ ParserTestUtil.assertSameParses(reference, parses);
+ Assertions.assertTrue(parses.length < wide.length, "a beam of one must
prune parses");
+ }
+
+ /**
+ * Verifies that the five-argument overload rejects a {@code null} model,
tagger or chunker.
+ *
+ * @param nullArgument The argument passed as {@code null}.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"model", "tagger", "chunker"})
+ void testCreateWithComponentsRejectsNullArgument(String nullArgument) {
+ ParserModel model = "model".equals(nullArgument) ? null : chunkingModel;
+ POSTagger tagger = "tagger".equals(nullArgument) ? null
+ : new POSTaggerME(chunkingModel.getParserTaggerModel());
+ Chunker chunker = "chunker".equals(nullArgument) ? null
+ : new ChunkerME(chunkingModel.getParserChunkerModel());
+ IllegalArgumentException e =
Assertions.assertThrows(IllegalArgumentException.class,
+ () -> ParserFactory.create(model, tagger, chunker,
AbstractBottomUpParser.defaultBeamSize,
+ AbstractBottomUpParser.defaultAdvancePercentage));
+ Assertions.assertEquals(nullArgument + " must not be null",
e.getMessage());
+ }
+
+ /**
+ * Verifies that the five-argument overload rejects a beam size or an
advance percentage
+ * outside its range. The ranges themselves are tested in {@code
ParamChecksTest}.
+ *
+ * @param beamSize The beam size to pass.
+ * @param advancePercentage The advance percentage to pass.
+ * @param expected The expected exception message.
+ */
+ @ParameterizedTest
+ @CsvSource({
+ "0, 0.95, beamSize must be positive",
+ "20, 0.0, 'advancePercentage must be in (0, 1]'"
+ })
+ void testCreateWithComponentsRejectsOutOfRangeArgument(int beamSize, double
advancePercentage,
+ String expected) {
+ POSTagger tagger = new POSTaggerME(chunkingModel.getParserTaggerModel());
+ Chunker chunker = new ChunkerME(chunkingModel.getParserChunkerModel());
+ IllegalArgumentException e =
Assertions.assertThrows(IllegalArgumentException.class,
+ () -> ParserFactory.create(chunkingModel, tagger, chunker, beamSize,
advancePercentage));
+ Assertions.assertEquals(expected, e.getMessage());
+ }
+
+ /**
+ * Verifies that {@link ParserFactory#create(ParserModel, int, double)}
rejects a
+ * {@code null} model, a beam size or an advance percentage outside its
range.
+ *
+ * @param nullModel Whether the model is passed as {@code null}.
+ * @param beamSize The beam size to pass.
+ * @param advancePercentage The advance percentage to pass.
+ * @param expected The expected exception message.
+ */
+ @ParameterizedTest
+ @CsvSource({
+ "true, 20, 0.95, model must not be null",
+ "false, 0, 0.95, beamSize must be positive",
+ "false, 20, 0.0, 'advancePercentage must be in (0, 1]'"
+ })
+ void testCreateRejectsInvalidArgument(boolean nullModel, int beamSize,
double advancePercentage,
+ String expected) {
+ ParserModel model = nullModel ? null : chunkingModel;
+ IllegalArgumentException e =
Assertions.assertThrows(IllegalArgumentException.class,
+ () -> ParserFactory.create(model, beamSize, advancePercentage));
+ Assertions.assertEquals(expected, e.getMessage());
+ }
+
+ /**
+ * Verifies that the five-argument overload accepts the boundaries of both
ranges and returns a
+ * parser that parses with them.
+ *
+ * @param beamSize The beam size to pass.
+ * @param advancePercentage The advance percentage to pass.
+ */
+ @ParameterizedTest
+ @CsvSource({"1, 1.0", "1, 0.01", "20, 1.0"})
+ void testCreateWithComponentsAcceptsRangeBoundaries(int beamSize, double
advancePercentage) {
+ Parser parser = ParserFactory.create(chunkingModel,
+ new POSTaggerME(chunkingModel.getParserTaggerModel()),
+ new ChunkerME(chunkingModel.getParserChunkerModel()), beamSize,
advancePercentage);
+
Assertions.assertNotNull(parser.parse(ParserTestUtil.createTestSentence()));
+ }
+}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/ParserTestUtil.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/ParserTestUtil.java
index df5039be5..2ca73fa11 100644
---
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/ParserTestUtil.java
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/ParserTestUtil.java
@@ -27,14 +27,45 @@ import java.nio.charset.StandardCharsets;
import org.junit.jupiter.api.Assertions;
+import opennlp.tools.chunker.Chunker;
+import opennlp.tools.chunker.ChunkerModel;
import opennlp.tools.formats.ResourceAsStreamFactory;
import opennlp.tools.parser.lang.en.HeadRules;
+import opennlp.tools.postag.POSModel;
+import opennlp.tools.postag.POSTagger;
+import opennlp.tools.tokenize.WhitespaceTokenizer;
import opennlp.tools.util.InputStreamFactory;
import opennlp.tools.util.ObjectStream;
import opennlp.tools.util.PlainTextByLineStream;
+import opennlp.tools.util.Sequence;
+import opennlp.tools.util.Span;
public class ParserTestUtil {
+ private static final String TEST_SENTENCE = "Eric is testing.";
+
+ /**
+ * @return A fresh, unparsed {@link Parse} over the tokens of a short
English test sentence.
+ */
+ public static Parse createTestSentence() {
+ return
Parse.createFromTokens(WhitespaceTokenizer.INSTANCE.tokenize(TEST_SENTENCE));
+ }
+
+ /**
+ * Asserts that {@code actual} holds the same bracketings with the same
probabilities
+ * as {@code expected}, in the same order.
+ *
+ * @param expected The reference parses.
+ * @param actual The parses under test.
+ */
+ public static void assertSameParses(Parse[] expected, Parse[] actual) {
+ Assertions.assertEquals(expected.length, actual.length);
+ for (int i = 0; i < expected.length; i++) {
+ Assertions.assertEquals(expected[i].toStringPennTreebank(),
actual[i].toStringPennTreebank());
+ Assertions.assertEquals(expected[i].getProb(), actual[i].getProb());
+ }
+ }
+
public static HeadRules createTestHeadRules() throws IOException {
try (InputStream headRulesIn = ParserTestUtil.class.getResourceAsStream(
"/opennlp/tools/parser/en_head_rules");
@@ -82,4 +113,117 @@ public class ParserTestUtil {
return resetableSampleStream;
}
+
+ /**
+ * Returns a copy of {@code model} whose {@link
ParserModel#getParserTaggerModel()} and
+ * {@link ParserModel#getParserChunkerModel()} fail, so a test can prove
that a parser
+ * built from it never constructs the model's own tagger or chunker.
+ *
+ * @param model The trained model to copy the build, check, attach and head
rule artifacts from.
+ * @return A model that fails on access to its tagger or chunker model.
+ */
+ public static ParserModel withInaccessibleComponentModels(ParserModel model)
{
+ return new ParserModel(model.getLanguage(), model.getBuildModel(),
model.getCheckModel(),
+ model.getAttachModel(), model.getParserTaggerModel(),
model.getParserChunkerModel(),
+ model.getHeadRules(), model.getParserType()) {
+
+ @Override
+ public POSModel getParserTaggerModel() {
+ throw new AssertionError("the model's tagger model must not be
accessed");
+ }
+
+ @Override
+ public ChunkerModel getParserChunkerModel() {
+ throw new AssertionError("the model's chunker model must not be
accessed");
+ }
+ };
+ }
+
+ /**
+ * A {@link POSTagger} that delegates every call and counts them.
+ */
+ public static final class CountingTagger implements POSTagger {
+
+ private final POSTagger delegate;
+ private int calls;
+
+ public CountingTagger(POSTagger delegate) {
+ this.delegate = delegate;
+ }
+
+ /**
+ * @return The number of calls made to this tagger so far.
+ */
+ public int calls() {
+ return calls;
+ }
+
+ @Override
+ public String[] tag(String[] sentence) {
+ calls++;
+ return delegate.tag(sentence);
+ }
+
+ @Override
+ public String[] tag(String[] sentence, Object[] additionalContext) {
+ calls++;
+ return delegate.tag(sentence, additionalContext);
+ }
+
+ @Override
+ public Sequence[] topKSequences(String[] sentence) {
+ calls++;
+ return delegate.topKSequences(sentence);
+ }
+
+ @Override
+ public Sequence[] topKSequences(String[] sentence, Object[]
additionalContext) {
+ calls++;
+ return delegate.topKSequences(sentence, additionalContext);
+ }
+ }
+
+ /**
+ * A {@link Chunker} that delegates every call and counts them.
+ */
+ public static final class CountingChunker implements Chunker {
+
+ private final Chunker delegate;
+ private int calls;
+
+ public CountingChunker(Chunker delegate) {
+ this.delegate = delegate;
+ }
+
+ /**
+ * @return The number of calls made to this chunker so far.
+ */
+ public int calls() {
+ return calls;
+ }
+
+ @Override
+ public String[] chunk(String[] toks, String[] tags) {
+ calls++;
+ return delegate.chunk(toks, tags);
+ }
+
+ @Override
+ public Span[] chunkAsSpans(String[] toks, String[] tags) {
+ calls++;
+ return delegate.chunkAsSpans(toks, tags);
+ }
+
+ @Override
+ public Sequence[] topKSequences(String[] sentence, String[] tags) {
+ calls++;
+ return delegate.topKSequences(sentence, tags);
+ }
+
+ @Override
+ public Sequence[] topKSequences(String[] sentence, String[] tags, double
minSequenceScore) {
+ calls++;
+ return delegate.topKSequences(sentence, tags, minSequenceScore);
+ }
+ }
}
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/chunking/ParserTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/chunking/ParserTest.java
index 37ae93e03..9bc271414 100644
---
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/chunking/ParserTest.java
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/chunking/ParserTest.java
@@ -22,11 +22,13 @@ import java.io.IOException;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
+import opennlp.tools.chunker.Chunker;
import opennlp.tools.parser.AbstractParserModelTest;
import opennlp.tools.parser.HeadRules;
import opennlp.tools.parser.Parse;
import opennlp.tools.parser.ParserModel;
import opennlp.tools.parser.ParserTestUtil;
+import opennlp.tools.postag.POSTagger;
import opennlp.tools.util.ObjectStream;
import opennlp.tools.util.TrainingParameters;
@@ -43,6 +45,12 @@ public class ParserTest extends AbstractParserModelTest {
return model;
}
+ @Override
+ protected Parser createParser(ParserModel model, POSTagger tagger, Chunker
chunker,
+ int beamSize, double advancePercentage) {
+ return new Parser(model, tagger, chunker, beamSize, advancePercentage);
+ }
+
@BeforeAll
public static void setupEnvironment() throws IOException {
ObjectStream<Parse> parseSamples = ParserTestUtil.openTestTrainingData();
diff --git
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/treeinsert/ParserTest.java
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/treeinsert/ParserTest.java
index ff5c16c4d..d732898ee 100644
---
a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/treeinsert/ParserTest.java
+++
b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/parser/treeinsert/ParserTest.java
@@ -22,11 +22,13 @@ import java.io.IOException;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
+import opennlp.tools.chunker.Chunker;
import opennlp.tools.parser.AbstractParserModelTest;
import opennlp.tools.parser.HeadRules;
import opennlp.tools.parser.Parse;
import opennlp.tools.parser.ParserModel;
import opennlp.tools.parser.ParserTestUtil;
+import opennlp.tools.postag.POSTagger;
import opennlp.tools.util.ObjectStream;
import opennlp.tools.util.TrainingParameters;
@@ -42,6 +44,12 @@ public class ParserTest extends AbstractParserModelTest {
protected ParserModel getModel() {
return model;
}
+
+ @Override
+ protected Parser createParser(ParserModel model, POSTagger tagger, Chunker
chunker,
+ int beamSize, double advancePercentage) {
+ return new Parser(model, tagger, chunker, beamSize, advancePercentage);
+ }
@BeforeAll
public static void setupEnvironment() throws IOException {
diff --git a/opennlp-docs/src/docbkx/parser.xml
b/opennlp-docs/src/docbkx/parser.xml
index c4b48fe0d..d1ae2600a 100644
--- a/opennlp-docs/src/docbkx/parser.xml
+++ b/opennlp-docs/src/docbkx/parser.xml
@@ -111,6 +111,17 @@ try {
an instance of the corresponding parser implementation.
<programlisting language="java">
<![CDATA[Parser parser = ParserFactory.create(model);]]>
+ </programlisting>
+ The parser can tag and chunk with components the
application supplies instead
+ of the ones stored in the model, for example a POS
tagger trained on domain
+ text. This factory method takes the build, attach and
check models and the head
+ rules from the parser model and does not construct the
model's own tagger or
+ chunker. To replace only one of them, pass the model's
component for the other.
+ <programlisting language="java">
+<![CDATA[POSTagger tagger = new POSTaggerME(customPosModel);
+Chunker chunker = new ChunkerME(model.getParserChunkerModel());
+Parser parser = ParserFactory.create(model, tagger, chunker,
+ AbstractBottomUpParser.defaultBeamSize,
AbstractBottomUpParser.defaultAdvancePercentage);]]>
</programlisting>
Right now the tree insert parser is still experimental
and there is no pre-trained model for it.
The parser expect a whitespace tokenized sentence. A
utility method from the command
