This is an automated email from the ASF dual-hosted git repository.
mawiesne pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new ef814bdba GH-1454: Tidy document, stemmer, corpora and doccat chapters
and README (#1455)
ef814bdba is described below
commit ef814bdbaf126e98e441cf6894f1f1bbfd4edc71
Author: Richard Zowalla <[email protected]>
AuthorDate: Fri Oct 9 10:16:49 2026 +0200
GH-1454: Tidy document, stemmer, corpora and doccat chapters and README
(#1455)
Remove test-class references, design rationale and implementation details
from the document, stemmer, corpora and doccat chapters, and add the
term-vector layer to the layer table. Extend the README breaking-change
list with the whitespace default, CoNLL-U, PlainTextByLineStream and
RealBasicEventStream changes.
---
README.md | 8 ++++++-
opennlp-docs/src/docbkx/corpora.xml | 6 ++----
opennlp-docs/src/docbkx/doccat.xml | 1 -
opennlp-docs/src/docbkx/document.xml | 42 +++++++++---------------------------
opennlp-docs/src/docbkx/stemmer.xml | 32 +++++++--------------------
5 files changed, 27 insertions(+), 62 deletions(-)
diff --git a/README.md b/README.md
index f3af72796..1ba0d1acc 100644
--- a/README.md
+++ b/README.md
@@ -120,7 +120,7 @@ For more details please check our
[documentation](https://opennlp.apache.org/doc
## Migrating from 2.x to 3.x
-The 3.x release line of Apache OpenNLP introduces only a few breaking changes,
listed below, but modularizes the project for better usage as a library and to
support future extensibility.
+The 3.x release line of Apache OpenNLP modularizes the project for better
usage as a library and to support future extensibility.
The core API remains stable and compatible with 2.x, but the project structure
has been reorganized into multiple modules.
That means, that you can continue to use the previous `opennlp-tools` artifact
as a dependency. However, we strongly recommend to switch to the new modular
structure
@@ -131,10 +131,16 @@ For users of the traditional CLI toolkit, nothing changes
with the 3.x release l
### Breaking changes
+The list below covers the changes most likely to affect existing code. It is
not exhaustive; the release notes and the [issue
tracker](https://github.com/apache/opennlp/issues) record all changes.
+
+- Whitespace: tokenization, corpus format parsing and feature generation
classify whitespace by the Unicode `White_Space` property. Set
`-Dopennlp.whitespace.mode=LEGACY` to reproduce 2.x tokenization exactly, for
example with models trained by 1.x or 2.x.
- `AbstractClassPathModelFinder`: the protected `asRegex` and `matchesPattern`
methods were removed. Custom model finders call `matchesWildcard(url, "*.bin")`
instead ([OPENNLP-1932](https://issues.apache.org/jira/browse/OPENNLP-1932)).
- `BasicContextGenerator(String)`: the separator is taken literally instead of
as a regular expression, and a separator containing a backslash throws
`IllegalArgumentException`. Empty predicates are no longer returned, and the
default constructor splits on runs of Unicode whitespace
([OPENNLP-1929](https://issues.apache.org/jira/browse/OPENNLP-1929)).
- AD reader: a self-closing markup line such as `<s/>`, `<s />` or `<s
id="1"/>` no longer opens a sentence, paragraph, title, box or text. A tag
whose name only starts with `s`, `p`, `t`, `caixa` or `ext`, such as `<sx>`, is
no longer read as that tag, and attributes must be separated from the tag name
by Unicode whitespace
([GH-1447](https://github.com/apache/opennlp/issues/1447)).
- MASC reader: region offsets must be written with ASCII digits only and no
sign. A document with a signed or non-ASCII offset is skipped
([OPENNLP-1990](https://issues.apache.org/jira/browse/OPENNLP-1990)).
+- CoNLL-U reader: a multiword token range whose word lines are missing from
the sentence throws `InvalidFormatException`.
+- `PlainTextByLineStream.reset()` closes the previous reader, so its
`InputStreamFactory` must return a new stream for each pass.
+- `RealBasicEventStream`: a line that contains only an outcome is read as an
event without contexts and no longer ends the stream.
### Thread safety
diff --git a/opennlp-docs/src/docbkx/corpora.xml
b/opennlp-docs/src/docbkx/corpora.xml
index a1b8bc4bd..ce69efc0a 100644
--- a/opennlp-docs/src/docbkx/corpora.xml
+++ b/opennlp-docs/src/docbkx/corpora.xml
@@ -418,10 +418,8 @@ Execution time: 1,955 seconds]]>
in the same sentence. For example, a range
<code>1-3</code> requires lines
<code>1</code>, <code>2</code>, and <code>3</code>.
Since 3.0.0, a missing word
line stops reading with
<code>InvalidFormatException</code> naming the range
- and missing id, rather than a
<code>NullPointerException</code>. The reader checks
- for missing words as it traverses the range, without
first allocating all of its
- ids. Ranges ending at the largest integer do not wrap
around. Correct the
- input before resuming conversion or training; missing
words are not fabricated.
+ and missing id. Correct the input before resuming
conversion or training;
+ missing words are not fabricated.
</para>
</section>
<section xml:id="tools.corpora.arvores-deitadas">
diff --git a/opennlp-docs/src/docbkx/doccat.xml
b/opennlp-docs/src/docbkx/doccat.xml
index 0207df2bc..0150b3527 100644
--- a/opennlp-docs/src/docbkx/doccat.xml
+++ b/opennlp-docs/src/docbkx/doccat.xml
@@ -170,7 +170,6 @@ final DocumentCategorizerDL myCategorizer = new
DocumentCategorizerDL(
double[] outcomes = myCategorizer.categorize(inputText);
String category = myCategorizer.getBestCategory(outcomes);]]>
</programlisting>
- For additional examples, refer to the
<code>DocumentCategorizerDLEval</code> class.
</para>
<para>
The vocabulary file is either plain text with
one token per line, as in
diff --git a/opennlp-docs/src/docbkx/document.xml
b/opennlp-docs/src/docbkx/document.xml
index c508ed46d..4bbe9cd5b 100644
--- a/opennlp-docs/src/docbkx/document.xml
+++ b/opennlp-docs/src/docbkx/document.xml
@@ -100,25 +100,6 @@ String language = tagged.get(LANGUAGE).get(0).value(); //
"eng", span is null]]>
since adding a layer is once-only, competing versions
always live under
distinct keys and never replace each other.
</para>
- <para>
- Each rule above is pinned by a test in
<code>DocumentContractTest</code>:
-
<code>testKeysWithSameIdButDifferentTypesAreDifferentLayers</code>,
- <code>testToolkitKeysCarryTheNamespacePrefix</code>,
-
<code>testDocumentScopedLayersCarrySpanlessValues</code>,
- <code>testLayerPreservesInsertionOrder</code>,
-
<code>testDuplicateLayerIsRejectedWithExactMessage</code>,
- <code>testGoldLayerLivesBesideThePredictedLayer</code>,
and
- <code>testCustomLayerNeedsNoContainerChange</code>.
- </para>
- <para>
- None of this is new architecture, and that is intended.
Standoff annotations
- over an unchanged original text are how the UIMA CAS
works; typed keys that let
- any module add a layer without the container knowing
about it are the pattern of
- Stanford CoreNLP's annotation map; and annotators that
declare what they require
- and provide, checked when the pipeline is assembled,
follow spaCy's pipeline
- analysis. The container takes the lightweight end of
each: string ids with a
- namespace instead of class identity, and no type system
descriptors.
- </para>
</section>
<section xml:id="tools.document.layers">
@@ -188,6 +169,12 @@ String language = tagged.get(LANGUAGE).get(0).value(); //
"eng", span is null]]>
<entry>constituent</entry>
<entry><code>ParserAnnotator</code></entry>
</row>
+ <row>
+
<entry><code>opennlp:term-vectors</code></entry>
+ <entry><code>TermVector</code>:
the term, its frequency and, except in <code>SCORING_ONLY</code> mode, its
occurrence spans</entry>
+ <entry>distinct term;
document-scoped, without a span</entry>
+
<entry><code>TermVectorAnnotator</code></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -200,8 +187,7 @@ String language = tagged.get(LANGUAGE).get(0).value(); //
"eng", span is null]]>
puts one annotation on each mention's span with the
chain it belongs to, and a
dependency layer puts one annotation on each token's
span naming the head token
by its index in <code>opennlp:tokens</code>. Adding
such a layer requires no change
- in this package;
<code>testCustomLayerNeedsNoContainerChange</code> pins that,
- and the section below shows how it is done.
+ in this package; the section below shows how it is done.
</para>
</section>
@@ -239,11 +225,6 @@ DocumentAnalyzer analyzer = DocumentAnalyzer.builder()
Document document = analyzer.analyze("The dog barks. It naps.");]]>
</programlisting>
- <para>
- <code>DocumentPipelineExampleTest</code> asserts the
pipeline and layer
- round-trip shown here;
<code>DocumentAnalyzerTest</code> covers the build-time
- checks, for example
<code>testMisorderedPipelineFailsAtBuildTime</code>.
- </para>
<para>
The resulting document carries exactly the four layers
the pipeline provides:
the figure's three plus the custom token-lengths layer.
The sentence
@@ -352,8 +333,6 @@ int firstTokenLength = lengths.get(0).value(); // 3, for
"The" at [0..3)]]>
only by a normalization fold group together while their
spans keep pointing into
the original text. A token whose normalized form is
empty is omitted from the
layer. The layer preserves first-occurrence order.
-
<code>TermVectorPipelineTest#testTokenizerAndTermVectorPipeline</code> asserts
the
- behavior shown here.
<programlisting language="java"><![CDATA[
DocumentAnalyzer analyzer = DocumentAnalyzer.builder()
.add(new TokenizerAnnotator(tokenizer))
@@ -371,8 +350,6 @@ List<Annotation<TermVector>> vectors =
document.get(TermVectorAnnotator.TERM_VEC
normalizer is applied to each token's covered text to
produce the term, so the
case, NFC, and accent folds that
<code>buildAligned()</code> rejects all work
here, and every occurrence span stays the token's own
span in the original text.
-
<code>TermVectorNormalizedExampleTest#testCaseFoldedTermsKeepOriginalSpans</code>
- asserts the behavior shown here.
<programlisting language="java"><![CDATA[
CharSequenceNormalizer folder = TextNormalizer.builder().caseFold().build();
DocumentAnalyzer analyzer = DocumentAnalyzer.builder()
@@ -388,8 +365,9 @@ List<Annotation<TermVector>> vectors =
document.get(TermVectorAnnotator.TERM_VEC
<para>
Scoring-only consumers that never read offsets can skip
storing them:
<code>new
TermVectorAnnotator(TermVectorAnnotator.Mode.SCORING_ONLY)</code> emits
- vectors that carry counts but no spans, and
- <code>TermVector.count("dog", 2)</code> builds the
matching expectation in tests.
+ vectors that carry counts but no spans.
<code>TermVector.count(term, frequency)</code>
+ creates such a span-free vector;
<code>TermVector.withSpans(term, spans)</code>
+ creates a full one.
</para>
</section>
</chapter>
diff --git a/opennlp-docs/src/docbkx/stemmer.xml
b/opennlp-docs/src/docbkx/stemmer.xml
index 1f8f41fa4..407331f96 100644
--- a/opennlp-docs/src/docbkx/stemmer.xml
+++ b/opennlp-docs/src/docbkx/stemmer.xml
@@ -35,8 +35,6 @@
configured <code>Stemmer</code> instances on demand. It
is a plain supplier
of stemmers, not a <code>BaseToolFactory</code> loaded
from a model
manifest. The English Snowball factory needs no
external data:
- <code>StemmerFactoryUsageExampleTest</code> asserts the
behavior shown
- here.
<programlisting language="java"><![CDATA[
StemmerFactory factory = new
SnowballStemmerFactory(SnowballStemmer.ALGORITHM.ENGLISH);
Stemmer stemmer = factory.newStemmer();
@@ -80,7 +78,6 @@ new CachingStemmer(factory).stem("running"); // "run"]]>
license text with downloaded files. OpenNLP's Apache
License does not
relicense those files. The dictionary is immutable and
safe to share;
<code>HunspellStemmerFactory</code> creates a fresh
stemmer per call.
- <code>HunspellManualExampleTest</code> asserts the
behavior shown here.
<programlisting language="java"><![CDATA[
HunspellDictionary dictionary = HunspellDictionary.load(
Path.of("dictionary.aff"), Path.of("dictionary.dic"));
@@ -89,12 +86,9 @@ Stemmer stemmer = new
HunspellStemmerFactory(dictionary).newStemmer();
stemmer.stem("workers"); // "work"
stemmer.stem("table"); // "table" (unknown vocabulary is unchanged)]]>
</programlisting>
- The stems above are those of the project-authored
miniature dictionary the
- test loads, which lists <code>work</code> with an
agentive and a plural
- suffix; the test asserts the same stem for
<code>worker</code>. Which stem a
- published dictionary yields for a given form is decided
by that dictionary.
- Acquisition helpers and the supported affix feature set
live in
- <code>dev/README-hunspell-dictionaries.md</code>.
+ The stems above assume a dictionary that lists
<code>work</code> with an
+ agentive and a plural suffix. Which stem a published
dictionary yields for a
+ given form is decided by that dictionary.
An opt-in catalog download accepts an
application-supplied
<code>DictionaryCatalog</code>, needs
<code>-Dopennlp.download.remote=true</code>, and
fetches through the
@@ -146,9 +140,7 @@ for (HunspellDictionary.UnsupportedDirective diagnostic :
partial.getUnsupported
the spell checker for recognition and the analyzer for
stems and fields
wherever the two agree, and resolves each disagreement
the same way for
every caller. The results deviate from the native
library in the
- following cases. <code>HunspellCompatibilityEval</code>
in
- <code>opennlp-eval-tests</code> records each expected
difference together
- with the Hunspell output.
+ following cases.
</para>
<itemizedlist>
<listitem>
@@ -225,23 +217,15 @@ for (HunspellDictionary.UnsupportedDirective diagnostic :
partial.getUnsupported
<code>FLAG</code> and <code>AF</code> declarations
apply to the complete
affix file, including rules listed before those
declarations. Parsing
rejects malformed text in parsed rules, invalid counts,
numeric flags
- outside the range 1 through 65535, and
<code>COMPOUNDMIN</code> values
- that cannot be doubled safely. Comments and unused
metadata may retain a
- legacy encoding. Default and <code>long</code> flag
modes preserve raw
- one-byte flag values in UTF-8 files. Compound length
and boundary checks
+ outside the range 1 through 65535 and out-of-range
<code>COMPOUNDMIN</code>
+ values. Compound length and boundary checks
count Unicode code points.
A rule that strips a whole stem applies only when the
affix file
declares <code>FULLSTRIP</code>, as in Hunspell itself.
Each affix or dictionary stream is rejected when it
exceeds
<code>HunspellDictionary.MAX_STREAM_BYTES</code>
(64 MiB).
- Search permits at most 64 compound components, 2048
compound candidate
- checks per case variant, and 2048 word-break attempts
per input. Sharp-s
- expansion permits at most 64 case variants. Output is
limited to 2048
- distinct results. Compound-rule patterns permit 4096
flag elements.
- Simplified triple letters can be restored at multiple
junctions;
- <code>CHECKCOMPOUNDPATTERN</code> replacement applies
at one junction.
- These limits can exclude valid analyses; candidates
must pass validation
- before inclusion in the result.
+ Compound and word-break search is bounded, so an input
with very many
+ candidate analyses may not yield all of them.
</para>
</section>
</chapter>
