This is an automated email from the ASF dual-hosted git repository.
mawiesne pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 8d2a8c310 GH-1419: Correct and trim opennlp-runtime Javadoc (#1426)
8d2a8c310 is described below
commit 8d2a8c3101262f539f242bfb0e9c17218646c981
Author: Richard Zowalla <[email protected]>
AuthorDate: Fri Oct 9 10:11:26 2026 +0200
GH-1419: Correct and trim opennlp-runtime Javadoc (#1426)
* GH-1419: Correct and trim opennlp-runtime Javadoc
Correct inaccurate stemmer, normalizer and sentence detector Javadoc, drop
design history and issue references, and link cited specifications.
Add missing summaries, @since tags and {@inheritDoc}, and use the "@throws
X Thrown if" phrasing.
EmojiFlags formats its hex code point with Locale.ROOT and
PorterStemmer.stem(CharSequence) gains @Override.
---
opennlp-core/opennlp-runtime/BENCHMARKS.md | 12 +++++++++++
.../java/opennlp/tools/ml/BeamSearchBenchmark.java | 11 ++---------
.../tools/doccat/EmojiFeatureGenerator.java | 5 +++--
.../opennlp/tools/namefind/RegexNameFinder.java | 3 ---
.../tools/namefind/RegexNameFinderFactory.java | 3 ---
.../tools/sentdetect/SentenceDetectorME.java | 23 ++++++++--------------
.../sentiment/EmojiSentimentContextGenerator.java | 5 +++--
.../tools/sentiment/EmojiSentimentFactory.java | 2 ++
.../java/opennlp/tools/stemmer/CachingStemmer.java | 9 ++++-----
.../opennlp/tools/stemmer/DelegatingStemmer.java | 16 +++++++++------
.../java/opennlp/tools/stemmer/PorterStemmer.java | 8 ++------
.../java/opennlp/tools/stemmer/SharingStemmer.java | 4 +++-
.../tools/stemmer/snowball/SnowballStemmer.java | 3 ++-
.../stemmer/snowball/SnowballStemmerFactory.java | 6 ++----
.../tools/tokenize/uax29/WordBreakProperty.java | 7 +++----
.../tools/tokenize/uax29/WordSegmenter.java | 10 +++++-----
.../tools/tokenize/uax29/WordTokenizer.java | 14 +++++++------
.../opennlp/tools/tokenize/uax29/WordType.java | 3 ++-
.../java/opennlp/tools/util/DictionaryCatalog.java | 2 ++
.../java/opennlp/tools/util/archive/TarStream.java | 14 ++++++++-----
.../EmojiAnnotationFeatureGenerator.java | 5 +++--
.../EmojiAnnotationFeatureGeneratorFactory.java | 1 +
.../AccentFoldCharSequenceNormalizer.java | 3 ---
.../normalizer/BulletCharSequenceNormalizer.java | 9 +++++----
.../normalizer/DashCharSequenceNormalizer.java | 2 ++
.../normalizer/DigitCharSequenceNormalizer.java | 3 +--
.../opennlp/tools/util/normalizer/Dimension.java | 10 ++++++++--
.../normalizer/EllipsisCharSequenceNormalizer.java | 5 +++--
.../tools/util/normalizer/EmojiAnnotation.java | 19 ++++++++++++------
.../tools/util/normalizer/EmojiAnnotations.java | 8 ++++----
.../tools/util/normalizer/EmojiAnnotator.java | 19 +++++++++---------
.../tools/util/normalizer/EmojiEmoticons.java | 10 +++++-----
.../opennlp/tools/util/normalizer/EmojiFlags.java | 11 ++++++-----
.../FullCaseFoldCharSequenceNormalizer.java | 23 +++++++++++-----------
.../GermanUmlautCharSequenceNormalizer.java | 5 +++--
.../InvisibleCharSequenceNormalizer.java | 11 ++++++-----
...PreservingWhitespaceCharSequenceNormalizer.java | 2 +-
.../util/normalizer/NfcCharSequenceNormalizer.java | 2 +-
.../normalizer/NfkcCharSequenceNormalizer.java | 2 +-
.../util/normalizer/NormalizationProfile.java | 16 +++++++--------
.../util/normalizer/NormalizationProfiles.java | 19 +++++++++---------
...PreservingWhitespaceCharSequenceNormalizer.java | 3 +--
.../normalizer/QuoteCharSequenceNormalizer.java | 3 +--
.../tools/util/normalizer/TextNormalizer.java | 7 ++++---
.../util/normalizer/UrlCharSequenceNormalizer.java | 5 ++++-
.../WhitespaceCharSequenceNormalizer.java | 2 +-
46 files changed, 193 insertions(+), 172 deletions(-)
diff --git a/opennlp-core/opennlp-runtime/BENCHMARKS.md
b/opennlp-core/opennlp-runtime/BENCHMARKS.md
index 280575ee4..17a9de721 100644
--- a/opennlp-core/opennlp-runtime/BENCHMARKS.md
+++ b/opennlp-core/opennlp-runtime/BENCHMARKS.md
@@ -131,6 +131,18 @@ cache, so the multiplier only applies to repeats within
one task;
workloads that stem a handful of words per task should expect
uncached-level throughput there.
+### BeamSearch on long sequences
+
+`BeamSearchBenchmark` (in `opennlp.tools.ml`) measures one
+`bestSequence` call per op on a synthetic sequence of
+`sequenceLength` tokens. It uses only the
+`BeamSearch(int, MaxentModel, int)` constructor and the
+`bestSequence(T[], Object[], BeamSearchContextGenerator, SequenceValidator)`
+method, so the same compiled class runs against other
+`opennlp-ml-commons` builds. To get a baseline, place the
+`opennlp-ml-commons` jar to compare against on the classpath ahead of
+the freshly built classes and rerun.
+
### POSTagger cache impact
The `POSTaggerMEBenchmark` uses `@Param({"0", "3"})` for cache
diff --git
a/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/ml/BeamSearchBenchmark.java
b/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/ml/BeamSearchBenchmark.java
index 18e2f8212..12f8d6cbc 100644
---
a/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/ml/BeamSearchBenchmark.java
+++
b/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/ml/BeamSearchBenchmark.java
@@ -48,15 +48,8 @@ import opennlp.tools.util.SequenceValidator;
* JMH benchmark for {@link BeamSearch} on long input sequences.
* <p>
* One op = one {@code bestSequence} call on a synthetic token sequence of
- * {@code sequenceLength} tokens. Long sequences are what expose the quadratic
- * per-candidate outcome-list copying removed with OPENNLP-1903; sentences of
- * 5-10 tokens, as used by the ME benchmarks, show no measurable difference.
- * Only public API that predates OPENNLP-1903 is used
- * ({@code BeamSearch(int, MaxentModel, int)} and
- * {@code bestSequence(T[], Object[], BeamSearchContextGenerator,
SequenceValidator)}),
- * so the same compiled class exercises both implementations: to produce a
- * baseline, place an {@code opennlp-ml-commons} jar built before OPENNLP-1903
- * on the classpath ahead of the freshly built classes and rerun.
+ * {@code sequenceLength} tokens, decoded with a seeded model over a fixed
outcome set.
+ * Long sequences measure how decoding cost grows with sequence length.
*/
@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.SECONDS)
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java
index c06e4fc2d..5382ea5c4 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/doccat/EmojiFeatureGenerator.java
@@ -38,6 +38,7 @@ import opennlp.tools.util.normalizer.EmojiAnnotator;
* manifest.</p>
*
* @see EmojiAnnotator
+ * @since 3.0.0
*/
public class EmojiFeatureGenerator implements FeatureGenerator {
@@ -54,7 +55,7 @@ public class EmojiFeatureGenerator implements
FeatureGenerator {
* Instantiates a generator over a configured annotator, for example one
with a gazetteer join.
*
* @param annotator The annotator to use. Must not be {@code null}.
- * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code annotator} is {@code
null}.
*/
public EmojiFeatureGenerator(EmojiAnnotator annotator) {
if (annotator == null) {
@@ -66,7 +67,7 @@ public class EmojiFeatureGenerator implements
FeatureGenerator {
/**
* {@inheritDoc}
*
- * @throws IllegalArgumentException if {@code text} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code text} is {@code null}.
*/
@Override
public Collection<String> extractFeatures(String[] text, Map<String, Object>
extraInformation) {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinder.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinder.java
index a3bde1021..205e4052f 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinder.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinder.java
@@ -29,9 +29,6 @@ import opennlp.tools.util.Span;
/**
* A {@link TokenNameFinder} implementation based on a series of regular
expressions.
- * <p>
- * Please note: RegEx is not allowed in the processing path of OpenNLP.
- * However, a RegEx step is an obvious exception.
*/
public final class RegexNameFinder implements TokenNameFinder {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
index c505dbb72..7a6097a5a 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/namefind/RegexNameFinderFactory.java
@@ -27,9 +27,6 @@ import opennlp.tools.util.ParamChecks;
*
* Returns a {@link RegexNameFinder} based on a selection of
* defaults or a configuration and a selection of defaults.
- * <p>
- * Please note: RegEx is not allowed in the processing path of OpenNLP.
- * However, a RegEx step is an obvious exception.
*/
public class RegexNameFinderFactory {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/SentenceDetectorME.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/SentenceDetectorME.java
index 09b334c15..c7bb2fc85 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/SentenceDetectorME.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/SentenceDetectorME.java
@@ -192,15 +192,8 @@ public class SentenceDetectorME implements
SentenceDetector, Probabilistic {
return sentences;
}
- // The whitespace definition of the sentence detector: the Unicode
White_Space set
- // (OPENNLP-205). It drives both the detection loop of sentPosDetect (the
delimiter-run skip
- // heuristic and the placement of sentence-start positions) and the
position-to-span mapping,
- // so both stages agree on what separates sentences. Unlike the previously
used
- // StringUtil.isWhitespace, this covers the next line control (U+0085) and
does not treat the
- // U+001C..U+001F information separators as whitespace; around those
characters the candidate
- // positions themselves can differ from pre-OPENNLP-205 releases, not only
the span edges.
- // Both deltas are pinned in SentenceDetectorMESpanMappingTest. Model
feature generation
- // (SDContextGenerator) is not affected.
+ // The Unicode White_Space set. It drives both the detection loop of
sentPosDetect and the
+ // position-to-span mapping, so both stages agree on what separates
sentences.
private static final CharClass WHITESPACE = CharClass.whitespace();
private int getFirstWS(CharSequence s, int pos) {
@@ -282,11 +275,7 @@ public class SentenceDetectorME implements
SentenceDetector, Probabilistic {
/**
* Maps accepted sentence-start positions to trimmed sentence {@link Span}s,
the core of the
- * end-of-sentence position to span mapping (OPENNLP-205). Not part of the
public API; the
- * package visibility is deliberate, kept so the mapping can be exercised
directly in tests.
- * Several branches (the whitespace-only candidate that keeps {@code probs}
aligned, and the
- * stale-{@code probs} reset in the zero-positions branch) are not reachable
through the public
- * {@code sentPosDetect} entry point.
+ * end-of-sentence position to span mapping (OPENNLP-205).
*
* <p>Each span runs from the previous position (or the text start) to the
next position, with
* Unicode {@code White_Space} trimmed from both edges. A candidate that is
whitespace-only is
@@ -425,7 +414,7 @@ public class SentenceDetectorME implements
SentenceDetector, Probabilistic {
/**
* The dictionary entries, folded to lower case unless the dictionary is
case-sensitive.
- * Only the first token of a multi-token entry participates, as before.
+ * Only the first token of a multi-token entry participates.
*/
private final Set<String> entries;
@@ -445,6 +434,8 @@ public class SentenceDetectorME implements
SentenceDetector, Probabilistic {
private final boolean caseSensitive;
/**
+ * Creates an {@link AbbreviationIndex} over a {@link Dictionary}.
+ *
* @param abbDict The {@link Dictionary} to index, may be {@code null}.
* @return An index over {@code abbDict}, or {@code null} if {@code
abbDict} is {@code null}.
*/
@@ -473,6 +464,8 @@ public class SentenceDetectorME implements
SentenceDetector, Probabilistic {
}
/**
+ * Checks whether a sentence break at a candidate position is allowed by
the {@link Dictionary}.
+ *
* @param s The text in which the break occurred.
* @param fromIndex The start of the segment currently being evaluated.
* @param candidateIndex The index of the candidate sentence ending.
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java
index a10aac0d7..0aa93b658 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentContextGenerator.java
@@ -38,6 +38,7 @@ import opennlp.tools.util.normalizer.EmojiAnnotator;
*
* @see EmojiSentimentFactory
* @see EmojiAnnotator
+ * @since 3.0.0
*/
public class EmojiSentimentContextGenerator extends SentimentContextGenerator {
@@ -55,7 +56,7 @@ public class EmojiSentimentContextGenerator extends
SentimentContextGenerator {
* gazetteer join.
*
* @param annotator The annotator to use. Must not be {@code null}.
- * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code annotator} is {@code
null}.
*/
public EmojiSentimentContextGenerator(EmojiAnnotator annotator) {
if (annotator == null) {
@@ -70,7 +71,7 @@ public class EmojiSentimentContextGenerator extends
SentimentContextGenerator {
* <p>Appends the emoji annotation features of every annotated token to the
default token
* context.</p>
*
- * @throws IllegalArgumentException if {@code text} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code text} is {@code null}.
*/
@Override
public String[] getContext(String[] text) {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentFactory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentFactory.java
index 97e03600d..92fe18d78 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentFactory.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentiment/EmojiSentimentFactory.java
@@ -24,6 +24,8 @@ package opennlp.tools.sentiment;
* opennlp.tools.util.TrainingParameters, SentimentFactory) SentimentME.train}
to opt in; the
* factory class is recorded in the trained model's manifest, so prediction
re-creates the same
* context. The default {@link SentimentFactory} is unchanged.
+ *
+ * @since 3.0.0
*/
public class EmojiSentimentFactory extends SentimentFactory {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/CachingStemmer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/CachingStemmer.java
index 0f32ea665..61395a74e 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/CachingStemmer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/CachingStemmer.java
@@ -42,8 +42,7 @@ import opennlp.tools.util.ParamChecks;
public final class CachingStemmer extends
DelegatingStemmer<CachingStemmer.ThreadState> {
/**
- * Covers the high-frequency vocabulary of most corpora while keeping the
per-thread footprint
- * small.
+ * The default maximum number of word-to-stem entries cached per thread:
{@value}.
*/
public static final int DEFAULT_CAPACITY = 1024;
@@ -51,7 +50,7 @@ public final class CachingStemmer extends
DelegatingStemmer<CachingStemmer.Threa
* Creates a caching stemmer with the {@linkplain #DEFAULT_CAPACITY default
capacity}.
*
* @param factory The factory that mints one delegate per thread. Must not
be {@code null}.
- * @throws IllegalArgumentException if {@code factory} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code factory} is {@code
null}.
*/
public CachingStemmer(StemmerFactory factory) {
this(factory, DEFAULT_CAPACITY);
@@ -62,7 +61,7 @@ public final class CachingStemmer extends
DelegatingStemmer<CachingStemmer.Threa
*
* @param factory The factory that mints one delegate per thread. Must not
be {@code null}.
* @param capacity The maximum number of word-to-stem entries kept per
thread; must be positive.
- * @throws IllegalArgumentException if {@code factory} is {@code null} or
{@code capacity} is
+ * @throws IllegalArgumentException Thrown if {@code factory} is {@code
null} or {@code capacity} is
* not positive.
*/
public CachingStemmer(StemmerFactory factory, int capacity) {
@@ -76,7 +75,7 @@ public final class CachingStemmer extends
DelegatingStemmer<CachingStemmer.Threa
* @param factory The factory that mints one delegate per thread. Must not
be {@code null}.
* @param capacity The maximum number of word-to-stem entries kept per
thread; must be positive.
* @return a supplier of fresh per-thread state.
- * @throws IllegalArgumentException if {@code factory} is {@code null} or
{@code capacity} is
+ * @throws IllegalArgumentException Thrown if {@code factory} is {@code
null} or {@code capacity} is
* not positive.
*/
private static Supplier<ThreadState> threadStateSupplier(StemmerFactory
factory, int capacity) {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/DelegatingStemmer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/DelegatingStemmer.java
index 8b17059e7..49a59efa1 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/DelegatingStemmer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/DelegatingStemmer.java
@@ -38,8 +38,11 @@ abstract class DelegatingStemmer<P> implements Stemmer {
protected final OwnerOrPerThreadState<P> state;
/**
- * @param init Mints the per-thread payload on first use by the owner and
by each extra thread.
- * @param cleanup Releases a per-thread payload when its thread clears state.
+ * Creates the payload holder and the owner payload.
+ *
+ * @param init Creates a payload: once for the owner during construction,
and once for each
+ * other thread on its first call.
+ * @param cleanup Resets the owner payload when the owner thread clears its
state.
*/
DelegatingStemmer(Supplier<P> init, Consumer<P> cleanup) {
this.state = new OwnerOrPerThreadState<>(init, cleanup);
@@ -51,7 +54,7 @@ abstract class DelegatingStemmer<P> implements Stemmer {
*
* @param factory The factory to validate. Must not be {@code null}.
* @return {@code factory}.
- * @throws IllegalArgumentException if {@code factory} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code factory} is {@code
null}.
*/
static StemmerFactory requireFactory(StemmerFactory factory) {
ParamChecks.requireNonNullArg(factory, "factory");
@@ -59,9 +62,10 @@ abstract class DelegatingStemmer<P> implements Stemmer {
}
/**
- * Removes this thread's payload to prevent classloader leaks in container
environments. Call
- * when the thread is returned to a pool or the stemmer is no longer needed,
mirroring
- * {@code clearThreadLocalState()} on the thread-safe {@code *ME} components.
+ * Clears the calling thread's state to prevent classloader leaks in
container environments: a
+ * per-thread payload is removed, and the owner payload is reset but stays
referenced by this
+ * instance. Call when the thread is returned to a pool or the stemmer is no
longer needed,
+ * mirroring {@code clearThreadLocalState()} on the thread-safe {@code *ME}
components.
*/
public void clearThreadLocalState() {
state.clearForCurrentThread();
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmer.java
index d81605efc..863a79637 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmer.java
@@ -597,13 +597,9 @@ public class PorterStemmer implements Stemmer {
}
/**
- * Stem a word provided as a CharSequence.
- * Returns the result as a CharSequence.
- *
- * @param word The word to stem. Must not be {@code null}.
- * @return the stemmed word.
- * @throws IllegalArgumentException if {@code word} is {@code null}.
+ * {@inheritDoc}
*/
+ @Override
public CharSequence stem(CharSequence word) {
ParamChecks.requireNonNullArg(word, "word");
return stem(word.toString());
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/SharingStemmer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/SharingStemmer.java
index 720df345d..a72c50938 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/SharingStemmer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/SharingStemmer.java
@@ -37,8 +37,10 @@ import opennlp.tools.util.ParamChecks;
public final class SharingStemmer extends DelegatingStemmer<Stemmer> {
/**
+ * Creates a sharing stemmer.
+ *
* @param factory The factory that mints per-thread delegates. Must not be
{@code null}.
- * @throws IllegalArgumentException if {@code factory} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code factory} is {@code
null}.
*/
public SharingStemmer(StemmerFactory factory) {
super(requireFactory(factory)::newStemmer, stemmer -> { });
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/snowball/SnowballStemmer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/snowball/SnowballStemmer.java
index bc634f1c4..9bec18b95 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/snowball/SnowballStemmer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/snowball/SnowballStemmer.java
@@ -104,7 +104,8 @@ public class SnowballStemmer implements Stemmer {
}
/**
- * Removes this thread's engine to prevent classloader leaks in container
environments. Call
+ * Clears the calling thread's state to prevent classloader leaks in
container environments: a
+ * per-thread engine is removed, and the owner engine stays referenced by
this instance. Call
* when the thread is returned to a pool or the stemmer is no longer needed,
mirroring
* {@code clearThreadLocalState()} on the thread-safe {@code *ME} components.
*/
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/snowball/SnowballStemmerFactory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/snowball/SnowballStemmerFactory.java
index da162161c..b0f012546 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/snowball/SnowballStemmerFactory.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/snowball/SnowballStemmerFactory.java
@@ -40,7 +40,7 @@ public class SnowballStemmerFactory implements StemmerFactory
{
* Creates a factory with {@code repeat = 1}.
*
* @param algorithm The Snowball algorithm. Must not be {@code null}.
- * @throws IllegalArgumentException if {@code algorithm} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code algorithm} is {@code
null}.
*/
public SnowballStemmerFactory(SnowballStemmer.ALGORITHM algorithm) {
this(algorithm, 1);
@@ -51,7 +51,7 @@ public class SnowballStemmerFactory implements StemmerFactory
{
*
* @param algorithm The Snowball algorithm. Must not be {@code null}.
* @param repeat How many times to apply the stemmer per word; must be
positive.
- * @throws IllegalArgumentException if {@code algorithm} is {@code null} or
{@code repeat}
+ * @throws IllegalArgumentException Thrown if {@code algorithm} is {@code
null} or {@code repeat}
* is not positive.
*/
public SnowballStemmerFactory(SnowballStemmer.ALGORITHM algorithm, int
repeat) {
@@ -90,8 +90,6 @@ public class SnowballStemmerFactory implements StemmerFactory
{
/**
* {@inheritDoc}
- *
- * @throws IllegalArgumentException if {@code word} is {@code null}.
*/
@Override
public CharSequence stem(CharSequence word) {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordBreakProperty.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordBreakProperty.java
index d6093386c..54017e232 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordBreakProperty.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordBreakProperty.java
@@ -30,10 +30,9 @@ import opennlp.tools.util.normalizer.HexCodePoints;
* Looks up the Unicode {@link WordBreak Word_Break} property of a code point.
*
* <p>The data is loaded on first use from the {@code WordBreakProperty.txt}
resource of the
- * Unicode Character Database (parsed with simple cursor scanning, no regular
expression). Lookup
- * is O(1) for the Basic Multilingual Plane (a direct array index) and O(log
n) for supplementary
- * code points (a binary search over a small sorted range table), so it
imposes no per-character
- * allocation on the word boundary algorithm.</p>
+ * Unicode Character Database. Lookup is O(1) for the Basic Multilingual Plane
(a direct array
+ * index) and O(log n) for supplementary code points (a binary search over a
small sorted range
+ * table), so it imposes no per-character allocation on the word boundary
algorithm.</p>
*/
public final class WordBreakProperty {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordSegmenter.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordSegmenter.java
index 3bfec882e..fdc7b22f1 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordSegmenter.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordSegmenter.java
@@ -29,11 +29,11 @@ import opennlp.tools.util.Span;
* (<a href="https://www.unicode.org/reports/tr29/";>UAX #29</a>), rules WB1
through WB999.
*
* <p>The implementation is a single forward cursor pass with O(1) {@link
WordBreakProperty}
- * lookups and no regular expression. It decodes each code point once, keeps
only a constant amount
- * of state, and allocates nothing per character. It implements the "ignore"
semantics of WB4 (a
- * base character absorbs following {@code Extend}, {@code Format}, and {@code
ZWJ}), the look-ahead
- * rules WB6/WB7/WB7b/WB12, the Hebrew quote rules WB7a-WB7c, the emoji
zero-width-joiner rule WB3c,
- * and regional-indicator pairing WB15/WB16. The look-ahead for the
WB6/WB7b/WB12 rules is resolved
+ * lookups. It decodes each code point once, keeps only a constant amount of
state, and allocates
+ * nothing per character. It implements the "ignore" semantics of WB4 (a base
character absorbs
+ * following {@code Extend}, {@code Format}, and {@code ZWJ}), the look-ahead
rules
+ * WB6/WB7/WB7b/WB12, the Hebrew quote rules WB7a-WB7c, the emoji
zero-width-joiner rule WB3c, and
+ * regional-indicator pairing WB15/WB16. The look-ahead for the WB6/WB7b/WB12
rules is resolved
* lazily and only at mid-word punctuation, so the common case never scans
ahead.</p>
*
* <p>{@link #forEachSegment(CharSequence, SegmentConsumer)} streams the
segments with no
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordTokenizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordTokenizer.java
index b5fc9ede7..0d4e1c36a 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordTokenizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordTokenizer.java
@@ -24,12 +24,14 @@ import opennlp.tools.util.ParamChecks;
import opennlp.tools.util.Span;
/**
- * A word tokenizer built on the Unicode Text Segmentation algorithm (UAX
#29). It finds segments
- * with {@link WordSegmenter}, keeps the ones that are words (letters, digits,
ideographs, kana,
- * Hangul, Southeast-Asian script, or emoji), drops whitespace and
punctuation, and classifies each
- * kept token with a {@link WordType}. Emoji here means any {@code
Extended_Pictographic} code point,
- * so symbol-like characters such as the copyright, trademark, and
double-exclamation signs are kept
- * (typed {@link WordType#EMOJI}) rather than dropped as punctuation.
+ * A word tokenizer built on the Unicode Text Segmentation algorithm
+ * (<a href="https://www.unicode.org/reports/tr29/";>UAX #29</a>).
+ * It finds segments with {@link WordSegmenter}, keeps the ones that are words
(letters, digits,
+ * ideographs, kana, Hangul, Southeast-Asian script, or emoji), drops
whitespace and punctuation,
+ * and classifies each kept token with a {@link WordType}.
+ * Emoji here means any {@code Extended_Pictographic} code point, so
symbol-like characters such as
+ * the copyright, trademark, and double-exclamation signs are kept (typed
{@link WordType#EMOJI})
+ * rather than dropped as punctuation.
*
* <p>A token longer than {@code maxTokenLength} is emitted as consecutive
pieces, never splitting a
* surrogate pair. The tokenizer reports offset {@link Span}s, so the original
text and its character
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordType.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordType.java
index 90c7d53a8..c44d7ef85 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordType.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/uax29/WordType.java
@@ -80,7 +80,8 @@ public enum WordType {
* Classifies the code points of {@code text} over {@code [start, end)} as a
word token type, or
* {@code null} when the range is not a word (pure whitespace, punctuation,
or symbols). Emoji
* win over scripts, scripts over the generic alphanumeric/numeric split.
The script category is
- * taken from the first script code point in the range; UAX #29 word
segments are
+ * taken from the first script code point in the range;
+ * <a href="https://www.unicode.org/reports/tr29/";>UAX #29</a> word
segments are
* single-script in practice, so for an unusual mixed-script run this
reports the leading script,
* not a per-character determination.
*
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
index 598e653ae..02135c415 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/DictionaryCatalog.java
@@ -174,6 +174,8 @@ public final class DictionaryCatalog {
*/
public record Entry(String id, URI uri, String sha512, String filename) {
/**
+ * Validates the components.
+ *
* @param id The catalog id. Must not be {@code null}.
* @param uri The absolute download URI. Must not be {@code null}.
* @param sha512 The expected SHA-512 hex digest. Must not be {@code null}.
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/archive/TarStream.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/archive/TarStream.java
index 888508f49..820933472 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/archive/TarStream.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/archive/TarStream.java
@@ -29,7 +29,11 @@ import opennlp.tools.commons.Internal;
import opennlp.tools.util.ParamChecks;
/**
- * A forward-only reader for classic v7, POSIX ustar, GNU, and pax tar streams.
+ * A forward-only reader for classic v7,
+ * <a
href="https://pubs.opengroup.org/onlinepubs/9799919799/utilities/pax.html";>POSIX
ustar</a>,
+ * <a
href="https://www.gnu.org/software/tar/manual/html_node/Standard.html";>GNU</a>,
and
+ * <a
href="https://pubs.opengroup.org/onlinepubs/9799919799/utilities/pax.html";>pax</a>
tar
+ * streams.
* {@link #next()} advances to the following entry and {@link #entryStream()}
exposes
* only the current entry's bytes.
*
@@ -425,22 +429,22 @@ public final class TarStream {
}
/**
- * @return The current entry's name as stored in the archive. Not {@code
null}
- * after a successful {@link #next()}.
+ * {@return the current entry's name as stored in the archive; not {@code
null} after a
+ * successful {@link #next()}}
*/
public String name() {
return name;
}
/**
- * @return The current entry's size in bytes.
+ * {@return the current entry's size in bytes}
*/
public long size() {
return size;
}
/**
- * @return {@code true} if the current entry is a regular file.
+ * {@return {@code true} if the current entry is a regular file}
*/
public boolean isFile() {
return type == TYPE_REGULAR_FILE || type == TYPE_REGULAR_FILE_CLASSIC;
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java
index 2e6d0d003..e3c750e99 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGenerator.java
@@ -37,6 +37,7 @@ import opennlp.tools.util.normalizer.EmojiAnnotator;
* join is): it keeps no adaptive data.</p>
*
* @see EmojiAnnotator
+ * @since 3.0.0
*/
public class EmojiAnnotationFeatureGenerator implements
AdaptiveFeatureGenerator {
@@ -53,7 +54,7 @@ public class EmojiAnnotationFeatureGenerator implements
AdaptiveFeatureGenerator
* Instantiates a generator over a configured annotator, for example one
with a gazetteer join.
*
* @param annotator The annotator to use. Must not be {@code null}.
- * @throws IllegalArgumentException if {@code annotator} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code annotator} is {@code
null}.
*/
public EmojiAnnotationFeatureGenerator(EmojiAnnotator annotator) {
if (annotator == null) {
@@ -67,7 +68,7 @@ public class EmojiAnnotationFeatureGenerator implements
AdaptiveFeatureGenerator
*
* <p>Adds one feature per present annotation attribute of the token at
{@code index}.</p>
*
- * @throws IllegalArgumentException if {@code features} or {@code tokens} is
{@code null}.
+ * @throws IllegalArgumentException Thrown if {@code features} or {@code
tokens} is {@code null}.
*/
@Override
public void createFeatures(List<String> features, String[] tokens, int index,
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorFactory.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorFactory.java
index 1a37185aa..01fc53c75 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorFactory.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/EmojiAnnotationFeatureGeneratorFactory.java
@@ -28,6 +28,7 @@ import opennlp.tools.util.InvalidFormatException;
* }</pre>
*
* @see EmojiAnnotationFeatureGenerator
+ * @since 3.0.0
*/
public class EmojiAnnotationFeatureGeneratorFactory
extends GeneratorFactory.AbstractXmlFeatureGeneratorFactory {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/AccentFoldCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/AccentFoldCharSequenceNormalizer.java
index 46a8a66b5..f2659c988 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/AccentFoldCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/AccentFoldCharSequenceNormalizer.java
@@ -37,9 +37,6 @@ import opennlp.tools.util.ParamChecks;
* enabled (the default) these are mapped to an ASCII approximation. Folding
is a recall
* optimization, not a linguistically correct transform, so it is intended for
a matching
* token rather than for display or language-specific analysis.</p>
- *
- * <p>Scanning is a single cursor pass over the decomposed text; no regular
expression is used, and
- * no global {@code \p{Mn}} strip is performed.</p>
*/
public class AccentFoldCharSequenceNormalizer implements
CharSequenceNormalizer {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/BulletCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/BulletCharSequenceNormalizer.java
index c5d9905dd..81ad02b42 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/BulletCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/BulletCharSequenceNormalizer.java
@@ -20,10 +20,9 @@ package opennlp.tools.util.normalizer;
* A {@link CharSequenceNormalizer} that replaces unambiguous list-bullet
characters with a space,
* so a bullet acts as a token separator rather than sticking to the following
word.
*
- * <p>Membership is an O(1) {@link CharClass} lookup and scanning is a single
cursor pass with no
- * regular expression. The middle dot ({@code U+00B7}) is deliberately
<em>not</em> included,
- * because it is a letter in Catalan ({@code l..l}) and other orthographies;
only characters that
- * are unambiguously list bullets are replaced.</p>
+ * <p>Membership is an O(1) {@link CharClass} lookup. The middle dot ({@code
U+00B7}) is
+ * deliberately <em>not</em> included, because it is a letter in Catalan
({@code l..l}) and other
+ * orthographies; only characters that are unambiguously list bullets are
replaced.</p>
*/
public class BulletCharSequenceNormalizer implements OffsetAwareNormalizer {
@@ -44,11 +43,13 @@ public class BulletCharSequenceNormalizer implements
OffsetAwareNormalizer {
return INSTANCE;
}
+ /** {@inheritDoc} */
@Override
public CharSequence normalize(CharSequence text) {
return BULLETS.normalize(text);
}
+ /** {@inheritDoc} */
@Override
public AlignedText normalizeAligned(CharSequence text) {
return BULLETS.normalizeAligned(text);
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/DashCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/DashCharSequenceNormalizer.java
index 70eeedd63..69431bae2 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/DashCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/DashCharSequenceNormalizer.java
@@ -38,11 +38,13 @@ public class DashCharSequenceNormalizer implements
OffsetAwareNormalizer {
return INSTANCE;
}
+ /** {@inheritDoc} */
@Override
public CharSequence normalize(CharSequence text) {
return DASHES.normalize(text);
}
+ /** {@inheritDoc} */
@Override
public AlignedText normalizeAligned(CharSequence text) {
return DASHES.normalizeAligned(text);
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/DigitCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/DigitCharSequenceNormalizer.java
index 6735dcdf0..13dbeb7cd 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/DigitCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/DigitCharSequenceNormalizer.java
@@ -25,8 +25,7 @@ import opennlp.tools.util.StringUtil;
* <p>It maps a code point when {@link Character#digit(int, int)} reports a
value of {@code 0}-
* {@code 9} in radix ten, that is, when the code point is a Unicode decimal
digit. Other numeric
* forms (Roman numerals, superscripts, circled numbers, fractions) are not
decimal digits and are
- * left unchanged. Scanning is a single O(1)-per-code-point cursor pass with
no regular
- * expression.</p>
+ * left unchanged.</p>
*/
public class DigitCharSequenceNormalizer implements OffsetAwareNormalizer {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/Dimension.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/Dimension.java
index bdc50de59..83d416e03 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/Dimension.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/Dimension.java
@@ -49,7 +49,10 @@ public enum Dimension {
/** Case folding; lossy and locale sensitive. */
CASE_FOLD(CaseFoldCharSequenceNormalizer::getInstance),
- /** Unicode full case folding (UTS #21); lossy and expanding (sharp s to ss,
the ligatures). */
+ /**
+ * Unicode full case folding (<a
href="https://www.unicode.org/reports/tr21/";>UTS #21</a>);
+ * lossy and expanding (sharp s to ss, the ligatures).
+ */
FULL_CASE_FOLD(FullCaseFoldCharSequenceNormalizer::getInstance),
/** Diacritic and accent folding; lossy, script gated, and language-wrong
for some languages. */
@@ -58,7 +61,10 @@ public enum Dimension {
/** Emoji folded to ASCII emoticons where a mapping exists; lossy, for
matching only. */
EMOJI_FOLD(EmojiToEmoticonCharSequenceNormalizer::getInstance),
- /** Confusable (homoglyph) skeleton folding per UTS #39; lossy, for matching
only. */
+ /**
+ * Confusable (homoglyph) skeleton folding per
+ * <a href="https://www.unicode.org/reports/tr39/";>UTS #39</a>; lossy, for
matching only.
+ */
CONFUSABLE_FOLD(ConfusableSkeletonCharSequenceNormalizer::getInstance),
/** Stemming through a configured {@link opennlp.tools.stemmer.Stemmer}. */
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EllipsisCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EllipsisCharSequenceNormalizer.java
index 8191913f5..870400912 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EllipsisCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EllipsisCharSequenceNormalizer.java
@@ -21,8 +21,7 @@ package opennlp.tools.util.normalizer;
* the horizontal ellipsis ({@code U+2026}) to {@code "..."} and the two-dot
leader
* ({@code U+2025}) to {@code ".."}.
*
- * <p>Scanning is a single O(1)-per-code-point cursor pass with no regular
expression. ASCII dot
- * runs are left unchanged.</p>
+ * <p>ASCII dot runs are left unchanged.</p>
*/
public class EllipsisCharSequenceNormalizer implements OffsetAwareNormalizer {
@@ -36,11 +35,13 @@ public class EllipsisCharSequenceNormalizer implements
OffsetAwareNormalizer {
return INSTANCE;
}
+ /** {@inheritDoc} */
@Override
public CharSequence normalize(CharSequence text) {
return CharClass.substitute(text,
EllipsisCharSequenceNormalizer::expansion);
}
+ /** {@inheritDoc} */
@Override
public AlignedText normalizeAligned(CharSequence text) {
return CharClass.substituteAligned(text,
EllipsisCharSequenceNormalizer::expansion);
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
index 393032074..b72fdc528 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotation.java
@@ -35,7 +35,8 @@ import java.util.OptionalInt;
*
* @param symbol The annotated code point sequence, without the U+FE0F
presentation selector.
* @param attributes The attribute values keyed by attribute name ({@link
#NAME},
- * {@link #SENTIMENT}, {@link #ENTITY_TYPE}, {@link
#CATEGORY}).
+ * {@link #SENTIMENT}, {@link #ENTITY_TYPE}, {@link
#CATEGORY},
+ * {@link #ISO_REGION}).
*/
public record EmojiAnnotation(String symbol, Map<String, Value> attributes) {
@@ -69,6 +70,12 @@ public record EmojiAnnotation(String symbol, Map<String,
Value> attributes) {
*/
public record Value(String value, String source, String notes) {
+ /**
+ * Validates the components.
+ *
+ * @throws IllegalArgumentException Thrown if {@code value} or {@code
source} is {@code null}
+ * or empty, or if {@code notes} is {@code null}.
+ */
public Value {
if (value == null || value.isEmpty()) {
throw new IllegalArgumentException("Value must not be null or empty");
@@ -88,7 +95,7 @@ public record EmojiAnnotation(String symbol, Map<String,
Value> attributes) {
* @param symbol The annotated code point sequence. Must not be {@code
null} or empty.
* @param attributes The attribute values keyed by attribute name. Must not
be {@code null} or
* contain {@code null} keys or values; it is defensively
copied.
- * @throws IllegalArgumentException if {@code symbol} is {@code null} or
empty, or if
+ * @throws IllegalArgumentException Thrown if {@code symbol} is {@code null}
or empty, or if
* {@code attributes} is {@code null} or contains a {@code null} key or
value.
*/
public EmojiAnnotation {
@@ -112,7 +119,7 @@ public record EmojiAnnotation(String symbol, Map<String,
Value> attributes) {
*
* @param name The attribute name, for example {@link #NAME}. Must not be
{@code null}.
* @return The value, or empty when this record does not carry the attribute.
- * @throws IllegalArgumentException if {@code name} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code name} is {@code null}.
*/
public Optional<Value> attribute(String name) {
if (name == null) {
@@ -141,7 +148,7 @@ public record EmojiAnnotation(String symbol, Map<String,
Value> attributes) {
/**
* {@return the project-authored coarse sentiment score in -2..2, or empty
when not annotated}
*
- * @throws IllegalStateException if the stored value is not an integer,
which cannot happen for
+ * @throws IllegalStateException Thrown if the stored value is not an
integer, which cannot happen for
* records loaded from the bundled data (the loader validates it).
*/
public OptionalInt sentiment() {
@@ -160,7 +167,7 @@ public record EmojiAnnotation(String symbol, Map<String,
Value> attributes) {
/**
* {@return the coarse entity type, or empty when not annotated}
*
- * @throws IllegalStateException if the stored value is not an {@link
EmojiEntityType} constant,
+ * @throws IllegalStateException Thrown if the stored value is not an {@link
EmojiEntityType} constant,
* which cannot happen for records loaded from the bundled data (the
loader validates it).
*/
public Optional<EmojiEntityType> entityType() {
@@ -179,7 +186,7 @@ public record EmojiAnnotation(String symbol, Map<String,
Value> attributes) {
/**
* {@return the document-category hint, or empty when not annotated}
*
- * @throws IllegalStateException if the stored value is not an {@link
EmojiCategory} constant,
+ * @throws IllegalStateException Thrown if the stored value is not an {@link
EmojiCategory} constant,
* which cannot happen for records loaded from the bundled data (the
loader validates it).
*/
public Optional<EmojiCategory> category() {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
index 511148697..32f9ab7dc 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotations.java
@@ -74,10 +74,10 @@ public final class EmojiAnnotations {
* @param symbol The code point sequence of one symbol, for example one
{@link Term#original()}
* token. U+FE0F presentation selectors are ignored. Must not
be {@code null}.
* @return The record, or empty when the bundled data does not annotate the
symbol.
- * @throws IllegalArgumentException if {@code symbol} is {@code null}, or if
the bundled data
- * is malformed.
- * @throws IllegalStateException if the bundled data resource is missing.
- * @throws UncheckedIOException if the bundled data resource cannot be read.
+ * @throws IllegalArgumentException Thrown if {@code symbol} is {@code
null}, or if the bundled
+ * data is malformed.
+ * @throws IllegalStateException Thrown if the bundled data resource is
missing.
+ * @throws UncheckedIOException Thrown if the bundled data resource cannot
be read.
*/
public static Optional<EmojiAnnotation> lookup(CharSequence symbol) {
if (symbol == null) {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java
index cf7c590a3..6e8618eaa 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiAnnotator.java
@@ -29,7 +29,7 @@ import java.util.Optional;
* thread-safe when the join is.
*
* <p>Annotations are per-symbol metadata, not text transforms; a parallel
surface beside
- * {@link Term} rather than {@link Dimension} constants. See OPENNLP-1870.</p>
+ * {@link Term} rather than {@link Dimension} constants.</p>
*/
public final class EmojiAnnotator {
@@ -70,7 +70,7 @@ public final class EmojiAnnotator {
*
* @param join The joined-facts hook, called while a record is assembled.
Must not be
* {@code null}.
- * @throws IllegalArgumentException if {@code join} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code join} is {@code null}.
*/
public EmojiAnnotator(EmojiAnnotationJoin join) {
if (join == null) {
@@ -80,13 +80,12 @@ public final class EmojiAnnotator {
}
/**
- * Annotates one term, keyed on its {@link Term#original() original} text
(see the class note on
- * why the original layer is the one annotations describe).
+ * Annotates one term, keyed on its {@link Term#original() original} text.
*
* @param term The term to annotate. Must not be {@code null}.
* @return The assembled record, or empty when the term is not an annotated
symbol.
- * @throws IllegalArgumentException if {@code term} is {@code null}.
- * @throws IllegalStateException if the configured join violates its
contract (returns
+ * @throws IllegalArgumentException Thrown if {@code term} is {@code null}.
+ * @throws IllegalStateException Thrown if the configured join violates its
contract (returns
* {@code null} or a key colliding with an existing attribute).
*/
public Optional<EmojiAnnotation> annotate(Term term) {
@@ -102,8 +101,8 @@ public final class EmojiAnnotator {
* @param symbol The code point sequence of one symbol (one token). U+FE0F
presentation
* selectors are ignored. Must not be {@code null}.
* @return The assembled record, or empty when {@code symbol} is not an
annotated symbol.
- * @throws IllegalArgumentException if {@code symbol} is {@code null}.
- * @throws IllegalStateException if the configured join violates its
contract (returns
+ * @throws IllegalArgumentException Thrown if {@code symbol} is {@code null}.
+ * @throws IllegalStateException Thrown if the configured join violates its
contract (returns
* {@code null} or a key colliding with an existing attribute).
*/
public Optional<EmojiAnnotation> annotate(CharSequence symbol) {
@@ -164,8 +163,8 @@ public final class EmojiAnnotator {
*
* @param token The token to describe. Must not be {@code null}.
* @param features The collection to append feature strings to. Must not be
{@code null}.
- * @throws IllegalArgumentException if {@code token} or {@code features} is
{@code null}.
- * @throws IllegalStateException if the configured join violates its
contract (see
+ * @throws IllegalArgumentException Thrown if {@code token} or {@code
features} is {@code null}.
+ * @throws IllegalStateException Thrown if the configured join violates its
contract (see
* {@link #annotate(CharSequence)}).
*/
public void collectFeatures(CharSequence token, Collection<String> features)
{
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEmoticons.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEmoticons.java
index 78fb4e9b6..3ce7d0716 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEmoticons.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiEmoticons.java
@@ -323,8 +323,8 @@ final class EmojiEmoticons {
/**
* {@return the tables parsed from the bundled {@code emoji-emoticons.txt}
resource}
*
- * @throws IllegalStateException if the resource is missing.
- * @throws UncheckedIOException if the resource cannot be read.
+ * @throws IllegalStateException Thrown if the resource is missing.
+ * @throws UncheckedIOException Thrown if the resource cannot be read.
*/
private static Tables loadBundled() {
return BundledUnicodeData.load(EmojiEmoticons.class, RESOURCE,
"emoji/emoticon fold",
@@ -340,8 +340,8 @@ final class EmojiEmoticons {
*
* @param in the stream to parse. Must not be {@code null}.
* @return the two direction tables.
- * @throws IOException if the stream cannot be read.
- * @throws IllegalArgumentException if the data is malformed.
+ * @throws IOException Thrown if the stream cannot be read.
+ * @throws IllegalArgumentException Thrown if the data is malformed.
*/
static Tables parse(InputStream in) throws IOException {
ParamChecks.requireNonNullArg(in, "in");
@@ -411,7 +411,7 @@ final class EmojiEmoticons {
* @param lineNumber the line number, for the error message.
* @param content the full line, for the error message.
* @return the decoded sequence.
- * @throws IllegalArgumentException if the field is empty, not valid
hexadecimal, or names a
+ * @throws IllegalArgumentException Thrown if the field is empty, not valid
hexadecimal, or names a
* value outside {@code [0, U+10FFFF]}.
*/
private static String decode(String hexCodePoints, int lineNumber, String
content) {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiFlags.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiFlags.java
index 10ef91cb7..dfa4316a0 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiFlags.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/EmojiFlags.java
@@ -16,6 +16,7 @@
*/
package opennlp.tools.util.normalizer;
+import java.util.Locale;
import java.util.Optional;
/**
@@ -67,7 +68,7 @@ public final class EmojiFlags {
* (per-token annotation of arbitrary text) can probe safely.
*
* @param symbol The code point sequence of one symbol. Must not be {@code
null}.
- * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code symbol} is {@code null}.
*/
public static boolean isFlag(CharSequence symbol) {
if (symbol == null) {
@@ -85,7 +86,7 @@ public final class EmojiFlags {
* @return The ISO 3166 code, or empty when {@code symbol} is not
flag-shaped at all (no leading
* regional indicator and no tag sequence over U+1F3F4; a lone U+1F3F4
and the ZWJ pirate
* flag are not region flags).
- * @throws IllegalArgumentException if {@code symbol} is {@code null}, or if
it is flag-shaped
+ * @throws IllegalArgumentException Thrown if {@code symbol} is {@code
null}, or if it is flag-shaped
* but malformed: a lone regional indicator, more or fewer than exactly
two regional
* indicators, or a tag sequence that is unterminated, too short for a
subdivision code, not
* letter-led, or followed by trailing content.
@@ -104,7 +105,7 @@ public final class EmojiFlags {
*
* @param symbol The code point sequence of one symbol. Must not be {@code
null}.
* @return The ISO 3166 code, or {@code null} when {@code symbol} is not a
well-formed flag.
- * @throws IllegalArgumentException if {@code symbol} is {@code null}.
+ * @throws IllegalArgumentException Thrown if {@code symbol} is {@code null}.
*/
static String isoRegionOrNull(CharSequence symbol) {
if (symbol == null) {
@@ -168,8 +169,8 @@ public final class EmojiFlags {
}
if (!isTagCharacter(codePoint)) {
return malformed(lenient, "Malformed emoji tag sequence: expected a
tag character or"
- + " CANCEL TAG, got U+" +
Integer.toHexString(codePoint).toUpperCase() + " in: "
- + symbol);
+ + " CANCEL TAG, got U+" +
Integer.toHexString(codePoint).toUpperCase(Locale.ROOT)
+ + " in: " + symbol);
}
decoded.append((char) (codePoint - TAG_OFFSET));
}
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/FullCaseFoldCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/FullCaseFoldCharSequenceNormalizer.java
index a4e65295e..4c79ebb52 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/FullCaseFoldCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/FullCaseFoldCharSequenceNormalizer.java
@@ -38,8 +38,7 @@ import opennlp.tools.util.StringUtil;
* perform: the sharp s (U+00DF) to {@code ss}, the Latin ligatures (for
example U+FB00 to
* {@code ff}), and the Greek and Armenian multi-character folds. It is
therefore an expanding,
* offset-changing transform, so it is offset-aware: {@link
#normalizeAligned(CharSequence)} reports
- * the {@link Alignment} from the folded text back to the input. A single
cursor pass with no regular
- * expression.</p>
+ * the {@link Alignment} from the folded text back to the input.</p>
*
* <p>The {@code S} simple and {@code T} Turkic status mappings are excluded,
so the Turkish and
* Azerbaijani dotless-i rule is not applied here. Input is expected in NFC:
the fold matches
@@ -99,9 +98,9 @@ public final class FullCaseFoldCharSequenceNormalizer
implements OffsetAwareNorm
/**
* {@inheritDoc}
*
- * @throws IllegalArgumentException if {@code text} is {@code null}.
- * @throws IllegalStateException if the bundled {@code CaseFolding.txt}
resource is missing.
- * @throws UncheckedIOException if the bundled {@code CaseFolding.txt}
resource cannot be read.
+ * @throws IllegalArgumentException Thrown if {@code text} is {@code null}.
+ * @throws IllegalStateException Thrown if the bundled {@code
CaseFolding.txt} resource is missing.
+ * @throws UncheckedIOException Thrown if the bundled {@code
CaseFolding.txt} resource cannot be read.
*/
@Override
public CharSequence normalize(CharSequence text) {
@@ -112,9 +111,9 @@ public final class FullCaseFoldCharSequenceNormalizer
implements OffsetAwareNorm
/**
* {@inheritDoc}
*
- * @throws IllegalArgumentException if {@code text} is {@code null}.
- * @throws IllegalStateException if the bundled {@code CaseFolding.txt}
resource is missing.
- * @throws UncheckedIOException if the bundled {@code CaseFolding.txt}
resource cannot be read.
+ * @throws IllegalArgumentException Thrown if {@code text} is {@code null}.
+ * @throws IllegalStateException Thrown if the bundled {@code
CaseFolding.txt} resource is missing.
+ * @throws UncheckedIOException Thrown if the bundled {@code
CaseFolding.txt} resource cannot be read.
*/
@Override
public AlignedText normalizeAligned(CharSequence text) {
@@ -125,8 +124,8 @@ public final class FullCaseFoldCharSequenceNormalizer
implements OffsetAwareNorm
/**
* {@return the folding table parsed from the bundled {@code
CaseFolding.txt} resource}
*
- * @throws IllegalStateException if the resource is missing.
- * @throws UncheckedIOException if the resource cannot be read.
+ * @throws IllegalStateException Thrown if the resource is missing.
+ * @throws UncheckedIOException Thrown if the resource cannot be read.
*/
private static Map<Integer, String> initFoldings() {
return
Map.copyOf(BundledUnicodeData.load(FullCaseFoldCharSequenceNormalizer.class,
RESOURCE,
@@ -141,8 +140,8 @@ public final class FullCaseFoldCharSequenceNormalizer
implements OffsetAwareNorm
*
* @param in the stream to parse, in {@code CaseFolding.txt} format. Must
not be {@code null}.
* @return the mapping from source code point to its full case folding.
- * @throws IOException if the stream cannot be read.
- * @throws IllegalArgumentException if the data is malformed.
+ * @throws IOException Thrown if the stream cannot be read.
+ * @throws IllegalArgumentException Thrown if the data is malformed.
*/
static Map<Integer, String> parse(InputStream in) throws IOException {
final Map<Integer, String> map = new HashMap<>();
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/GermanUmlautCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/GermanUmlautCharSequenceNormalizer.java
index 4a115ded8..dc2f994d8 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/GermanUmlautCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/GermanUmlautCharSequenceNormalizer.java
@@ -25,8 +25,7 @@ package opennlp.tools.util.normalizer;
* <p>This is the correct diacritic fold for German, where the generic
* {@link AccentFoldCharSequenceNormalizer} (which would yield {@code a},
{@code o}, {@code u}) is
* wrong. It is an expanding, offset-changing transform, so like the other
folds it belongs to the
- * derived matching form rather than to anything offset-preserving. A cursor
pass with no regular
- * expression.</p>
+ * derived matching form rather than to anything offset-preserving.</p>
*
* <p>The fold matches the precomposed code points. A base letter followed by
a combining diaeresis
* (for example {@code a} + U+0308) is not a member and passes through
unchanged, so apply NFC
@@ -56,11 +55,13 @@ public class GermanUmlautCharSequenceNormalizer implements
OffsetAwareNormalizer
return INSTANCE;
}
+ /** {@inheritDoc} */
@Override
public CharSequence normalize(CharSequence text) {
return CharClass.substitute(text,
GermanUmlautCharSequenceNormalizer::expansion);
}
+ /** {@inheritDoc} */
@Override
public AlignedText normalizeAligned(CharSequence text) {
return CharClass.substituteAligned(text,
GermanUmlautCharSequenceNormalizer::expansion);
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/InvisibleCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/InvisibleCharSequenceNormalizer.java
index 1e24ef846..b41781587 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/InvisibleCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/InvisibleCharSequenceNormalizer.java
@@ -22,11 +22,10 @@ package opennlp.tools.util.normalizer;
* byte-order mark, zero width space, word joiner, bidi
marks/embeddings/overrides/isolates, the
* invisible math operators, soft hyphen, and the Arabic letter mark).
*
- * <p>Membership is an O(1) {@link CharClass} lookup and removal is a single
cursor pass with no
- * regular expression. The zero width joiner ({@code U+200D}) and non-joiner
({@code U+200C}) are
- * deliberately <em>kept</em>, because they carry meaning in Persian, Indic
scripts, and emoji
- * sequences; so are variation selectors. Use this only for a matching/search
form, not for
- * display.</p>
+ * <p>Membership is an O(1) {@link CharClass} lookup. The zero width joiner
({@code U+200D}) and
+ * non-joiner ({@code U+200C}) are deliberately <em>kept</em>, because they
carry meaning in
+ * Persian, Indic scripts, and emoji sequences; so are variation selectors.
Use this only for a
+ * matching/search form, not for display.</p>
*/
public class InvisibleCharSequenceNormalizer implements OffsetAwareNormalizer {
@@ -64,11 +63,13 @@ public class InvisibleCharSequenceNormalizer implements
OffsetAwareNormalizer {
return INSTANCE;
}
+ /** {@inheritDoc} */
@Override
public CharSequence normalize(CharSequence text) {
return INVISIBLE.removeAll(text);
}
+ /** {@inheritDoc} */
@Override
public AlignedText normalizeAligned(CharSequence text) {
return INVISIBLE.removeAllAligned(text);
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/LineBreakPreservingWhitespaceCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/LineBreakPreservingWhitespaceCharSequenceNormalizer.java
index 6c64ab867..2d9f43f18 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/LineBreakPreservingWhitespaceCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/LineBreakPreservingWhitespaceCharSequenceNormalizer.java
@@ -26,7 +26,7 @@ package opennlp.tools.util.normalizer;
* become a single space, yet a blank line between paragraphs survives as one
newline rather than
* being flattened into the surrounding text. It reuses the cursor based
* {@link CharClass#collapsePreserving(CharSequence, CodePointSet, int)}
engine, so it recognizes the
- * full Unicode {@code White_Space} set with no regular expression.</p>
+ * full Unicode {@code White_Space} set.</p>
*/
public class LineBreakPreservingWhitespaceCharSequenceNormalizer implements
OffsetAwareNormalizer {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NfcCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NfcCharSequenceNormalizer.java
index 4b625f295..16c30240f 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NfcCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NfcCharSequenceNormalizer.java
@@ -22,7 +22,7 @@ import opennlp.tools.util.ParamChecks;
/**
* A {@link CharSequenceNormalizer} that applies Unicode Normalization Form C
(canonical
- * composition, UAX #15).
+ * composition, <a href="https://www.unicode.org/reports/tr15/";>UAX #15</a>).
*
* <p>NFC is the safe, lossless (under canonical equivalence) baseline for
matching: precomposed
* and decomposed spellings of the same text (for example {@code U+00E9}
versus {@code e} plus a
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NfkcCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NfkcCharSequenceNormalizer.java
index 672c39c88..3aec2bf3f 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NfkcCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NfkcCharSequenceNormalizer.java
@@ -22,7 +22,7 @@ import opennlp.tools.util.ParamChecks;
/**
* A {@link CharSequenceNormalizer} that applies Unicode Normalization Form KC
(compatibility
- * composition, UAX #15).
+ * composition, <a href="https://www.unicode.org/reports/tr15/";>UAX #15</a>).
*
* <p>NFKC folds compatibility variants to their canonical form: fullwidth and
halfwidth letters,
* the {@code U+FB01} ligature to {@code fi}, and super/subscript digits to
plain digits. It is
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfile.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfile.java
index 07697e2d8..5f0d1c855 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfile.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfile.java
@@ -28,14 +28,14 @@ import opennlp.tools.util.ParamChecks;
* diacritic fold appropriate for that language (if any).
*
* <p>The {@code accentFold} normalizer is the language's diacritic transform
for a matching form, or
- * {@code null} when folding is not appropriate. It is the generic
- * {@link AccentFoldCharSequenceNormalizer} for English and the major Romance
languages (where
- * accented letters are matching variants of their base letter), the
German-specific
- * {@link GermanUmlautCharSequenceNormalizer} (a-umlaut to {@code ae}, eszett
to {@code ss}, ...) for
- * German, and {@code null} where diacritics mark distinct letters (the Nordic
languages and the
- * non-Latin scripts), because folding there is language-wrong. This is a
search-recall choice, not a
- * statement of linguistic correctness; callers can build a {@link
TermAnalyzer} directly to
- * override it.</p>
+ * {@code null} when folding is not appropriate. In the profiles of {@link
NormalizationProfiles} it
+ * is the generic {@link AccentFoldCharSequenceNormalizer} for English,
Catalan, French, Italian,
+ * Portuguese and Spanish (where accented letters are matching variants of
their base letter), the
+ * German-specific {@link GermanUmlautCharSequenceNormalizer} (a-umlaut to
{@code ae}, eszett to
+ * {@code ss}, ...) for German, and {@code null} for every other language,
including those where
+ * diacritics mark distinct letters (such as the Nordic languages) and the
non-Latin scripts. This
+ * is a search-recall choice, not a statement of linguistic correctness;
callers can build a
+ * {@link TermAnalyzer} directly to override it.</p>
*
* @param language The language, as an ISO 639-3 code (for example
{@code "eng"}). Must
* not be {@code null} or blank.
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfiles.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfiles.java
index 71411a5f1..c0ece8806 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfiles.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfiles.java
@@ -27,13 +27,12 @@ import opennlp.tools.util.LanguageCodeValidator;
import opennlp.tools.util.ParamChecks;
/**
- * A registry of {@link NormalizationProfile}s by language, with
detection-based fallback. This is
- * the language dispatch the design note calls for: pick the profile for a
requested language, or
- * detect the language with a {@link LanguageDetector} when it is unspecified.
The covered languages
- * are the Snowball stemmer algorithms that name a natural language -- every
- * {@link SnowballStemmer.ALGORITHM} except {@code PORTER}, which is an
English-only algorithm
- * variant rather than a distinct language. Several codes can map to one
algorithm (the three
- * Norwegian written standards all use {@code NORWEGIAN}).
+ * A registry of {@link NormalizationProfile}s by language, with
detection-based fallback: pick the
+ * profile for a requested language, or detect the language with a {@link
LanguageDetector} when it
+ * is unspecified. The covered languages are the Snowball stemmer algorithms
that name a natural
+ * language -- every {@link SnowballStemmer.ALGORITHM} except {@code PORTER},
which is an
+ * English-only algorithm variant rather than a distinct language. Several
codes can map to one
+ * algorithm (the three Norwegian written standards all use {@code NORWEGIAN}).
*
* <p>Profiles are keyed by ISO 639-3 code (what {@link LanguageDetector}
produces);
* {@link #forLanguage(String)} also accepts ISO 639-1 two-letter codes and
the ISO 639-2
@@ -48,9 +47,9 @@ public final class NormalizationProfiles {
private static Map<String, NormalizationProfile> build() {
final Map<String, NormalizationProfile> map = new HashMap<>();
- // The generic accent fold is used for English and the major Romance
languages, German uses its
- // own ae/oe/ue/ss fold, and folding is disabled elsewhere (Nordic,
non-Latin) where diacritics
- // mark distinct letters.
+ // The generic accent fold is used for English, Catalan, French, Italian,
Portuguese and
+ // Spanish, German uses its own ae/oe/ue/ss fold, and folding is disabled
for all other
+ // languages.
final CharSequenceNormalizer latin =
AccentFoldCharSequenceNormalizer.getInstance();
final CharSequenceNormalizer german =
GermanUmlautCharSequenceNormalizer.getInstance();
add(map, "ara", SnowballStemmer.ALGORITHM.ARABIC, null);
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/ParagraphPreservingWhitespaceCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/ParagraphPreservingWhitespaceCharSequenceNormalizer.java
index 81e3b8185..03363c236 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/ParagraphPreservingWhitespaceCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/ParagraphPreservingWhitespaceCharSequenceNormalizer.java
@@ -25,8 +25,7 @@ package opennlp.tools.util.normalizer;
* <p>This is the form wanted before sentence detection on Gutenberg-style or
other fixed-width
* plain text: intra-paragraph line wraps become spaces, blank-line paragraph
boundaries survive as
* newlines. It reuses the cursor based {@link
CharClass#collapseParagraphPreserving(CharSequence,
- * CodePointSet, int)} engine, so it recognizes the full Unicode {@code
White_Space} set with no
- * regular expression.</p>
+ * CodePointSet, int)} engine, so it recognizes the full Unicode {@code
White_Space} set.</p>
*
* <p>For display-oriented text where every line break should survive, use
* {@link LineBreakPreservingWhitespaceCharSequenceNormalizer} instead. For
raw markdown with lists,
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/QuoteCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/QuoteCharSequenceNormalizer.java
index f88221e5e..45ca730be 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/QuoteCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/QuoteCharSequenceNormalizer.java
@@ -22,8 +22,7 @@ package opennlp.tools.util.normalizer;
*
* <p>This is high value for matching, since curly quotes, guillemets, and
fullwidth quotes
* otherwise prevent {@code "don't"} from matching {@code "don" + U+2019 +
"t"}. It is built from
- * two {@link CharClass} sets, so membership is O(1) and scanning is a single
cursor pass with no
- * regular expression. ASCII quotes are left unchanged.</p>
+ * two {@link CharClass} sets, so membership is O(1). ASCII quotes are left
unchanged.</p>
*/
public class QuoteCharSequenceNormalizer implements OffsetAwareNormalizer {
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/TextNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/TextNormalizer.java
index 78de56fca..d90c66e1b 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/TextNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/TextNormalizer.java
@@ -25,9 +25,10 @@ import opennlp.tools.util.ParamChecks;
* Entry point for composing normalization steps into a single {@link
CharSequenceNormalizer}.
*
* <p>Use {@link #builder()} to assemble a chain, or {@link #defaultChain()}
for a conservative,
- * ready-made chain. Steps run in the order they are added. Each step is a
shared, stateless
- * normalizer; the built result is an {@link AggregateCharSequenceNormalizer}
that applies them
- * in sequence.</p>
+ * ready-made chain. Steps run in the order they are added. Each built-in step
is a shared,
+ * stateless normalizer; steps added through {@link
Builder#with(CharSequenceNormalizer)} are used
+ * as given. The built result is an {@link AggregateCharSequenceNormalizer}
that applies them in
+ * sequence.</p>
*
* <pre>{@code
* CharSequenceNormalizer n = TextNormalizer.builder()
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UrlCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UrlCharSequenceNormalizer.java
index 630410c8c..1b2348252 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UrlCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/UrlCharSequenceNormalizer.java
@@ -56,7 +56,10 @@ public class UrlCharSequenceNormalizer implements
CharSequenceNormalizer {
private static final String HTTP = "http";
private static final String HTTPS = "https";
- /** The characters a scheme continues with after its first letter, per RFC
3986. */
+ /**
+ * The characters a scheme continues with after its first letter, per
+ * <a href="https://www.rfc-editor.org/rfc/rfc3986";>RFC 3986</a>.
+ */
private static final CodePointSet SCHEME_BODY =
AsciiChars.ALPHANUMERIC.union(CodePointSet.of('+', '-', '.'));
diff --git
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/WhitespaceCharSequenceNormalizer.java
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/WhitespaceCharSequenceNormalizer.java
index 2afbc2d77..cb34b60e9 100644
---
a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/WhitespaceCharSequenceNormalizer.java
+++
b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/WhitespaceCharSequenceNormalizer.java
@@ -23,7 +23,7 @@ package opennlp.tools.util.normalizer;
* <p>Unlike a {@code \s} regular expression, this recognizes the full Unicode
{@code White_Space}
* set (no-break space, ideographic space, the typographic spaces, line and
paragraph separators,
* and so on), so spacing copied from the web, PDFs, or non-Latin sources
normalizes consistently.
- * It is the Unicode-aware, regex-free counterpart to {@link
ShrinkCharSequenceNormalizer}.</p>
+ * It is the Unicode-aware counterpart to {@link
ShrinkCharSequenceNormalizer}.</p>
*/
public class WhitespaceCharSequenceNormalizer implements OffsetAwareNormalizer
{
