This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch OPENNLP-1893-hunspell in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit d020e16eeb484e5ea70bf50549527853525ecc43 Author: Kristian Rickert <[email protected]> AuthorDate: Tue Jul 21 06:39:17 2026 -0400 OPENNLP-1893: Apply the review-convention pass: factual license prose, split null contracts, thread-safety annotations --- .../dev => dev}/README-hunspell-dictionaries.md | 4 +- .../dev => dev}/download-hunspell-dictionary.sh | 7 +- .../tools/stemmer/hunspell/HunspellDictionary.java | 97 +++++++++------------- .../tools/stemmer/hunspell/HunspellStemmer.java | 31 +++---- .../stemmer/hunspell/HunspellStemmerFactory.java | 3 +- .../hunspell/HunspellRealDictionaryTest.java | 12 ++- .../hunspell/HunspellStemmerFactoryTest.java | 19 +++-- .../stemmer/hunspell/HunspellStemmerTest.java | 22 +++-- opennlp-docs/src/docbkx/stemmer.xml | 10 +-- 9 files changed, 103 insertions(+), 102 deletions(-) diff --git a/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md b/dev/README-hunspell-dictionaries.md similarity index 85% rename from opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md rename to dev/README-hunspell-dictionaries.md index 440814163..34cb5c278 100644 --- a/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md +++ b/dev/README-hunspell-dictionaries.md @@ -17,11 +17,11 @@ # Hunspell dictionaries for the affix stemmer -The Hunspell stemmer (`opennlp.tools.stemmer.hunspell`) is a clean-room engine over the documented Hunspell dictionary format: a `.dic` word list plus its `.aff` affix companion, both supplied by the user. Apache OpenNLP bundles no dictionary data, so the dictionaries' own licenses never attach to the library; whichever dictionary you download, its license is stated in the readme shipped alongside it and is yours to comply with. +The Hunspell stemmer (`opennlp.tools.stemmer.hunspell`) implements the documented Hunspell dictionary format: a `.dic` word list plus its `.aff` affix companion, both supplied by the user. Apache OpenNLP bundles no dictionary data; whichever dictionary you download, its license is stated in the readme shipped alongside it. ## Where dictionaries come from -The LibreOffice project maintains a large collection of Hunspell dictionaries, one directory per language, at `github.com/LibreOffice/dictionaries`. Licenses differ per dictionary, which is exactly why nothing is bundled: for example, the `en_US` dictionary derives from SCOWL and states its terms in `README_en_US.txt` in the same directory. Many other sources work too; the engine only cares that the pair follows the Hunspell format. +The LibreOffice project maintains a large collection of Hunspell dictionaries, one directory per language, at `github.com/LibreOffice/dictionaries`. Licenses differ per dictionary, which is why nothing is bundled: for example, the `en_US` dictionary derives from SCOWL and states its terms in `README_en_US.txt` in the same directory. Many other sources work too; the engine only cares that the pair follows the Hunspell format. The helper next to this file fetches a pair together with its readme files: diff --git a/opennlp-core/opennlp-runtime/dev/download-hunspell-dictionary.sh b/dev/download-hunspell-dictionary.sh similarity index 92% rename from opennlp-core/opennlp-runtime/dev/download-hunspell-dictionary.sh rename to dev/download-hunspell-dictionary.sh index 5ed95511b..afaff59e6 100755 --- a/opennlp-core/opennlp-runtime/dev/download-hunspell-dictionary.sh +++ b/dev/download-hunspell-dictionary.sh @@ -16,10 +16,9 @@ # Fetches one Hunspell dictionary pair (.aff and .dic) plus its license/readme files # from the LibreOffice dictionaries collection. Each dictionary carries its own -# license, stated in the readme files this script downloads alongside it, and you -# accept that license by using the dictionary. Apache OpenNLP bundles no dictionary -# data. See README-hunspell-dictionaries.md in this directory for the Java steps that -# follow. +# license, stated in the readme files this script downloads alongside it. Apache +# OpenNLP bundles no dictionary data. See README-hunspell-dictionaries.md in this +# directory for the Java steps that follow. set -euo pipefail diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java index 7197185ff..5970304a4 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java @@ -17,11 +17,12 @@ package opennlp.tools.stemmer.hunspell; -import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; import java.nio.charset.Charset; +import java.nio.charset.IllegalCharsetNameException; import java.nio.charset.StandardCharsets; +import java.nio.charset.UnsupportedCharsetException; import java.nio.file.Files; import java.nio.file.Path; import java.util.ArrayList; @@ -30,14 +31,14 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import opennlp.tools.commons.ThreadSafe; import opennlp.tools.util.StringUtil; /** * An immutable, in-memory Hunspell-format dictionary: the word list of a {@code .dic} * file and the prefix and suffix rules of its {@code .aff} companion, loaded from - * user-supplied files. The engine is a clean-room implementation of the documented - * format; no dictionary data is bundled, so the dictionaries' own licenses never attach - * to this library. + * user-supplied files. The engine implements the documented format directly; no + * dictionary data is bundled, dictionaries are supplied by the user. * * <p>Supported affix features: {@code PFX} and {@code SFX} rules with strip strings, * character-class conditions, and cross-product combination of one prefix with one @@ -63,8 +64,8 @@ import opennlp.tools.util.StringUtil; * * @see HunspellStemmer * @see HunspellStemmerFactory - * @since 3.0.0 */ +@ThreadSafe public final class HunspellDictionary { /** @@ -93,17 +94,6 @@ public final class HunspellDictionary { } } - private final Map<String, List<int[]>> entries; - private final List<Affix> prefixes; - private final List<Affix> suffixes; - private final Map<Character, List<Affix>> suffixesByLast; - private final List<Affix> suffixesWithoutMaterial; - private final Map<Character, List<Affix>> prefixesByFirst; - private final List<Affix> prefixesWithoutMaterial; - private final int compoundFlag; - private final int compoundBegin; - private final int compoundEnd; - private final int compoundMin; /** The place a part takes in a compound, deciding which positional flag admits it. */ enum CompoundPosition { /** The first part. */ @@ -114,6 +104,18 @@ public final class HunspellDictionary { END } + /** The shared empty bucket answered for characters no affix rule is keyed under. */ + private static final List<Affix> NO_AFFIXES = List.of(); + + private final Map<String, List<int[]>> entries; + private final Map<Character, List<Affix>> suffixesByLast; + private final List<Affix> suffixesWithoutMaterial; + private final Map<Character, List<Affix>> prefixesByFirst; + private final List<Affix> prefixesWithoutMaterial; + private final int compoundFlag; + private final int compoundBegin; + private final int compoundEnd; + private final int compoundMin; private final int needAffix; private final int onlyInCompound; private final int forbiddenWord; @@ -143,8 +145,6 @@ public final class HunspellDictionary { this.checkCompoundCase = affix.checkCompoundCase; this.checkCompoundTriple = affix.checkCompoundTriple; this.entries = entries; - this.prefixes = List.copyOf(affix.prefixes); - this.suffixes = List.copyOf(affix.suffixes); // Undoing a suffix requires the word to end with the rule's affix material, so // only rules whose material ends in the word's last character can ever apply; // the same holds for prefixes and the first character. Bucketing by that @@ -153,7 +153,7 @@ public final class HunspellDictionary { // everywhere. this.suffixesByLast = new HashMap<>(); this.suffixesWithoutMaterial = new ArrayList<>(); - for (final Affix suffix : suffixes) { + for (final Affix suffix : affix.suffixes) { final String material = suffix.affix(); if (material.isEmpty()) { suffixesWithoutMaterial.add(suffix); @@ -164,7 +164,7 @@ public final class HunspellDictionary { } this.prefixesByFirst = new HashMap<>(); this.prefixesWithoutMaterial = new ArrayList<>(); - for (final Affix prefix : prefixes) { + for (final Affix prefix : affix.prefixes) { final String material = prefix.affix(); if (material.isEmpty()) { prefixesWithoutMaterial.add(prefix); @@ -186,8 +186,11 @@ public final class HunspellDictionary { */ public static HunspellDictionary load(Path affixFile, Path dictionaryFile) throws IOException { - if (affixFile == null || dictionaryFile == null) { - throw new IllegalArgumentException("affixFile and dictionaryFile must not be null"); + if (affixFile == null) { + throw new IllegalArgumentException("affixFile must not be null"); + } + if (dictionaryFile == null) { + throw new IllegalArgumentException("dictionaryFile must not be null"); } try (InputStream affix = Files.newInputStream(affixFile); InputStream dictionary = Files.newInputStream(dictionaryFile)) { @@ -208,14 +211,17 @@ public final class HunspellDictionary { */ public static HunspellDictionary load(InputStream affixStream, InputStream dictionaryStream) throws IOException { - if (affixStream == null || dictionaryStream == null) { - throw new IllegalArgumentException("streams must not be null"); + if (affixStream == null) { + throw new IllegalArgumentException("affixStream must not be null"); + } + if (dictionaryStream == null) { + throw new IllegalArgumentException("dictionaryStream must not be null"); } - final byte[] affixBytes = readAll(affixStream); + final byte[] affixBytes = affixStream.readAllBytes(); final Charset charset = declaredCharset(affixBytes); final AffixFile affix = parseAffix(new String(affixBytes, charset)); final Map<String, List<int[]>> entries = parseWordList( - new String(readAll(dictionaryStream), charset), affix.flagMode, + new String(dictionaryStream.readAllBytes(), charset), affix.flagMode, affix.flagAliases); return new HunspellDictionary(entries, affix); } @@ -230,18 +236,6 @@ public final class HunspellDictionary { return entries.get(word); } - /** @return The prefix rules. */ - List<Affix> prefixes() { - return prefixes; - } - - /** @return The suffix rules. */ - List<Affix> suffixes() { - return suffixes; - } - - private static final List<Affix> NO_AFFIXES = List.of(); - /** * The suffix rules whose affix material ends in the given character, which are the * only material-bearing rules that can be undone from a word ending in it. @@ -550,23 +544,6 @@ public final class HunspellDictionary { return circumfix != 0 && affix.allowsContinuation(circumfix); } - /** - * Reads a stream fully into memory. The stream is not closed. - * - * @param in The stream to drain. - * @return All bytes the stream produced. Never {@code null}. - * @throws IOException Thrown if reading fails. - */ - private static byte[] readAll(InputStream in) throws IOException { - final ByteArrayOutputStream out = new ByteArrayOutputStream(); - final byte[] buffer = new byte[8192]; - int read; - while ((read = in.read(buffer)) >= 0) { - out.write(buffer, 0, read); - } - return out.toByteArray(); - } - /** * Finds the {@code SET} declaration by scanning the raw affix bytes as ASCII, which * is safe because the declaration itself is ASCII in every supported encoding. Both @@ -584,7 +561,7 @@ public final class HunspellDictionary { final String name = trim(trimmed.substring(4)); try { return Charset.forName(name); - } catch (RuntimeException e) { + } catch (IllegalCharsetNameException | UnsupportedCharsetException e) { throw new IOException("unsupported SET encoding: " + name, e); } } @@ -852,7 +829,13 @@ public final class HunspellDictionary { } } if (!flagAliases.isEmpty() && isCount(flagRun)) { - final int alias = Integer.parseInt(flagRun); + final int alias; + try { + alias = Integer.parseInt(flagRun); + } catch (NumberFormatException e) { + throw new IOException("malformed flag alias '" + flagRun + "' at line " + + (i + 1), e); + } if (alias < 1 || alias > flagAliases.size()) { throw new IOException("flag alias " + alias + " at line " + (i + 1) + " is outside the AF table of " + flagAliases.size() + " aliases"); diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java index 8703217ad..1f42f3686 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java @@ -22,6 +22,7 @@ import java.util.LinkedHashSet; import java.util.List; import java.util.Set; +import opennlp.tools.commons.ThreadSafe; import opennlp.tools.stemmer.Stemmer; import opennlp.tools.stemmer.hunspell.HunspellDictionary.Affix; import opennlp.tools.stemmer.hunspell.HunspellDictionary.CompoundPosition; @@ -45,10 +46,19 @@ import opennlp.tools.util.StringUtil; * implementation reads only the immutable dictionary state, so a single instance is * safe to share between threads.</p> * - * @since 3.0.0 */ +@ThreadSafe public class HunspellStemmer implements Stemmer { + /** + * The most part-licensing attempts one decomposition search may spend. Compounding + * searches every split of every tail, which on adversarial input with a + * one-character minimum part length grows without useful bound; the budget stops + * the search there, missing analyses rather than stalling, in line with the + * engine's fail-closed posture. + */ + private static final int PART_CHECK_BUDGET = 2048; + private final HunspellDictionary dictionary; /** @@ -93,7 +103,7 @@ public class HunspellStemmer implements Stemmer { if (analyses.isEmpty()) { return List.of(surface); } - return List.copyOf(new ArrayList<CharSequence>(analyses)); + return List.copyOf(analyses); } /** @@ -104,7 +114,7 @@ public class HunspellStemmer implements Stemmer { * @param surface The surface form. * @return The variants in analysis order. Never {@code null} or empty. */ - private static List<String> variants(String surface) { + private List<String> variants(String surface) { final String lowered = StringUtil.toLowerCase(surface); return lowered.equals(surface) ? List.of(surface) : List.of(surface, lowered); } @@ -141,15 +151,6 @@ public class HunspellStemmer implements Stemmer { } } - /** - * The most part-licensing attempts one decomposition search may spend. Compounding - * searches every split of every tail, which on adversarial input with a - * one-character minimum part length grows without useful bound; the budget stops - * the search there, missing analyses rather than stalling, in line with the - * engine's fail-closed posture. - */ - private static final int PART_CHECK_BUDGET = 2048; - /** * Decomposes a word into listed compound parts when the affix analysis found * nothing: the first part must be admitted to open a compound, every further part @@ -417,7 +418,7 @@ public class HunspellStemmer implements Stemmer { * @param suffix Whether the rule is a suffix rule. * @return The candidate stem, or {@code null} when the rule does not apply. */ - private static String removeAffixInCompound(String part, Affix affix, boolean suffix) { + private String removeAffixInCompound(String part, Affix affix, boolean suffix) { if (affix.affix().isEmpty() && affix.strip().isEmpty()) { return affix.condition().matches(part) ? part : null; } @@ -561,7 +562,7 @@ public class HunspellStemmer implements Stemmer { * @param suffix The rule to undo. * @return The candidate stem, or {@code null} when the rule does not apply. */ - private static String removeSuffix(String word, Affix suffix) { + private String removeSuffix(String word, Affix suffix) { final String affix = suffix.affix(); final String strip = suffix.strip(); if (affix.isEmpty() && strip.isEmpty() || !word.endsWith(affix) @@ -583,7 +584,7 @@ public class HunspellStemmer implements Stemmer { * @param prefix The rule to undo. * @return The candidate stem, or {@code null} when the rule does not apply. */ - private static String removePrefix(String word, Affix prefix) { + private String removePrefix(String word, Affix prefix) { final String affix = prefix.affix(); final String strip = prefix.strip(); if (affix.isEmpty() && strip.isEmpty() || !word.startsWith(affix) diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactory.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactory.java index 21d66be2b..840c7a2c0 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactory.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactory.java @@ -17,6 +17,7 @@ package opennlp.tools.stemmer.hunspell; +import opennlp.tools.commons.ThreadSafe; import opennlp.tools.stemmer.Stemmer; import opennlp.tools.stemmer.StemmerFactory; @@ -26,8 +27,8 @@ import opennlp.tools.stemmer.StemmerFactory; * * <p>The factory is immutable and safe to share across threads.</p> * - * @since 3.0.0 */ +@ThreadSafe public class HunspellStemmerFactory implements StemmerFactory { private final HunspellDictionary dictionary; diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java index d6d12faa4..dcc7e997f 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java @@ -75,7 +75,8 @@ public class HunspellRealDictionaryTest { void testGermanInflections() throws IOException { final HunspellStemmer stemmer = loadOrSkip("de_DE_frami"); Assertions.assertEquals("Kind", stemmer.stem("Kinder").toString()); - Assertions.assertEquals("Haus", stemmer.stem("Häuser").toString()); + // Haeuser, written with a-umlaut, stems to Haus + Assertions.assertEquals("Haus", stemmer.stem("H\u00E4user").toString()); Assertions.assertEquals("schnell", stemmer.stem("schnellsten").toString()); } @@ -84,7 +85,8 @@ public class HunspellRealDictionaryTest { final HunspellStemmer stemmer = loadOrSkip("de_DE_frami"); // the exact part spellings follow the dictionary's own entries and may shift // between revisions; that ordinary compounds decompose at all must not - Assertions.assertTrue(stemmer.stemAll("Haustür").size() >= 2); + // Haustuer, written with u-umlaut, is Haus + Tuer + Assertions.assertTrue(stemmer.stemAll("Haust\u00FCr").size() >= 2); Assertions.assertTrue(stemmer.stemAll("Kinderzimmer").size() >= 2); Assertions.assertTrue(stemmer.stemAll("Abbildungsverzeichnis").size() >= 2); } @@ -92,8 +94,10 @@ public class HunspellRealDictionaryTest { @Test void testHungarianInflections() throws IOException { final HunspellStemmer stemmer = loadOrSkip("hu_HU"); - Assertions.assertEquals("kutya", stemmer.stem("kutyák").toString()); + // kutyak, written with a-acute, is the plural of kutya + Assertions.assertEquals("kutya", stemmer.stem("kuty\u00E1k").toString()); Assertions.assertEquals("asztal", stemmer.stem("asztalon").toString()); - Assertions.assertEquals("könyv", stemmer.stem("könyveket").toString()); + // konyveket, written with o-umlaut, is an inflected form of konyv + Assertions.assertEquals("k\u00F6nyv", stemmer.stem("k\u00F6nyveket").toString()); } } diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactoryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactoryTest.java index e19ce1072..5acbb793c 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactoryTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerFactoryTest.java @@ -167,13 +167,20 @@ public class HunspellStemmerFactoryTest { } /** - * Verifies that the file-based entry point rejects {@code null} paths with the - * documented exception instead of failing later with an obscure error. + * Verifies that the file-based entry point rejects each {@code null} path with the + * documented exception naming the offending argument. + * + * @param tempDir A scratch directory managed by the test framework. */ @Test - void testNullPathsAreRejected() { - final IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class, - () -> HunspellDictionary.load((Path) null, (Path) null)); - Assertions.assertEquals("affixFile and dictionaryFile must not be null", e.getMessage()); + void testNullPathsAreRejected(@TempDir Path tempDir) { + final Path present = tempDir.resolve("present.aff"); + IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class, + () -> HunspellDictionary.load(null, present)); + Assertions.assertEquals("affixFile must not be null", e.getMessage()); + + e = Assertions.assertThrows(IllegalArgumentException.class, + () -> HunspellDictionary.load(present, null)); + Assertions.assertEquals("dictionaryFile must not be null", e.getMessage()); } } diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java index d4fe1a196..f386ba5d9 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java @@ -19,6 +19,7 @@ package opennlp.tools.stemmer.hunspell; import java.io.ByteArrayInputStream; import java.io.IOException; +import java.io.InputStream; import java.nio.charset.Charset; import java.nio.charset.StandardCharsets; import java.util.List; @@ -532,8 +533,7 @@ public class HunspellStemmerTest { new ByteArrayInputStream("SFX S 0 s [a\n".getBytes(StandardCharsets.UTF_8)), new ByteArrayInputStream("0\n".getBytes(StandardCharsets.UTF_8)))); Assertions.assertThrows(IllegalArgumentException.class, - () -> HunspellDictionary.load((java.io.InputStream) null, - (java.io.InputStream) null)); + () -> HunspellDictionary.load((InputStream) null, (InputStream) null)); Assertions.assertThrows(IllegalArgumentException.class, () -> new HunspellStemmer(null)); Assertions.assertThrows(IllegalArgumentException.class, @@ -629,10 +629,10 @@ public class HunspellStemmerTest { Assertions.assertTrue(HunspellDictionary.hasFlag(emoji.lookup("walk"), 0x1F600)); Assertions.assertFalse(HunspellDictionary.hasFlag(emoji.lookup("walk"), 0xD83D)); - final HunspellStemmer stemmer = new HunspellStemmer(load( + final HunspellStemmer supplementaryFlags = new HunspellStemmer(load( "FLAG UTF-8\nSFX \uD83D\uDE00 Y 1\nSFX \uD83D\uDE00 0 s .\n", "1\nwalk/\uD83D\uDE00\n")); - Assertions.assertEquals("walk", stemmer.stem("walks").toString()); + Assertions.assertEquals("walk", supplementaryFlags.stem("walks").toString()); } /** @@ -700,18 +700,24 @@ public class HunspellStemmerTest { /** * Verifies that an alias reference outside the AF table fails loud with the line - * and the table size, instead of silently flagging the entry with nothing. - * - * @throws IOException Thrown if the affix fixture fails to load. + * and the table size, instead of silently flagging the entry with nothing, and that + * a digit run too large for an alias number fails loud as well. */ @Test void testAliasReferenceOutsideTheTableFailsLoud() { - final IOException e = Assertions.assertThrows(IOException.class, () -> load( + IOException e = Assertions.assertThrows(IOException.class, () -> load( "AF 1\nAF S # 1\nSFX S Y 1\nSFX S 0 s .\n", "1\nwalk/2\n")); Assertions.assertEquals( "flag alias 2 at line 2 is outside the AF table of 1 aliases", e.getMessage()); + + e = Assertions.assertThrows(IOException.class, () -> load( + "AF 1\nAF S # 1\nSFX S Y 1\nSFX S 0 s .\n", + "1\nwalk/99999999999999999999\n")); + Assertions.assertEquals( + "malformed flag alias '99999999999999999999' at line 2", + e.getMessage()); } /** diff --git a/opennlp-docs/src/docbkx/stemmer.xml b/opennlp-docs/src/docbkx/stemmer.xml index 8dc078d97..0c999099b 100644 --- a/opennlp-docs/src/docbkx/stemmer.xml +++ b/opennlp-docs/src/docbkx/stemmer.xml @@ -73,11 +73,11 @@ new CachingStemmer(factory).stem("running"); // "run"]]> <section xml:id="tools.stemmer.hunspell"> <title>Hunspell dictionaries</title> <para> - <code>opennlp.tools.stemmer.hunspell</code> is a clean-room engine over the - documented Hunspell dictionary format: a user-supplied + <code>opennlp.tools.stemmer.hunspell</code> implements the documented + Hunspell dictionary format: a user-supplied <code>.aff</code> affix file and its <code>.dic</code> word list. OpenNLP - bundles no dictionary data, so each dictionary's own license stays with - the files you download. The dictionary is immutable and safe to share; + bundles no dictionary data; dictionaries are downloaded separately, and + each states its own license. The dictionary is immutable and safe to share; <code>HunspellStemmerFactory</code> hands out a fresh stemmer per call. <code>HunspellManualExampleTest</code> asserts the behavior shown here. <programlisting language="java"><![CDATA[ @@ -92,7 +92,7 @@ stemmer.stem("table"); // "table" (unknown vocabulary is unchanged)]]> published one, and asserts the same stems for <code>workers</code> and <code>worker</code>. Acquisition helpers and the supported affix feature set live in - <code>opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md</code>. + <code>dev/README-hunspell-dictionaries.md</code>. </para> </section> </chapter>
