This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch OPENNLP-1893-hunspell in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 369d61934c05bb70a942a11da90145756e3e5bfa Author: Kristian Rickert <[email protected]> AuthorDate: Mon Jul 20 00:43:29 2026 -0400 OPENNLP-1893: Honor the blocking flags, circumfixes, and compound positioning A NEEDAFFIX (or PSEUDOROOT) entry is a virtual stem that exists only to be affixed, an ONLYINCOMPOUND entry appears only inside compounds, and a FORBIDDENWORD entry is listed to be blocked; none of them is a standalone analysis anymore, per homonym flag set, and an affix carrying NEEDAFFIX among its continuation classes yields no single-removal analysis while its twofold and cross-product removals stand, the other affix being exactly the further one required. A cross-product now also requires both removed affixes' flags in the same homonym's flag set, and CIRCUMFIX binds marked prefix and suffix halves to one another, so neither half analyzes alone and a marked half never combines with an unmarked affix. Decomposition grows from two verbatim parts to the compound machinery the published German dictionary actually uses: any number of parts under the positional COMPOUNDBEGIN/COMPOUNDMIDDLE/COMPOUNDEND flags and COMPOUNDWORDMAX, parts standing on an entry plus one affix with COMPOUNDPERMITFLAG required at internal boundaries and COMPOUNDFORBIDFLAG barring marked forms, zero and dash linking suffixes included, an uppercased retry for capitalized entries spelled lowercase inside a compound, and the CHECKCOMPOUNDDUP, CHECKCOMPOUNDCASE, and CHECKCOMPOUNDTRIPLE junction guards, case judged against the original surface. A listed forbidden word never decomposes, and a fixed part-licensing budget keeps adversarial input bounded, missing analyses rather than stalling. Abbildungsverzeichnis, Haustuer, and Kinderzimmer now decompose against de_DE_frami at 137k words/s single-threaded. An opt-in test class checks everyday morphology against downloaded dictionaries under -Dopennlp.hunspell.dict.dir; nothing is bundled. --- .../dev/README-hunspell-dictionaries.md | 11 +- .../tools/stemmer/hunspell/HunspellDictionary.java | 395 ++++++++++++++++++--- .../tools/stemmer/hunspell/HunspellStemmer.java | 356 +++++++++++++++++-- .../hunspell/HunspellRealDictionaryTest.java | 99 ++++++ .../stemmer/hunspell/HunspellStemmerTest.java | 379 ++++++++++++++++++++ 5 files changed, 1159 insertions(+), 81 deletions(-) diff --git a/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md b/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md index 31edf4fe0..457fd01ee 100644 --- a/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md +++ b/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md @@ -50,6 +50,15 @@ What `stem` evaluates to is decided by the dictionary you loaded, and this proje The dictionary is immutable and safe to share between threads; the factory hands out a fresh stemmer per call, so each thread takes its own from `newStemmer()`. A dictionary that declares a non-UTF-8 encoding through the `SET` directive in its `.aff` file is decoded accordingly; nothing needs converting beforehand. +## Testing against real dictionaries + +The in-tree tests run against project-authored fixtures only. An opt-in test class, `HunspellRealDictionaryTest`, additionally checks everyday morphology against published dictionaries when pointed at a directory of `<name>.aff`/`<name>.dic` pairs (each test skips when its pair is absent): + +``` +./mvnw test -pl opennlp-core/opennlp-runtime -Dtest=HunspellRealDictionaryTest \ + -Dopennlp.hunspell.dict.dir=/tmp/hunspell-dicts +``` + ## What the engine supports -Supported affix features: `PFX` and `SFX` rules with strip strings, character-class conditions, cross-product combination of one prefix with one suffix, twofold suffixes through continuation classes, `FLAG` modes `char`, `UTF-8`, `long`, and `num`, and the `SET` encoding declaration. Compounding and conversion tables are not interpreted; rules that use them simply do not fire, so unsupported analyses are missed rather than invented. A malformed `.aff` file fails loudly at load time with [...] +Supported affix features: `PFX` and `SFX` rules with strip strings, character-class conditions, cross-product combination of one prefix with one suffix, twofold suffixes through continuation classes, `FLAG` modes `char`, `UTF-8`, `long`, and `num`, the `AF` flag alias table, the `SET` encoding declaration, compound decomposition under `COMPOUNDFLAG`, the positional `COMPOUNDBEGIN`/`COMPOUNDMIDDLE`/`COMPOUNDEND` flags, `COMPOUNDMIN`, `COMPOUNDWORDMAX`, `COMPOUNDPERMITFLAG`, `COMPOUNDFORBI [...] diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java index 0f51d0e24..7197185ff 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java @@ -43,10 +43,21 @@ import opennlp.tools.util.StringUtil; * character-class conditions, and cross-product combination of one prefix with one * suffix; twofold suffixes through the continuation classes on suffix rules; * {@code FLAG} modes {@code char} (default), {@code UTF-8}, {@code long}, and - * {@code num}; the {@code SET} encoding declaration. Compounding and conversion tables - * are not - * interpreted in this version; rules using them simply do not fire, so unsupported - * analyses are missed rather than invented.</p> + * {@code num}; the {@code AF} flag alias table; the {@code SET} encoding declaration; + * compound decomposition under {@code COMPOUNDFLAG}, the positional + * {@code COMPOUNDBEGIN}/{@code COMPOUNDMIDDLE}/{@code COMPOUNDEND} flags, + * {@code COMPOUNDMIN}, {@code COMPOUNDWORDMAX}, {@code COMPOUNDPERMITFLAG}, + * {@code COMPOUNDFORBIDFLAG}, and the {@code CHECKCOMPOUNDDUP}, + * {@code CHECKCOMPOUNDCASE}, and {@code CHECKCOMPOUNDTRIPLE} declarations, with + * compound parts standing on their entries alone or on an entry plus one affix; the + * blocking flags + * {@code NEEDAFFIX} (with its historical alias {@code PSEUDOROOT}), + * {@code ONLYINCOMPOUND}, and {@code FORBIDDENWORD}, which suppress analyses the + * dictionary marks as virtual stems, compound-only parts, or forbidden words; and + * {@code CIRCUMFIX}, which binds marked prefix and suffix halves to one another. + * Conversion tables and the remaining compound machinery are not interpreted in this + * version; rules using them simply do not fire, so unsupported analyses are missed + * rather than invented.</p> * * <p>Instances are immutable and safe to share between threads.</p> * @@ -93,17 +104,47 @@ public final class HunspellDictionary { private final int compoundBegin; private final int compoundEnd; private final int compoundMin; + /** The place a part takes in a compound, deciding which positional flag admits it. */ + enum CompoundPosition { + /** The first part. */ + BEGIN, + /** Any part between the first and the last. */ + MIDDLE, + /** The last part. */ + END + } + + private final int needAffix; + private final int onlyInCompound; + private final int forbiddenWord; + private final int circumfix; + private final int compoundMiddle; + private final int compoundPermit; + private final int compoundForbid; + private final int compoundWordMax; + private final boolean checkCompoundDup; + private final boolean checkCompoundCase; + private final boolean checkCompoundTriple; - private HunspellDictionary(Map<String, List<int[]>> entries, List<Affix> prefixes, - List<Affix> suffixes, int compoundFlag, int compoundBegin, int compoundEnd, - int compoundMin) { - this.compoundFlag = compoundFlag; - this.compoundBegin = compoundBegin; - this.compoundEnd = compoundEnd; - this.compoundMin = compoundMin; + private HunspellDictionary(Map<String, List<int[]>> entries, AffixFile affix) { + this.compoundFlag = affix.compoundFlag; + this.compoundBegin = affix.compoundBegin; + this.compoundEnd = affix.compoundEnd; + this.compoundMin = affix.compoundMin; + this.needAffix = affix.needAffix; + this.onlyInCompound = affix.onlyInCompound; + this.forbiddenWord = affix.forbiddenWord; + this.circumfix = affix.circumfix; + this.compoundMiddle = affix.compoundMiddle; + this.compoundPermit = affix.compoundPermit; + this.compoundForbid = affix.compoundForbid; + this.compoundWordMax = affix.compoundWordMax; + this.checkCompoundDup = affix.checkCompoundDup; + this.checkCompoundCase = affix.checkCompoundCase; + this.checkCompoundTriple = affix.checkCompoundTriple; this.entries = entries; - this.prefixes = prefixes; - this.suffixes = suffixes; + this.prefixes = List.copyOf(affix.prefixes); + this.suffixes = List.copyOf(affix.suffixes); // Undoing a suffix requires the word to end with the rule's affix material, so // only rules whose material ends in the word's last character can ever apply; // the same holds for prefixes and the first character. Bucketing by that @@ -176,9 +217,7 @@ public final class HunspellDictionary { final Map<String, List<int[]>> entries = parseWordList( new String(readAll(dictionaryStream), charset), affix.flagMode, affix.flagAliases); - return new HunspellDictionary(entries, List.copyOf(affix.prefixes), - List.copyOf(affix.suffixes), affix.compoundFlag, affix.compoundBegin, - affix.compoundEnd, affix.compoundMin); + return new HunspellDictionary(entries, affix); } /** @@ -237,7 +276,8 @@ public final class HunspellDictionary { /** @return Whether the affix file declares any compounding flag at all. */ boolean compoundsDeclared() { - return compoundFlag != 0 || compoundBegin != 0 || compoundEnd != 0; + return compoundFlag != 0 || compoundBegin != 0 || compoundEnd != 0 + || compoundMiddle != 0; } /** @return The smallest length a compound part may have; at least {@code 1}. */ @@ -245,28 +285,136 @@ public final class HunspellDictionary { return compoundMin; } + /** @return The largest number of parts a compound may have; {@code 0} is unbounded. */ + int compoundWordMax() { + return compoundWordMax; + } + + /** @return Whether {@code CHECKCOMPOUNDDUP} forbids a part repeating its neighbor. */ + boolean checkCompoundDup() { + return checkCompoundDup; + } + + /** @return Whether {@code CHECKCOMPOUNDCASE} forbids uppercase at part boundaries. */ + boolean checkCompoundCase() { + return checkCompoundCase; + } + + /** @return Whether {@code CHECKCOMPOUNDTRIPLE} forbids triple letters at boundaries. */ + boolean checkCompoundTriple() { + return checkCompoundTriple; + } + /** - * Checks whether a listed word may open a compound: it carries the general - * compounding flag or the dedicated begin flag. + * The flag admitting a part at a compound position, next to the general + * compounding flag. + * + * @param position The part's place in the compound. + * @return The dedicated positional flag, or {@code 0} when undeclared. + */ + private int positionalFlag(CompoundPosition position) { + return switch (position) { + case BEGIN -> compoundBegin; + case MIDDLE -> compoundMiddle; + case END -> compoundEnd; + }; + } + + /** + * Checks whether a listed word may stand at a compound position: some homonym's + * flag set carries the general compounding flag or the position's dedicated flag + * and is not forbidden. A compound-only or virtual-stem homonym may take the + * position; that is what those flags permit. * * @param flagSets The word's flag sets from {@link #lookup(String)}. - * @return {@code true} if the word may stand first in a compound. + * @param position The part's place in the compound. + * @return {@code true} if the word may stand at the position. + */ + boolean mayStand(List<int[]> flagSets, CompoundPosition position) { + final int positional = positionalFlag(position); + for (final int[] flags : flagSets) { + if ((contains(flags, compoundFlag) || contains(flags, positional)) + && !contains(flags, forbiddenWord) && !contains(flags, needAffix)) { + return true; + } + } + return false; + } + + /** + * Checks whether some homonym supports an affixed compound part: its flag set + * carries the removed affix's flag, is not forbidden, and either the affix itself + * admits the position or the set carries the compounding or positional flag. + * + * @param flagSets The part stem's flag sets from {@link #lookup(String)}. + * @param affixFlag The removed affix's flag. + * @param position The part's place in the compound. + * @param affixAdmits Whether the affix's continuation classes admit the position, + * from {@link #affixAdmits(Affix, CompoundPosition)}. + * @return {@code true} if some homonym stands affixed at the position. + */ + boolean supportsPart(List<int[]> flagSets, int affixFlag, CompoundPosition position, + boolean affixAdmits) { + final int positional = positionalFlag(position); + for (final int[] flags : flagSets) { + if (contains(flags, affixFlag) && !contains(flags, forbiddenWord) + && (affixAdmits || contains(flags, compoundFlag) + || contains(flags, positional))) { + return true; + } + } + return false; + } + + /** + * Checks whether an affix admits its derived form at a compound position: its + * continuation classes carry the general compounding flag or the position's + * dedicated flag. Published dictionaries position their linking forms this way, + * through zero or dash suffixes whose continuation classes hold the positional + * flags. + * + * @param affix The affix rule applied to the part. + * @param position The part's place in the compound. + * @return {@code true} if the affixed form may stand at the position. + */ + boolean affixAdmits(Affix affix, CompoundPosition position) { + return (compoundFlag != 0 && affix.allowsContinuation(compoundFlag)) + || (positionalFlag(position) != 0 + && affix.allowsContinuation(positionalFlag(position))); + } + + /** + * Checks whether an affix may sit at a compound-internal boundary: it carries the + * {@code COMPOUNDPERMITFLAG} among its continuation classes. Without the flag a + * suffix fits only the last part and a prefix only the first. + * + * @param affix The affix rule applied to the part. + * @return {@code true} if the affix may face another part. */ - boolean mayBeginCompound(List<int[]> flagSets) { - return (compoundFlag != 0 && hasFlag(flagSets, compoundFlag)) - || (compoundBegin != 0 && hasFlag(flagSets, compoundBegin)); + boolean permitsInside(Affix affix) { + return compoundPermit != 0 && affix.allowsContinuation(compoundPermit); } /** - * Checks whether a listed word may close a compound: it carries the general - * compounding flag or the dedicated end flag. + * Checks whether an affix bars its derived form from compounds altogether: it + * carries the {@code COMPOUNDFORBIDFLAG} among its continuation classes. + * + * @param affix The affix rule applied to the part. + * @return {@code true} if the affixed form may not join a compound. + */ + boolean forbidsInCompound(Affix affix) { + return compoundForbid != 0 && affix.allowsContinuation(compoundForbid); + } + + /** + * Checks whether any of a word's flag sets is forbidden, which a dictionary uses + * to block one specific ill-formed compound while its parts stay productive. * * @param flagSets The word's flag sets from {@link #lookup(String)}. - * @return {@code true} if the word may stand last in a compound. + * @return {@code true} if some homonym carries the forbidden-word flag. */ - boolean mayEndCompound(List<int[]> flagSets) { - return (compoundFlag != 0 && hasFlag(flagSets, compoundFlag)) - || (compoundEnd != 0 && hasFlag(flagSets, compoundEnd)); + boolean anyForbidden(List<int[]> flagSets) { + return hasFlag(flagSets, forbiddenWord); } /** @@ -278,15 +426,130 @@ public final class HunspellDictionary { */ static boolean hasFlag(List<int[]> flagSets, int flag) { for (final int[] flags : flagSets) { - for (final int candidate : flags) { - if (candidate == flag) { - return true; - } + if (contains(flags, flag)) { + return true; } } return false; } + /** + * Checks one flag set for a flag. An undeclared flag, encoded as {@code 0}, is + * carried by no entry. + * + * @param flags One entry's flag set. + * @param flag The flag to look for. + * @return {@code true} if the set contains the flag. + */ + private static boolean contains(int[] flags, int flag) { + if (flag == 0) { + return false; + } + for (final int candidate : flags) { + if (candidate == flag) { + return true; + } + } + return false; + } + + /** + * Checks whether a listed word is valid on its own: some homonym's flag set carries + * none of the blocking flags. An entry whose every flag set is marked + * {@code NEEDAFFIX} is a virtual stem that exists only to be affixed, one marked + * {@code ONLYINCOMPOUND} appears only inside compounds, and one marked + * {@code FORBIDDENWORD} is listed to be blocked; none of them is a word by itself. + * + * @param flagSets The word's flag sets from {@link #lookup(String)}. + * @return {@code true} if some homonym stands on its own. + */ + boolean validStandalone(List<int[]> flagSets) { + for (final int[] flags : flagSets) { + if (!contains(flags, needAffix) && !contains(flags, onlyInCompound) + && !contains(flags, forbiddenWord)) { + return true; + } + } + return false; + } + + /** + * Checks whether some homonym supports an affix analysis: its flag set carries the + * affix's flag and is neither compound-only nor forbidden. A {@code NEEDAFFIX} set + * does support the analysis, because the removed affix is exactly what the virtual + * stem needs. + * + * @param flagSets The stem's flag sets from {@link #lookup(String)}. + * @param flag The removed affix's flag. + * @return {@code true} if some homonym carries the flag and may stand affixed. + */ + boolean supports(List<int[]> flagSets, int flag) { + for (final int[] flags : flagSets) { + if (contains(flags, flag) && !contains(flags, onlyInCompound) + && !contains(flags, forbiddenWord)) { + return true; + } + } + return false; + } + + /** + * Checks whether some homonym supports a cross-product analysis: one flag set + * carries both removed affixes' flags and is neither compound-only nor forbidden. + * The two flags must sit in the same set, because homonyms are separate words and + * each removal must be licensed by the same one. + * + * @param flagSets The stem's flag sets from {@link #lookup(String)}. + * @param prefixFlag The removed prefix's flag. + * @param suffixFlag The removed suffix's flag. + * @return {@code true} if some homonym carries both flags and may stand affixed. + */ + boolean supports(List<int[]> flagSets, int prefixFlag, int suffixFlag) { + for (final int[] flags : flagSets) { + if (contains(flags, prefixFlag) && contains(flags, suffixFlag) + && !contains(flags, onlyInCompound) && !contains(flags, forbiddenWord)) { + return true; + } + } + return false; + } + + /** + * Checks whether a form made with this affix alone is still a virtual stem: the + * affix carries the {@code NEEDAFFIX} flag among its continuation classes, so a + * further affix must join before the form is a word. + * + * @param affix The affix rule to inspect. + * @return {@code true} if the affix alone does not finish a word. + */ + boolean needsFurtherAffix(Affix affix) { + return needAffix != 0 && affix.allowsContinuation(needAffix); + } + + /** + * Checks whether an affix applies only inside compounds: it carries the + * {@code ONLYINCOMPOUND} flag among its continuation classes. + * + * @param affix The affix rule to inspect. + * @return {@code true} if the affix never applies to a standalone word. + */ + boolean compoundOnly(Affix affix) { + return onlyInCompound != 0 && affix.allowsContinuation(onlyInCompound); + } + + /** + * Checks whether an affix is one half of a circumfix: it carries the + * {@code CIRCUMFIX} flag among its continuation classes, so it is only valid on a + * word that also carries a circumfix-marked affix of the other kind, the German + * {@code ge...t} participle being the model. + * + * @param affix The affix rule to inspect. + * @return {@code true} if the affix never applies without its other half. + */ + boolean circumfixOnly(Affix affix) { + return circumfix != 0 && affix.allowsContinuation(circumfix); + } + /** * Reads a stream fully into memory. The stream is not closed. * @@ -354,14 +617,25 @@ public final class HunspellDictionary { private int compoundBegin; private int compoundEnd; private int compoundMin = 3; + private int needAffix; + private int onlyInCompound; + private int forbiddenWord; + private int circumfix; + private int compoundMiddle; + private int compoundPermit; + private int compoundForbid; + private int compoundWordMax; + private boolean checkCompoundDup; + private boolean checkCompoundCase; + private boolean checkCompoundTriple; } /** * Parses the affix file: the {@code FLAG} declaration, the {@code AF} flag alias - * table, and the {@code PFX} and {@code SFX} blocks. Directives outside the - * supported set (compounding, conversion tables, suggestion options, ...) are - * skipped, so their rules never fire and unsupported analyses are missed rather - * than invented. + * table, the compound and blocking flag declarations, and the {@code PFX} and + * {@code SFX} blocks. Directives outside the supported set (conversion tables, + * suggestion options, the remaining compound machinery, ...) are skipped, so their + * rules never fire and unsupported analyses are missed rather than invented. * * @param content The decoded affix file content. * @return The parsed rules and flag mode. Never {@code null}. @@ -393,29 +667,62 @@ public final class HunspellDictionary { break; case "COMPOUNDFLAG": case "COMPOUNDBEGIN": + case "COMPOUNDMIDDLE": case "COMPOUNDEND": + case "COMPOUNDPERMITFLAG": + case "COMPOUNDFORBIDFLAG": + case "NEEDAFFIX": + case "PSEUDOROOT": + case "ONLYINCOMPOUND": + case "FORBIDDENWORD": + case "CIRCUMFIX": if (fields.length < 2) { throw new IOException(fields[0] + " line without a flag at line " + (i + 1)); } - final int compound = parseFlag(fields[1], result.flagMode, i + 1); + final int declared = parseFlag(fields[1], result.flagMode, i + 1); switch (fields[0]) { - case "COMPOUNDFLAG" -> result.compoundFlag = compound; - case "COMPOUNDBEGIN" -> result.compoundBegin = compound; - default -> result.compoundEnd = compound; + case "COMPOUNDFLAG" -> result.compoundFlag = declared; + case "COMPOUNDBEGIN" -> result.compoundBegin = declared; + case "COMPOUNDMIDDLE" -> result.compoundMiddle = declared; + case "COMPOUNDEND" -> result.compoundEnd = declared; + case "COMPOUNDPERMITFLAG" -> result.compoundPermit = declared; + case "COMPOUNDFORBIDFLAG" -> result.compoundForbid = declared; + // PSEUDOROOT is the directive's name before hunspell renamed it + case "NEEDAFFIX", "PSEUDOROOT" -> result.needAffix = declared; + case "ONLYINCOMPOUND" -> result.onlyInCompound = declared; + case "CIRCUMFIX" -> result.circumfix = declared; + default -> result.forbiddenWord = declared; } i++; break; case "COMPOUNDMIN": + case "COMPOUNDWORDMAX": if (fields.length < 2) { - throw new IOException("COMPOUNDMIN line without a value at line " + (i + 1)); + throw new IOException(fields[0] + " line without a value at line " + (i + 1)); } try { - result.compoundMin = Math.max(1, Integer.parseInt(fields[1])); + if ("COMPOUNDMIN".equals(fields[0])) { + result.compoundMin = Math.max(1, Integer.parseInt(fields[1])); + } else { + result.compoundWordMax = Math.max(0, Integer.parseInt(fields[1])); + } } catch (NumberFormatException e) { - throw new IOException("malformed COMPOUNDMIN at line " + (i + 1), e); + throw new IOException("malformed " + fields[0] + " at line " + (i + 1), e); } i++; break; + case "CHECKCOMPOUNDDUP": + result.checkCompoundDup = true; + i++; + break; + case "CHECKCOMPOUNDCASE": + result.checkCompoundCase = true; + i++; + break; + case "CHECKCOMPOUNDTRIPLE": + result.checkCompoundTriple = true; + i++; + break; case "AF": // the first AF line declares the alias count; every further AF line is one // alias, a flag run whose 1-based position numeric dictionary flags refer to diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java index 6ac1a48aa..8703217ad 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java @@ -24,6 +24,7 @@ import java.util.Set; import opennlp.tools.stemmer.Stemmer; import opennlp.tools.stemmer.hunspell.HunspellDictionary.Affix; +import opennlp.tools.stemmer.hunspell.HunspellDictionary.CompoundPosition; import opennlp.tools.util.StringUtil; /** @@ -35,7 +36,10 @@ import opennlp.tools.util.StringUtil; * dictionary entry; {@link #stemAll(CharSequence)} returns every distinct analysis. A * word with no analysis is returned unchanged, so the stemmer degrades to identity on * unknown vocabulary. A form containing uppercase characters is also analyzed in its - * lowercase variant, so sentence-initial capitalization does not hide an entry.</p> + * lowercase variant, so sentence-initial capitalization does not hide an entry. + * Entries the dictionary marks as virtual stems ({@code NEEDAFFIX}), compound-only + * parts ({@code ONLYINCOMPOUND}), or forbidden words ({@code FORBIDDENWORD}) never + * count as standalone analyses, matching how hunspell reads those flags.</p> * * <p>The {@link Stemmer} interface leaves thread safety to the implementation. This * implementation reads only the immutable dictionary state, so a single instance is @@ -83,7 +87,7 @@ public class HunspellStemmer implements Stemmer { } if (analyses.isEmpty() && dictionary.compoundsDeclared()) { for (final String variant : variants(surface)) { - decompose(variant, analyses); + decompose(variant, surface, analyses); } } if (analyses.isEmpty()) { @@ -116,7 +120,8 @@ public class HunspellStemmer implements Stemmer { * @param analyses The mutable, insertion-ordered set collecting the stems found. */ private void analyze(String word, Set<String> analyses) { - if (dictionary.lookup(word) != null) { + final List<int[]> own = dictionary.lookup(word); + if (own != null && dictionary.validStandalone(own)) { analyses.add(word); } // Only rules whose affix material ends in the word's last character can be @@ -137,50 +142,314 @@ public class HunspellStemmer implements Stemmer { } /** - * Decomposes a word into two listed compound parts when the affix analysis found - * nothing: at every split point that leaves both sides at least the declared - * minimum length, the left side must be listed and allowed to open a compound and - * the right side listed and allowed to close one. The parts of the first splitting - * that succeeds are reported left to right, so the head-most material comes last, - * and further splittings add any parts not already reported. + * The most part-licensing attempts one decomposition search may spend. Compounding + * searches every split of every tail, which on adversarial input with a + * one-character minimum part length grows without useful bound; the budget stops + * the search there, missing analyses rather than stalling, in line with the + * engine's fail-closed posture. + */ + private static final int PART_CHECK_BUDGET = 2048; + + /** + * Decomposes a word into listed compound parts when the affix analysis found + * nothing: the first part must be admitted to open a compound, every further part + * to continue or close one, each at least the declared minimum length and counted + * against the declared maximum. A part stands on its own entry or on an entry plus + * one affix, the way published dictionaries position their linking forms through + * zero or dash suffixes. The stems of the parts of every successful splitting are + * reported left to right, so the head-most material comes last. A word the + * dictionary lists as forbidden never decomposes; that is how one specific + * ill-formed compound is blocked while its parts stay productive. * * @param word The case variant to decompose. - * @param analyses The mutable, insertion-ordered set collecting the parts. + * @param surface The surface form the variant was derived from; character case at + * junctions is judged against it, so lowercasing a variant cannot + * sidestep a {@code CHECKCOMPOUNDCASE} declaration. + * @param analyses The mutable, insertion-ordered set collecting the part stems. + */ + private void decompose(String word, String surface, Set<String> analyses) { + final List<int[]> own = dictionary.lookup(word); + if (own != null && dictionary.anyForbidden(own)) { + return; + } + if (word.length() < 2 * dictionary.compoundMin()) { + return; + } + // lowercasing may change the length in exceptional mappings, in which case the + // offsets no longer align and the variant itself is the only usable case source + final String caseSource = surface.length() == word.length() ? surface : word; + search(word, caseSource, 0, new ArrayList<>(), new ArrayList<>(), analyses, + new int[] {PART_CHECK_BUDGET}); + } + + /** + * Extends a partial decomposition with the part starting at {@code from}, trying + * every admissible length and recursing on the remainder. The boundary into this + * part honors the {@code CHECKCOMPOUNDCASE} and {@code CHECKCOMPOUNDTRIPLE} + * declarations, a part repeating its left neighbor honors + * {@code CHECKCOMPOUNDDUP}, and a completed decomposition flushes every part's + * stems into the analyses in part order. + * + * @param word The case variant under decomposition. + * @param caseSource The character-case source for junction checks, the surface + * form when its offsets align with the variant. + * @param from The index the next part starts at. + * @param surfaces The surface strings of the parts taken so far. + * @param stems The licensed stems of the parts taken so far, one list per part. + * @param analyses The mutable, insertion-ordered set collecting the part stems. + * @param budget The remaining part-licensing attempts, counted down in place. */ - private void decompose(String word, Set<String> analyses) { + private void search(String word, String caseSource, int from, List<String> surfaces, + List<List<String>> stems, Set<String> analyses, int[] budget) { + if (from > 0 && violatesBoundaryChecks(word, caseSource, from)) { + return; + } final int min = dictionary.compoundMin(); - for (int split = min; split <= word.length() - min; split++) { - final String left = word.substring(0, split); - final List<int[]> leftFlags = dictionary.lookup(left); - if (leftFlags == null || !dictionary.mayBeginCompound(leftFlags)) { - continue; + final int max = dictionary.compoundWordMax(); + final boolean first = from == 0; + // every split leaving room for a further part; a first-position part must also + // leave the closing part, so the whole word is never one part + if (max == 0 || surfaces.size() + 2 <= max) { + for (int end = from + min; end <= word.length() - min; end++) { + if (budget[0] <= 0) { + return; + } + budget[0]--; + final String part = word.substring(from, end); + if (duplicatesNeighbor(part, surfaces)) { + continue; + } + final List<String> partStems = partStems(part, + first ? CompoundPosition.BEGIN : CompoundPosition.MIDDLE, first, false); + if (partStems.isEmpty()) { + continue; + } + surfaces.add(part); + stems.add(partStems); + search(word, caseSource, end, surfaces, stems, analyses, budget); + surfaces.remove(surfaces.size() - 1); + stems.remove(stems.size() - 1); } - final String right = word.substring(split); - final List<int[]> rightFlags = dictionary.lookup(right); - if (rightFlags == null || !dictionary.mayEndCompound(rightFlags)) { - continue; + } + // the closing part takes the whole remainder; a compound has at least two parts + if (first || word.length() - from < min + || (max > 0 && surfaces.size() + 1 > max) || budget[0] <= 0) { + return; + } + budget[0]--; + final String part = word.substring(from); + if (duplicatesNeighbor(part, surfaces)) { + return; + } + final List<String> partStems = partStems(part, CompoundPosition.END, false, true); + if (partStems.isEmpty()) { + return; + } + for (final List<String> earlier : stems) { + analyses.addAll(earlier); + } + analyses.addAll(partStems); + } + + /** + * Applies the {@code CHECKCOMPOUNDDUP} declaration: a part must not repeat the + * part directly before it. + * + * @param part The candidate part. + * @param surfaces The surface strings of the parts taken so far. + * @return {@code true} if the declaration forbids this part here. + */ + private boolean duplicatesNeighbor(String part, List<String> surfaces) { + return dictionary.checkCompoundDup() && !surfaces.isEmpty() + && part.equals(surfaces.get(surfaces.size() - 1)); + } + + /** + * Applies the character-level boundary declarations at the junction before + * {@code from}: {@code CHECKCOMPOUNDCASE} forbids an uppercase character on either + * side of the junction, and {@code CHECKCOMPOUNDTRIPLE} forbids the same character + * three times in a row across it. + * + * @param word The case variant under decomposition. + * @param caseSource The character-case source for the uppercase judgment. + * @param from The index the junction sits before; greater than zero. + * @return {@code true} if a declaration forbids this junction. + */ + private boolean violatesBoundaryChecks(String word, String caseSource, int from) { + final char before = word.charAt(from - 1); + final char after = word.charAt(from); + if (dictionary.checkCompoundCase() + && (Character.isUpperCase(caseSource.charAt(from - 1)) + || Character.isUpperCase(caseSource.charAt(from)))) { + return true; + } + if (dictionary.checkCompoundTriple() && before == after + && ((from >= 2 && word.charAt(from - 2) == after) + || (from + 1 < word.length() && word.charAt(from + 1) == after))) { + return true; + } + return false; + } + + /** + * Collects the listed stems that admit one part at its compound position: the part + * as its own entry, or an entry plus one suffix or one prefix whose removal leaves + * a listed stem, zero-material rules included, because published dictionaries + * position their linking forms through zero and dash suffixes. An affix at a + * compound-internal boundary must carry the permit flag, a suffix facing the next + * part or a prefix facing the previous one. A part not found as written is also + * tried with its first letter uppercased, the way nouns listed capitalized appear + * lowercase inside a compound. + * + * @param part The part's surface text. + * @param position The part's place in the compound. + * @param first Whether the part opens the word. + * @param last Whether the part closes the word. + * @return The stems admitting the part, in discovery order. Never {@code null}. + */ + private List<String> partStems(String part, CompoundPosition position, + boolean first, boolean last) { + final Set<String> stems = new LinkedHashSet<>(); + collectPartStems(part, position, first, last, stems); + if (stems.isEmpty() && !part.isEmpty()) { + final int initial = part.codePointAt(0); + final int upper = Character.toUpperCase(initial); + if (upper != initial) { + collectPartStems(new StringBuilder().appendCodePoint(upper) + .append(part, Character.charCount(initial), part.length()).toString(), + position, first, last, stems); } - analyses.add(left); - analyses.add(right); + } + return List.copyOf(stems); + } + + /** + * Collects the stems admitting one spelling of a part, bare and through one affix. + * + * @param part The part spelling to look up. + * @param position The part's place in the compound. + * @param first Whether the part opens the word. + * @param last Whether the part closes the word. + * @param stems The mutable, insertion-ordered set collecting the stems. + */ + private void collectPartStems(String part, CompoundPosition position, + boolean first, boolean last, Set<String> stems) { + final List<int[]> own = dictionary.lookup(part); + if (own != null && dictionary.mayStand(own, position)) { + stems.add(part); + } + for (final Affix suffix : dictionary.suffixesEndingWith(part.charAt(part.length() - 1))) { + collectSuffixedPartStem(part, suffix, position, last, stems); + } + for (final Affix suffix : dictionary.suffixesWithoutMaterial()) { + collectSuffixedPartStem(part, suffix, position, last, stems); + } + for (final Affix prefix : dictionary.prefixesStartingWith(part.charAt(0))) { + collectPrefixedPartStem(part, prefix, position, first, stems); + } + for (final Affix prefix : dictionary.prefixesWithoutMaterial()) { + collectPrefixedPartStem(part, prefix, position, first, stems); + } + } + + /** + * Adds the stem of one suffixed part reading when the rule and the stem's entry + * admit it at the position. + * + * @param part The part spelling under analysis. + * @param suffix The suffix rule to undo. + * @param position The part's place in the compound. + * @param last Whether the part closes the word. + * @param stems The mutable, insertion-ordered set collecting the stems. + */ + private void collectSuffixedPartStem(String part, Affix suffix, + CompoundPosition position, boolean last, Set<String> stems) { + if (dictionary.circumfixOnly(suffix) || dictionary.forbidsInCompound(suffix) + || (!last && !dictionary.permitsInside(suffix))) { + return; + } + final String stem = removeAffixInCompound(part, suffix, true); + if (stem == null) { + return; + } + final List<int[]> flagSets = dictionary.lookup(stem); + if (flagSets != null && dictionary.supportsPart(flagSets, suffix.flag(), position, + dictionary.affixAdmits(suffix, position))) { + stems.add(stem); + } + } + + /** + * Adds the stem of one prefixed part reading when the rule and the stem's entry + * admit it at the position. + * + * @param part The part spelling under analysis. + * @param prefix The prefix rule to undo. + * @param position The part's place in the compound. + * @param first Whether the part opens the word. + * @param stems The mutable, insertion-ordered set collecting the stems. + */ + private void collectPrefixedPartStem(String part, Affix prefix, + CompoundPosition position, boolean first, Set<String> stems) { + if (dictionary.circumfixOnly(prefix) || dictionary.forbidsInCompound(prefix) + || (!first && !dictionary.permitsInside(prefix))) { + return; + } + final String stem = removeAffixInCompound(part, prefix, false); + if (stem == null) { + return; + } + final List<int[]> flagSets = dictionary.lookup(stem); + if (flagSets != null && dictionary.supportsPart(flagSets, prefix.flag(), position, + dictionary.affixAdmits(prefix, position))) { + stems.add(stem); } } + /** + * Undoes one affix rule on a compound part. Unlike the standalone removals, a rule + * that neither adds nor removes material is undone here, to its own spelling with + * the condition checked, because dictionaries position compound parts through + * exactly such zero rules. + * + * @param part The part spelling under analysis. + * @param affix The rule to undo. + * @param suffix Whether the rule is a suffix rule. + * @return The candidate stem, or {@code null} when the rule does not apply. + */ + private static String removeAffixInCompound(String part, Affix affix, boolean suffix) { + if (affix.affix().isEmpty() && affix.strip().isEmpty()) { + return affix.condition().matches(part) ? part : null; + } + return suffix ? removeSuffix(part, affix) : removePrefix(part, affix); + } + /** * Undoes one suffix rule and, through continuation classes, one further suffix on - * the intermediate stem, adding every dictionary-confirmed analysis. + * the intermediate stem, adding every dictionary-confirmed analysis. A rule that + * applies only inside compounds or only as half of a circumfix is not undone at all, + * the latter because no prefix accompanies it on this path; a rule marked as needing + * a further affix yields no single-removal analysis, because the surface form it + * makes alone is a virtual stem; its twofold analyses stand, the inner affix being + * exactly the further one required. * * @param word The case variant under analysis. * @param suffix The suffix rule to undo. * @param analyses The mutable, insertion-ordered set collecting the stems found. */ private void undoSuffix(String word, Affix suffix, Set<String> analyses) { + if (dictionary.compoundOnly(suffix) || dictionary.circumfixOnly(suffix)) { + return; + } final String stem = removeSuffix(word, suffix); if (stem == null) { return; } - final List<int[]> flagSets = dictionary.lookup(stem); - if (flagSets != null && HunspellDictionary.hasFlag(flagSets, suffix.flag())) { - analyses.add(stem); + if (!dictionary.needsFurtherAffix(suffix)) { + final List<int[]> flagSets = dictionary.lookup(stem); + if (flagSets != null && dictionary.supports(flagSets, suffix.flag())) { + analyses.add(stem); + } } for (final Affix inner : dictionary.suffixesEndingWith(stem.charAt(stem.length() - 1))) { undoInnerSuffix(stem, suffix, inner, analyses); @@ -201,7 +470,8 @@ public class HunspellStemmer implements Stemmer { */ private void undoInnerSuffix(String stem, Affix outer, Affix inner, Set<String> analyses) { - if (!inner.allowsContinuation(outer.flag())) { + if (!inner.allowsContinuation(outer.flag()) || dictionary.compoundOnly(inner) + || dictionary.circumfixOnly(inner)) { return; } final String doubleStem = removeSuffix(stem, inner); @@ -209,27 +479,36 @@ public class HunspellStemmer implements Stemmer { return; } final List<int[]> innerFlags = dictionary.lookup(doubleStem); - if (innerFlags != null && HunspellDictionary.hasFlag(innerFlags, inner.flag())) { + if (innerFlags != null && dictionary.supports(innerFlags, inner.flag())) { analyses.add(doubleStem); } } /** * Undoes one prefix rule and, for cross-product rules, one further suffix on the - * intermediate stem, adding every dictionary-confirmed analysis. + * intermediate stem, adding every dictionary-confirmed analysis. A rule that + * applies only inside compounds is not undone at all. A rule marked as needing a + * further affix or as half of a circumfix yields no single-removal analysis; its + * cross-product analyses stand, the suffix being exactly the further affix or the + * other circumfix half required. * * @param word The case variant under analysis. * @param prefix The prefix rule to undo. * @param analyses The mutable, insertion-ordered set collecting the stems found. */ private void undoPrefix(String word, Affix prefix, Set<String> analyses) { + if (dictionary.compoundOnly(prefix)) { + return; + } final String stem = removePrefix(word, prefix); if (stem == null) { return; } - final List<int[]> flagSets = dictionary.lookup(stem); - if (flagSets != null && HunspellDictionary.hasFlag(flagSets, prefix.flag())) { - analyses.add(stem); + if (!dictionary.needsFurtherAffix(prefix) && !dictionary.circumfixOnly(prefix)) { + final List<int[]> flagSets = dictionary.lookup(stem); + if (flagSets != null && dictionary.supports(flagSets, prefix.flag())) { + analyses.add(stem); + } } if (!prefix.crossProduct()) { return; @@ -243,7 +522,10 @@ public class HunspellStemmer implements Stemmer { } /** - * Undoes the suffix half of a cross-product removal when both rules opted in. + * Undoes the suffix half of a cross-product removal when both rules opted in. The + * two rules must agree on circumfixing: a circumfix-marked affix is only valid with + * a marked affix of the other kind, so a pair of which exactly one is marked mixes + * an ordinary affix into a circumfix and is rejected. * * @param stem The intermediate stem after the prefix removal. * @param prefix The already-undone prefix rule. @@ -252,16 +534,18 @@ public class HunspellStemmer implements Stemmer { */ private void undoCrossProductSuffix(String stem, Affix prefix, Affix suffix, Set<String> analyses) { - if (!suffix.crossProduct()) { + if (!suffix.crossProduct() || dictionary.compoundOnly(suffix) + || dictionary.circumfixOnly(prefix) != dictionary.circumfixOnly(suffix)) { return; } final String doubleStem = removeSuffix(stem, suffix); if (doubleStem == null) { return; } + // a needs-further-affix marker on either rule is satisfied by the other rule, + // so no such check applies here; both flags must sit in one homonym's flag set final List<int[]> both = dictionary.lookup(doubleStem); - if (both != null && HunspellDictionary.hasFlag(both, prefix.flag()) - && HunspellDictionary.hasFlag(both, suffix.flag())) { + if (both != null && dictionary.supports(both, prefix.flag(), suffix.flag())) { analyses.add(doubleStem); } } diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java new file mode 100644 index 000000000..d6d12faa4 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java @@ -0,0 +1,99 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer.hunspell; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.Test; + +/** + * Gated checks against published dictionaries, which are never bundled: the tests run + * only when {@code -Dopennlp.hunspell.dict.dir} names a directory holding + * {@code <name>.aff}/{@code <name>.dic} pairs, and each test additionally skips when + * its dictionary pair is absent. The download helper in {@code dev/} fetches the pairs + * together with their license files; see {@code dev/README-hunspell-dictionaries.md}. + * + * <p>The assertions are limited to morphology stable across dictionary revisions: + * everyday inflections, and for German the decomposability of ordinary compounds.</p> + */ +public class HunspellRealDictionaryTest { + + private static final String DICT_DIR_PROPERTY = "opennlp.hunspell.dict.dir"; + + /** + * Loads one dictionary pair from the gated directory, skipping the test when the + * gate or the pair is absent. + * + * @param name The dictionary base name, such as {@code en_US}. + * @return A stemmer over the loaded pair. Never {@code null}. + * @throws IOException Thrown if a present pair fails to load, which is a failure, + * not a skip. + */ + private static HunspellStemmer loadOrSkip(String name) throws IOException { + final String dir = System.getProperty(DICT_DIR_PROPERTY); + Assumptions.assumeTrue(dir != null && !dir.isBlank(), + "no " + DICT_DIR_PROPERTY + " given"); + final Path affix = Path.of(dir, name + ".aff"); + final Path words = Path.of(dir, name + ".dic"); + Assumptions.assumeTrue(Files.isReadable(affix) && Files.isReadable(words), + name + " pair not present under " + dir); + return new HunspellStemmer(HunspellDictionary.load(affix, words)); + } + + @Test + void testEnglishInflections() throws IOException { + final HunspellStemmer stemmer = loadOrSkip("en_US"); + Assertions.assertEquals("worker", stemmer.stem("workers").toString()); + Assertions.assertEquals("cat", stemmer.stem("cats").toString()); + Assertions.assertEquals("unhappy", stemmer.stem("unhappiest").toString()); + Assertions.assertEquals("quick", stemmer.stem("quickly").toString()); + Assertions.assertEquals("look", stemmer.stem("looked").toString()); + // unknown vocabulary degrades to identity + Assertions.assertEquals("zyzzyvax", stemmer.stem("zyzzyvax").toString()); + } + + @Test + void testGermanInflections() throws IOException { + final HunspellStemmer stemmer = loadOrSkip("de_DE_frami"); + Assertions.assertEquals("Kind", stemmer.stem("Kinder").toString()); + Assertions.assertEquals("Haus", stemmer.stem("Häuser").toString()); + Assertions.assertEquals("schnell", stemmer.stem("schnellsten").toString()); + } + + @Test + void testGermanCompoundsDecompose() throws IOException { + final HunspellStemmer stemmer = loadOrSkip("de_DE_frami"); + // the exact part spellings follow the dictionary's own entries and may shift + // between revisions; that ordinary compounds decompose at all must not + Assertions.assertTrue(stemmer.stemAll("Haustür").size() >= 2); + Assertions.assertTrue(stemmer.stemAll("Kinderzimmer").size() >= 2); + Assertions.assertTrue(stemmer.stemAll("Abbildungsverzeichnis").size() >= 2); + } + + @Test + void testHungarianInflections() throws IOException { + final HunspellStemmer stemmer = loadOrSkip("hu_HU"); + Assertions.assertEquals("kutya", stemmer.stem("kutyák").toString()); + Assertions.assertEquals("asztal", stemmer.stem("asztalon").toString()); + Assertions.assertEquals("könyv", stemmer.stem("könyveket").toString()); + } +} diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java index 9ffd1289b..d4fe1a196 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java @@ -775,4 +775,383 @@ public class HunspellStemmerTest { // the left side would be three characters, below the declared minimum of four Assertions.assertEquals(List.of("doghouse"), stemmer.stemAll("doghouse")); } + + /** + * Verifies the NEEDAFFIX flag on entries: a virtual stem exists only to be affixed, + * the linking forms of the published German dictionary being the model, so its bare + * form is no analysis of itself while its affixed forms still reduce to it. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testNeedAffixEntryIsNoStandaloneAnalysis() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "NEEDAFFIX h\nSFX S Y 1\nSFX S 0 s .\nSFX K Y 1\nSFX K 0 k .\n", + "2\nlink/hS\nlin/K\n")); + // the virtual entry no longer explains the bare form; only the k analysis remains + Assertions.assertEquals(List.of("lin"), stemmer.stemAll("link")); + // affixed, the virtual stem is exactly what the s removal lands on + Assertions.assertEquals(List.of("link"), stemmer.stemAll("links")); + } + + /** + * Verifies NEEDAFFIX against homonyms: the flag blocks one entry's flag set, not + * the word, so a second listing without the flag keeps the bare form valid. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testNeedAffixHomonymKeepsTheBareWord() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "NEEDAFFIX h\nSFX S Y 1\nSFX S 0 s .\n", + "2\nlink/hS\nlink\n")); + Assertions.assertEquals(List.of("link"), stemmer.stemAll("link")); + } + + /** + * Verifies the historical PSEUDOROOT alias, the directive's name before hunspell + * renamed it to NEEDAFFIX; older dictionaries still declare it. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testPseudoRootIsNeedAffixByItsOldName() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "PSEUDOROOT h\nSFX S Y 1\nSFX S 0 s .\nSFX K Y 1\nSFX K 0 k .\n", + "2\nlink/hS\nlin/K\n")); + Assertions.assertEquals(List.of("lin"), stemmer.stemAll("link")); + } + + /** + * Verifies the NEEDAFFIX flag on affix rules: a rule carrying the flag among its + * continuation classes makes a form that still needs another affix, so its + * single-removal analysis is suppressed while a twofold removal, whose inner affix + * is the further one required, still reports the stem. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testNeedAffixOnAnAffixRequiresAnotherAffix() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "NEEDAFFIX h", + "SFX A Y 1", + "SFX A 0 er/hB .", + "SFX B Y 1", + "SFX B 0 s .", + ""), + "1\nwork/A\n")); + // work + er alone is virtual, so worker has no analysis and passes through + Assertions.assertEquals(List.of("worker"), stemmer.stemAll("worker")); + // work + er + s is complete; the twofold removal reaches the listed stem + Assertions.assertEquals(List.of("work"), stemmer.stemAll("workers")); + } + + /** + * Verifies that a cross-product analysis satisfies an affix's NEEDAFFIX marker: + * the prefix is the further affix the marked suffix requires, mirroring how + * hunspell accepts a prefix plus a needs-affix suffix together. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testCrossProductSatisfiesNeedAffixOnTheSuffix() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "NEEDAFFIX h", + "PFX P Y 1", + "PFX P 0 un .", + "SFX A Y 1", + "SFX A 0 er/h .", + ""), + "1\nwork/AP\n")); + Assertions.assertEquals(List.of("worker"), stemmer.stemAll("worker")); + Assertions.assertEquals(List.of("work"), stemmer.stemAll("unworker")); + } + + /** + * Verifies the ONLYINCOMPOUND flag: an entry carrying it appears only inside + * compounds, the ordinal parts of the published US English dictionary being the + * model, so neither its bare form nor its affixed forms are standalone analyses, + * while compound decomposition may still use it. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testOnlyInCompoundEntrySupportsNoStandaloneAnalyses() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "ONLYINCOMPOUND c\nCOMPOUNDFLAG Z\nCOMPOUNDMIN 3\nSFX S Y 1\nSFX S 0 s .\n", + "3\npart/cSZ\nhouse/Z\nwalk/S\n")); + // the affix analysis is suppressed because part's only flag set is compound-only + Assertions.assertEquals(List.of("parts"), stemmer.stemAll("parts")); + Assertions.assertEquals(List.of("part"), stemmer.stemAll("part")); + // inside a compound the entry serves exactly its declared purpose + Assertions.assertEquals(List.of("part", "house"), stemmer.stemAll("parthouse")); + Assertions.assertEquals(List.of("walk"), stemmer.stemAll("walks")); + } + + /** + * Verifies the FORBIDDENWORD flag: an entry carrying it is listed to be blocked, + * so it supports no analysis and no compound part. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testForbiddenWordSupportsNothing() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "FORBIDDENWORD w\nCOMPOUNDFLAG Z\nCOMPOUNDMIN 3\nSFX S Y 1\nSFX S 0 s .\n", + "3\nfoo/wSZ\nhouse/Z\nbar/S\n")); + Assertions.assertEquals(List.of("foo"), stemmer.stemAll("foo")); + Assertions.assertEquals(List.of("foos"), stemmer.stemAll("foos")); + Assertions.assertEquals(List.of("foohouse"), stemmer.stemAll("foohouse")); + Assertions.assertEquals(List.of("bar"), stemmer.stemAll("bars")); + } + + /** The circumfix fixture: the German {@code ge...t} participle in miniature. */ + private static final String CIRCUMFIX_AFFIX = String.join("\n", + "CIRCUMFIX f", + "PFX G Y 1", + "PFX G 0 ge/f .", + "SFX T Y 1", + "SFX T en et/f en", + "PFX U Y 1", + "PFX U 0 un .", + "SFX S Y 1", + "SFX S 0 s .", + ""); + + /** + * Verifies the CIRCUMFIX flag: two marked halves analyze together and neither + * analyzes alone, so the participle reduces to its verb while the half-applied + * forms stay unexplained. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testCircumfixHalvesOnlyAnalyzeTogether() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + CIRCUMFIX_AFFIX, "1\narbeiten/GT\n")); + Assertions.assertEquals(List.of("arbeiten"), stemmer.stemAll("gearbeitet")); + // the suffix half alone is no word, although the stem carries its flag + Assertions.assertEquals(List.of("arbeitet"), stemmer.stemAll("arbeitet")); + // the prefix half alone is no word either + Assertions.assertEquals(List.of("gearbeiten"), stemmer.stemAll("gearbeiten")); + } + + /** + * Verifies that circumfixing rejects mixed pairs: a marked half never combines + * with an unmarked affix of the other kind, in either direction, while a fully + * unmarked cross-product in the same dictionary still analyzes. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testCircumfixRejectsMixedPairs() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + CIRCUMFIX_AFFIX, "2\narbeiten/GTUS\nlauf/US\n")); + // unmarked prefix with the marked suffix half + Assertions.assertEquals(List.of("unarbeitet"), stemmer.stemAll("unarbeitet")); + // the marked prefix half with an unmarked suffix + Assertions.assertEquals(List.of("gearbeitens"), stemmer.stemAll("gearbeitens")); + // both halves marked still analyze beside the rejected mixtures + Assertions.assertEquals(List.of("arbeiten"), stemmer.stemAll("gearbeitet")); + // a fully unmarked cross-product is untouched by the circumfix declaration + Assertions.assertEquals(List.of("lauf"), stemmer.stemAll("unlaufs")); + } + + /** + * Verifies decomposition beyond two parts: the positional flags admit a begin, a + * middle, and an end part, a part fit only for the middle neither opens nor closes, + * and repeated middles fold into the reported set. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testCompoundMiddleAdmitsInnerParts() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "COMPOUNDBEGIN B\nCOMPOUNDMIDDLE M\nCOMPOUNDEND E\nCOMPOUNDMIN 3\n", + "3\ndog/B\ncat/M\nhouse/E\n")); + Assertions.assertEquals(List.of("dog", "cat", "house"), + stemmer.stemAll("dogcathouse")); + Assertions.assertEquals(List.of("dog", "cat", "house"), + stemmer.stemAll("dogcatcathouse")); + Assertions.assertEquals(List.of("dog", "house"), stemmer.stemAll("doghouse")); + // cat holds only the middle flag, so it neither opens nor closes + Assertions.assertEquals(List.of("cathouse"), stemmer.stemAll("cathouse")); + Assertions.assertEquals(List.of("dogcat"), stemmer.stemAll("dogcat")); + } + + /** + * Verifies COMPOUNDWORDMAX: a decomposition needing more parts than declared is + * rejected while one within the bound still analyzes. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testCompoundWordMaxBoundsThePartCount() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "COMPOUNDBEGIN B\nCOMPOUNDMIDDLE M\nCOMPOUNDEND E\nCOMPOUNDMIN 3\n" + + "COMPOUNDWORDMAX 2\n", + "3\ndog/B\ncat/M\nhouse/E\n")); + Assertions.assertEquals(List.of("dog", "house"), stemmer.stemAll("doghouse")); + Assertions.assertEquals(List.of("dogcathouse"), stemmer.stemAll("dogcathouse")); + } + + /** + * Verifies affixed compound parts, the German linking form being the model: a part + * is its entry plus one suffix whose continuation classes position the derived form + * and permit it at the internal boundary, and the reported analysis is the entry, + * not the linking form. The lowercase interior spelling of a capitalized entry is + * found through the part's uppercased retry. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testLinkingSuffixJoinsCompoundParts() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "COMPOUNDBEGIN x", + "COMPOUNDEND z", + "COMPOUNDPERMITFLAG c", + "COMPOUNDMIN 2", + "SFX j Y 1", + "SFX j 0 s/xc .", + ""), + "2\nAbbildung/j\nVerzeichnis/z\n")); + Assertions.assertEquals(List.of("Abbildung", "Verzeichnis"), + stemmer.stemAll("Abbildungsverzeichnis")); + // without the linking s the first part has no admitting reading + Assertions.assertEquals(List.of("Abbildungverzeichnis"), + stemmer.stemAll("Abbildungverzeichnis")); + } + + /** + * Verifies zero-suffix part positioning, the pattern the published German + * dictionary uses: a virtual stem enters compounds through a rule that adds no + * material but whose continuation classes carry the positional and permit flags. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testZeroSuffixPositionsAVirtualStemInCompounds() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "NEEDAFFIX h", + "COMPOUNDBEGIN x", + "COMPOUNDEND z", + "COMPOUNDPERMITFLAG c", + "COMPOUNDMIN 3", + "SFX j Y 1", + "SFX j 0 0/xc .", + ""), + "2\nfugen/hj\nwerk/z\n")); + Assertions.assertEquals(List.of("fugen", "werk"), stemmer.stemAll("fugenwerk")); + // the virtual stem alone is still no word + Assertions.assertEquals(List.of("fugen"), stemmer.stemAll("fugen")); + } + + /** + * Verifies COMPOUNDFORBIDFLAG: an affixed form whose rule carries the flag stays + * out of compounds although its positioning otherwise admits it. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testCompoundForbidFlagBarsAnAffixedPart() throws IOException { + final String words = "2\ndog/ZS\nhouse/Z\n"; + final HunspellStemmer barred = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDPERMITFLAG c\nCOMPOUNDFORBIDFLAG F\nCOMPOUNDMIN 3\n" + + "SFX S Y 1\nSFX S 0 s/cF .\n", + words)); + Assertions.assertEquals(List.of("dogshouse"), barred.stemAll("dogshouse")); + final HunspellStemmer allowed = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDPERMITFLAG c\nCOMPOUNDMIN 3\n" + + "SFX S Y 1\nSFX S 0 s/c .\n", + words)); + Assertions.assertEquals(List.of("dog", "house"), allowed.stemAll("dogshouse")); + } + + /** + * Verifies that an affix without the permit flag keeps off internal boundaries: a + * suffixed reading fits the last part but not an earlier one. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testAffixWithoutPermitFlagStaysAtTheEdge() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nSFX S Y 1\nSFX S 0 s .\n", + "2\ndog/ZS\nhouse/ZS\n")); + // the suffix closes the word, so the last part may carry it + Assertions.assertEquals(List.of("dog", "house"), stemmer.stemAll("doghouses")); + // an internal suffix without the permit flag blocks the split + Assertions.assertEquals(List.of("dogshouse"), stemmer.stemAll("dogshouse")); + } + + /** + * Verifies CHECKCOMPOUNDDUP: a part must not repeat its left neighbor, while the + * same dictionary without the declaration accepts the repetition. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testCheckCompoundDupForbidsRepeatedParts() throws IOException { + final String words = "2\ndog/Z\nhouse/Z\n"; + final HunspellStemmer checked = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nCHECKCOMPOUNDDUP\n", words)); + Assertions.assertEquals(List.of("dogdoghouse"), checked.stemAll("dogdoghouse")); + final HunspellStemmer unchecked = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\n", words)); + Assertions.assertEquals(List.of("dog", "house"), unchecked.stemAll("dogdoghouse")); + } + + /** + * Verifies CHECKCOMPOUNDCASE: an uppercase character on either side of a junction + * forbids the split, while the same dictionary without the declaration accepts it. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testCheckCompoundCaseForbidsUppercaseJunctions() throws IOException { + final String words = "2\ndog/Z\nHouse/Z\n"; + final HunspellStemmer checked = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nCHECKCOMPOUNDCASE\n", words)); + Assertions.assertEquals(List.of("dogHouse"), checked.stemAll("dogHouse")); + final HunspellStemmer unchecked = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\n", words)); + Assertions.assertEquals(List.of("dog", "House"), unchecked.stemAll("dogHouse")); + } + + /** + * Verifies CHECKCOMPOUNDTRIPLE: the same character three times in a row across a + * junction forbids the split, while the same dictionary without the declaration + * accepts it. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testCheckCompoundTripleForbidsTripleLetters() throws IOException { + final String words = "2\nbell/Z\nlow/Z\n"; + final HunspellStemmer checked = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nCHECKCOMPOUNDTRIPLE\n", words)); + Assertions.assertEquals(List.of("belllow"), checked.stemAll("belllow")); + final HunspellStemmer unchecked = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\n", words)); + Assertions.assertEquals(List.of("bell", "low"), unchecked.stemAll("belllow")); + } + + /** + * Verifies that a listed forbidden word never decomposes: the dictionary blocks + * one specific ill-formed compound while its parts stay productive elsewhere. + * + * @throws IOException Thrown if a fixture fails to load. + */ + @Test + void testForbiddenEntryBlocksItsDecomposition() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "FORBIDDENWORD w\nCOMPOUNDFLAG Z\nCOMPOUNDMIN 3\n", + "4\ndog/Z\nhouse/Z\ncat/Z\ndoghouse/w\n")); + Assertions.assertEquals(List.of("doghouse"), stemmer.stemAll("doghouse")); + Assertions.assertEquals(List.of("cat", "house"), stemmer.stemAll("cathouse")); + } }
