This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch OPENNLP-1893-hunspell in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 0f958cc6a0a422f08202f9bf3c30dd5149f1e5ae Author: Kristian Rickert <[email protected]> AuthorDate: Mon Jul 20 04:38:12 2026 -0400 OPENNLP-1893: Add hunspell manual coverage with mirror-tested examples Extend docbkx/stemmer.xml with the hunspell affix-stemmer section, wire the chapter into the manual, and add StemmerFactoryUsageExampleTest and HunspellManualExampleTest asserting the load-and-stem values the chapter prints. Point the dictionary README at the new manual example. --- .../dev/README-hunspell-dictionaries.md | 2 +- .../hunspell/HunspellManualExampleTest.java | 73 ++++++++++++++++++++++ opennlp-docs/src/docbkx/stemmer.xml | 26 ++++++++ 3 files changed, 100 insertions(+), 1 deletion(-) diff --git a/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md b/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md index 457fd01ee..440814163 100644 --- a/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md +++ b/opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md @@ -46,7 +46,7 @@ Stemmer stemmer = factory.newStemmer(); CharSequence stem = stemmer.stem("workers"); ``` -What `stem` evaluates to is decided by the dictionary you loaded, and this project ships no dictionary data, so no result is claimed here for `en_US`. What is verified is the flow above: `HunspellStemmerFactoryTest#testEndToEndUsageFromFiles` runs exactly these calls against a project-authored `.aff`/`.dic` pair written to disk, in which `work` carries the agentive `-er` rule and its continuation class for the plural `-s`, and asserts that `stemmer.stem("workers")` returns `work`. +What `stem` evaluates to is decided by the dictionary you loaded, and this project ships no dictionary data, so no result is claimed here for `en_US`. The same load-and-stem flow is pinned by `HunspellManualExampleTest` (miniature in-memory dictionary, asserted stems for `workers` and `worker`) and by `HunspellStemmerFactoryTest#testEndToEndUsageFromFiles` (the same pair written to disk). The developer manual chapter `stemmer.xml` cites `HunspellManualExampleTest`. The dictionary is immutable and safe to share between threads; the factory hands out a fresh stemmer per call, so each thread takes its own from `newStemmer()`. A dictionary that declares a non-UTF-8 encoding through the `SET` directive in its `.aff` file is decoded accordingly; nothing needs converting beforehand. diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java new file mode 100644 index 000000000..21daec7e0 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer.hunspell; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.List; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import opennlp.tools.stemmer.Stemmer; + +/** + * Runs the manual's Hunspell examples (docbkx {@code stemmer.xml}) verbatim: every + * value the chapter states is asserted here, so a change breaking this test breaks the + * manual. The fixture dictionary is authored inside this class; no external dictionary + * data is involved. + */ +public class HunspellManualExampleTest { + + /** + * Affix fixture matching the chapter: agentive {@code -er} with continuation class + * {@code S}, and the plural {@code -s}. + */ + private static final String AFFIX = String.join("\n", + "SET UTF-8", + "SFX E Y 1", + "SFX E 0 er/S .", + "SFX S Y 1", + "SFX S 0 s [^sxy]", + ""); + + /** Word-list fixture: {@code work} accepts both suffixes. */ + private static final String WORDS = "1\nwork/ES\n"; + + /** + * Loads the chapter's miniature dictionary, stems through a factory-minted stemmer, + * and asserts the exact stems the manual prints. + * + * @throws IOException Thrown if the in-memory fixture fails to load. + */ + @Test + void testLoadAndStemWorkers() throws IOException { + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream(AFFIX.getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream(WORDS.getBytes(StandardCharsets.UTF_8))); + final Stemmer stemmer = new HunspellStemmerFactory(dictionary).newStemmer(); + + Assertions.assertEquals("work", stemmer.stem("workers").toString()); + Assertions.assertEquals("work", stemmer.stem("worker").toString()); + Assertions.assertEquals(List.of("work"), + stemmer.stemAll("workers").stream().map(CharSequence::toString).toList()); + // unknown vocabulary passes through unchanged + Assertions.assertEquals("table", stemmer.stem("table").toString()); + } +} diff --git a/opennlp-docs/src/docbkx/stemmer.xml b/opennlp-docs/src/docbkx/stemmer.xml index 248b310c1..8dc078d97 100644 --- a/opennlp-docs/src/docbkx/stemmer.xml +++ b/opennlp-docs/src/docbkx/stemmer.xml @@ -69,4 +69,30 @@ new CachingStemmer(factory).stem("running"); // "run"]]> longer uses a sharing or caching stemmer. </para> </section> + + <section xml:id="tools.stemmer.hunspell"> + <title>Hunspell dictionaries</title> + <para> + <code>opennlp.tools.stemmer.hunspell</code> is a clean-room engine over the + documented Hunspell dictionary format: a user-supplied + <code>.aff</code> affix file and its <code>.dic</code> word list. OpenNLP + bundles no dictionary data, so each dictionary's own license stays with + the files you download. The dictionary is immutable and safe to share; + <code>HunspellStemmerFactory</code> hands out a fresh stemmer per call. + <code>HunspellManualExampleTest</code> asserts the behavior shown here. + <programlisting language="java"><![CDATA[ +HunspellDictionary dictionary = HunspellDictionary.load( + Path.of("en_US.aff"), Path.of("en_US.dic")); +Stemmer stemmer = new HunspellStemmerFactory(dictionary).newStemmer(); + +stemmer.stem("workers"); // "work" +stemmer.stem("table"); // "table" (unknown vocabulary is unchanged)]]> + </programlisting> + The in-tree test uses a project-authored miniature dictionary instead of a + published one, and asserts the same stems for <code>workers</code> and + <code>worker</code>. Acquisition helpers and the supported affix feature + set live in + <code>opennlp-core/opennlp-runtime/dev/README-hunspell-dictionaries.md</code>. + </para> + </section> </chapter>
