krickert opened a new pull request, #1166: URL: https://github.com/apache/opennlp/pull/1166
STACKED on #1163 (stemmer-factory): this branch builds on that PR's StemmerFactory contract, and since that head lives on a fork it cannot be this PR's base ref. Until #1163 merges, the diff here shows its commits too; only the last commit (592380e3) is this PR. After #1163 lands, the branch rebases onto main and the diff collapses to this change alone. Adds sixteen stemmers in a new opennlp.tools.stemmer.light package: light and minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk varieties), and Swedish, plus light stemmers for Finnish, Hungarian, Italian, Portuguese, and Russian, and the English minimal stemmer. These fill the tier between no stemming and the aggressive Snowball algorithms: they remove plural and inflectional endings with far less conflation. Adapted from Apache Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's BSD notice in each source file and in the distribution LICENSE. Every stemmer is stateless and thread-safe, implements Stemmer, and is its own StemmerFactory. Input is expected lowercase, matching the algorithms' original contract; null input fails loudly. Parity is asserted against the original implementations: bundled fixtures sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per stemmer, complete lists for Norwegian) regenerated by running the originals, so any behavioral drift fails the test. The Galician and Portuguese minimal stemmers are excluded because they build on the RSLP rule engine, which is its own effort. JIRA to follow; title gets the key once filed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
