Kristian Rickert created OPENNLP-1886:
-----------------------------------------
Summary: UniNE light and minimal stemmer tiers with vocabulary
parity fixtures
Key: OPENNLP-1886
URL: https://issues.apache.org/jira/browse/OPENNLP-1886
Project: OpenNLP
Issue Type: New Feature
Components: Sentence Detector
Reporter: Kristian Rickert
Assignee: Kristian Rickert
STACKED on [#1163|https://github.com/apache/opennlp/pull/1163]
(stemmer-factory): this branch builds on that PR's StemmerFactory contract, and
since that head lives on a fork it cannot be this PR's base ref. Until
[#1163|https://github.com/apache/opennlp/pull/1163] merges, the diff here shows
its commits too; only the last commit
([{{592380e}}|https://github.com/apache/opennlp/commit/592380e351ce0aa9d3f558ed10fbfecc95d85d99])
is this PR. After [#1163|https://github.com/apache/opennlp/pull/1163] lands,
the branch rebases onto main and the diff collapses to this change alone.
Adds sixteen stemmers in a new opennlp.tools.stemmer.light package: light and
minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk
varieties), and Swedish, plus light stemmers for Finnish, Hungarian, Italian,
Portuguese, and Russian, and the English minimal stemmer. These fill the tier
between no stemming and the aggressive Snowball algorithms: they remove plural
and inflectional endings with far less conflation. Adapted from Apache Lucene's
analysis-common module; the UniNE algorithms carry Jacques Savoy's BSD notice
in each source file and in the distribution LICENSE.
Every stemmer is stateless and thread-safe, implements Stemmer, and is its own
StemmerFactory. Input is expected lowercase, matching the algorithms' original
contract; null input fails loudly.
Parity is asserted against the original implementations: bundled fixtures
sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per
stemmer, complete lists for Norwegian) regenerated by running the originals, so
any behavioral drift fails the test. The Galician and Portuguese minimal
stemmers are excluded because they build on the RSLP rule engine, which is its
own effort.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
