Kristian Rickert created OPENNLP-1884:
-----------------------------------------
Summary: Extend stemmer coverage: regenerate from Snowball 3.1.1
and add light stemmers
Key: OPENNLP-1884
URL: https://issues.apache.org/jira/browse/OPENNLP-1884
Project: OpenNLP
Issue Type: Improvement
Reporter: Kristian Rickert
Assignee: Kristian Rickert
OpenNLP currently ships 21 Snowball algorithms (20 languages, last regenerated
for OPENNLP-1701). Upstream Snowball 3.1.1 (June 2026) provides ~35 algorithms
across ~33 languages. Regenerating adds Armenian, Basque, Czech, Esperanto,
Estonian, Hindi, Latin, Lithuanian, Nepali, Persian, Polish, Serbian, Sesotho,
Tamil and Yiddish, plus the Early Modern English variant - generated code only,
no algorithm work. Stemmer tests would now run in CI (OPENNLP-1882), so
regeneration regressions surface immediately.
Additionally, port the hand-written light/minimal stemmers from Apache Lucene
(Arabic light10, Hindi light, Portuguese RSLP, Latvian, Bulgarian, and others)
- ASL 2.0 within the ASF, with attribution in NOTICE. Light stemmers under-stem
relative to Snowball, which is often preferable for high-precision matching.
Expose everything through the {{StemmerFactory}} seam and register new
languages in {{{}NormalizationProfiles{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
