Kristian Rickert created OPENNLP-1883:
-----------------------------------------
Summary: Thread-safe stemmers: StemmerFactory, thread-safe
SnowballStemmer, per-thread stem caching
Key: OPENNLP-1883
URL: https://issues.apache.org/jira/browse/OPENNLP-1883
Project: OpenNLP
Issue Type: Improvement
Components: Stemmer
Affects Versions: 3.0.0-M4
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Fix For: 3.0.0-M5
{{SnowballStemmer}} holds mutable per-call buffers in the generated engine, so
sharing an instance across threads silently corrupts results — no exception,
just wrong stems. With 3.0.0 unreleased, now is the time to align stemmers with
the thread-safe {{*ME}} components.
- {{SnowballStemmer}} routes each call to a per-thread generated engine via
{{OwnerOrPerThreadState}} (same pattern as the {{*ME}} classes); the generated
Snowball code is untouched. One instance is now safe to share; {{@ThreadSafe}}.
- New {{StemmerFactory}} (opennlp-api): an immutable, thread-safe capture of
stemmer configuration, with {{SnowballStemmerFactory}} and
{{PorterStemmerFactory}} implementations and a {{SharingStemmer}} adapter for
engines that are not thread-safe themselves (e.g. {{PorterStemmer}}).
- New {{CachingStemmer}}: bounded per-thread LRU memoization of word→stem
mappings. Natural text is Zipf-distributed, so most tokens are repeats; JMH
shows ~34x throughput on a Zipf workload and break-even on a cache-hostile
one.{{TermAnalyzer.Builder.stem(StemmerFactory)}} and
{{NormalizationProfile.matchingAnalyzer()}} use it, so profile analyzers are
shareable and cached by default.
- Tests: concurrency unit tests for all 21 algorithms (fixed pools,
owner-transition, virtual threads), {{MultiThreadedStemmerEval}} mirroring
{{MultiThreadedToolsEval}}, JMH benchmarks with results in BENCHMARKS.md.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
