[
https://issues.apache.org/jira/browse/OPENNLP-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner updated OPENNLP-1883:
------------------------------------
Summary: Make stemmers thread-safe, add StemmerFactory and per-thread stem
caching (was: Thread-safe stemmers: StemmerFactory, thread-safe
SnowballStemmer, per-thread stem caching)
> Make stemmers thread-safe, add StemmerFactory and per-thread stem caching
> -------------------------------------------------------------------------
>
> Key: OPENNLP-1883
> URL: https://issues.apache.org/jira/browse/OPENNLP-1883
> Project: OpenNLP
> Issue Type: Improvement
> Components: Stemmer
> Affects Versions: 3.0.0-M4
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M5
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> {{SnowballStemmer}} holds mutable per-call buffers in the generated engine,
> so sharing an instance across threads silently corrupts results — no
> exception, just wrong stems. With 3.0.0 unreleased, now is the time to align
> stemmers with the thread-safe {{*ME}} components.
> - {{SnowballStemmer}} routes each call to a per-thread generated engine via
> {{OwnerOrPerThreadState}} (same pattern as the {{*ME}} classes); the
> generated Snowball code is untouched. One instance is now safe to share;
> {{@ThreadSafe}}.
> - New {{StemmerFactory}} (opennlp-api): an immutable, thread-safe capture of
> stemmer configuration, with {{SnowballStemmerFactory}} and
> {{PorterStemmerFactory}} implementations and a {{SharingStemmer}} adapter for
> engines that are not thread-safe themselves (e.g. {{PorterStemmer}}).
> - New {{CachingStemmer}}: bounded per-thread LRU memoization of word→stem
> mappings. Natural text is Zipf-distributed, so most tokens are repeats; JMH
> shows ~34x throughput on a Zipf workload and break-even on a cache-hostile
> one.{{TermAnalyzer.Builder.stem(StemmerFactory)}} and
> {{NormalizationProfile.matchingAnalyzer()}} use it, so profile analyzers are
> shareable and cached by default.
> - Tests: concurrency unit tests for all 21 algorithms (fixed pools,
> owner-transition, virtual threads), {{MultiThreadedStemmerEval}} mirroring
> {{MultiThreadedToolsEval}}, JMH benchmarks with results in BENCHMARKS.md.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
