[
https://issues.apache.org/jira/browse/OPENNLP-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096646#comment-18096646
]
Kristian Rickert commented on OPENNLP-1883:
-------------------------------------------
PR 1166 (light-stemmers, OPENNLP-1886) is built on top of PR 1163
(stemmer-factory, OPENNLP-1883) in git, same as the wordnet pair. Adding the
link once from 1883 automatically shows as is blocked by on 1886, so when you
get to 1886 it only needs its epic link.
> Thread-safe stemmers: StemmerFactory, thread-safe SnowballStemmer, per-thread
> stem caching
> ------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1883
> URL: https://issues.apache.org/jira/browse/OPENNLP-1883
> Project: OpenNLP
> Issue Type: Improvement
> Components: Stemmer
> Affects Versions: 3.0.0-M4
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M5
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> {{SnowballStemmer}} holds mutable per-call buffers in the generated engine,
> so sharing an instance across threads silently corrupts results — no
> exception, just wrong stems. With 3.0.0 unreleased, now is the time to align
> stemmers with the thread-safe {{*ME}} components.
> - {{SnowballStemmer}} routes each call to a per-thread generated engine via
> {{OwnerOrPerThreadState}} (same pattern as the {{*ME}} classes); the
> generated Snowball code is untouched. One instance is now safe to share;
> {{@ThreadSafe}}.
> - New {{StemmerFactory}} (opennlp-api): an immutable, thread-safe capture of
> stemmer configuration, with {{SnowballStemmerFactory}} and
> {{PorterStemmerFactory}} implementations and a {{SharingStemmer}} adapter for
> engines that are not thread-safe themselves (e.g. {{PorterStemmer}}).
> - New {{CachingStemmer}}: bounded per-thread LRU memoization of word→stem
> mappings. Natural text is Zipf-distributed, so most tokens are repeats; JMH
> shows ~34x throughput on a Zipf workload and break-even on a cache-hostile
> one.{{TermAnalyzer.Builder.stem(StemmerFactory)}} and
> {{NormalizationProfile.matchingAnalyzer()}} use it, so profile analyzers are
> shareable and cached by default.
> - Tests: concurrency unit tests for all 21 algorithms (fixed pools,
> owner-transition, virtual threads), {{MultiThreadedStemmerEval}} mirroring
> {{MultiThreadedToolsEval}}, JMH benchmarks with results in BENCHMARKS.md.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
