serhiy-bzhezytskyy opened a new pull request, #4812: URL: https://github.com/apache/solr/pull/4812
https://issues.apache.org/jira/browse/SOLR-18362 Removes the deprecated `org.apache.solr.spelling.Token` -- a class whose own TODO called it out: "Refactor the spellchecker API to use TokenStreams properly, rather than this hack" (it worked by extending Lucene's internal `PackedTokenAttributeImpl`, never meant as a general-purpose value holder). - `SpellingQueryConverter`/`SuggestQueryConverter`/`SimpleQueryConverter` now build a real `TokenStream` (new package-private `QueryWordsTokenStream`, analyzing each parsed query word lazily) instead of materializing a `Collection<Token>` up front - Every `SolrSpellChecker` implementation (`DirectSolrSpellChecker`, `AbstractLuceneSpellChecker`, `Suggester`, `WordBreakSolrSpellChecker`) consumes via `incrementToken()` and Lucene's own attribute interfaces (`CharTermAttribute`, `OffsetAttribute`, `FlagsAttribute`, etc.), not a pre-built collection - One plain value type remains, `SpellCheckToken` (not a Lucene `AttributeImpl` subclass) -- it's structurally forced by `SolrSpellChecker#mergeSuggestions`, which correlates suggestions from remote-shard responses by `(text, offset)` pairs deserialized off the wire, where there's no `TokenStream` to read from at all - `SpellingOptions.tokens` (a `Collection<Token>`) is now `tokenStreamSupplier` (a `Supplier<TokenStream>`): a `TokenStream` is single-use (reset/incrementToken-loop/end/close), so multiple consumers of the same query's terms (e.g. `ConjunctionSolrSpellChecker`'s sub-checkers, or `SpellCheckComponent`'s own emptiness-check + real use + originalTerms echo) each get a fresh instance rather than sharing/resetting one - Breaking change for anyone with a custom `QueryConverter` or `SolrSpellChecker` registered via `solrconfig.xml`: `convert()`'s return type, `SpellingOptions`'s field, and `SpellingResult`'s map key all changed - Trade-off: the query is now re-analyzed up to 3x per request (peek-for-non-empty, the real suggestion pass, and -- for shard requests -- the originalTerms echo) instead of once. Cheap for short queries; flagging it as a deliberate, known cost rather than leaving it implicit 118 tests, 0 failures (full `spelling`/`suggest` package coverage plus `SpellCheckComponentTest` and the suggest-component tests). AI-assisted (Claude Sonnet 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
