[
https://issues.apache.org/jira/browse/SOLR-18348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105010#comment-18105010
]
Eric Pugh commented on SOLR-18348:
----------------------------------
Did some digging around, and started playwiht adding a
`**`buildOnCommitAsync`**` config. There's one important interaction worth
flagging, though, since it bears directly on the bug: **`waitSearcher=false` is
already an existing escape hatch that would have prevented the *old*
synchronous `buildOnCommit` from blocking the commit response** — because
`DirectUpdateHandler2.commit()` only calls that blocking
`waitSearcher[0].get()` when `waitSearcher=true`. If the customer had been
sending `waitSearcher=false` on their commits, the suggester rebuild would
still run synchronously on the single-threaded `searcherExecutor`, but the
commit request itself wouldn't wait around for it.
The difference from my in progress fix: `waitSearcher=false` is a blunt, global
instrument — it decouples *everything* tied to the new-searcher event (warming,
spellcheck rebuilds, custom listeners, and the searcher becoming visible for
search at all) from the commit response, not just the suggester.
`buildOnCommitAsync` is scoped specifically to the suggester rebuild, leaving
warming/registration/search-visibility semantics for everything else untouched.
However`waitSearcher=false` is a possible workaround, distinct from the actual
fix.
> `buildOnCommit=true` suggester rebuild blocks commit (and the client request)
> indefinitely with no timeout
> ----------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18348
> URL: https://issues.apache.org/jira/browse/SOLR-18348
> Project: Solr
> Issue Type: Bug
> Components: Suggester
> Affects Versions: 9.10.1
> Reporter: Eric Pugh
> Assignee: Eric Pugh
> Priority: Major
>
> h3. Problem
> When a suggester is configured with `buildOnCommit=true`, a slow suggester
> rebuild does not just make that commit take longer in the background — it
> synchronously blocks the commit call itself, and therefore the client/update
> request waiting on it, with *no timeout*. Under sustained heavy indexing
> combined with throttled storage (e.g. an Azure disk after burst credits are
> exhausted), this manifests as outright request timeouts requiring manual
> operator intervention, rather than graceful degradation.
> A customer hit this in production: ~4 million product documents being indexed
> against a suggester with an I/O-intensive `AnalyzingInfixSuggester`
> configuration, on a disk that had exhausted its burst-credit allowance.
> Indexing requests began timing out. Thread dumps showed threads stuck in:
> {code}
> "searcherExecutor-250-thread-3-processing-...-core_node4" ... RUNNABLE
> at org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:138)
> at
> org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictDecompressor.decompress(...)
> at
> org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader...
> at
> org.apache.lucene.search.suggest.DocumentDictionary$DocumentInputIterator.next(DocumentDictionary.java:169)
> at
> org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.build(AnalyzingInfixSuggester.java:346)
> {code}
> h3. Root cause
> * \{{SuggestComponent.SuggesterListener.newSearcher()}} calls
> \{{suggester.build(core, newSearcher)}} synchronously — see
> [SuggestComponent.java|https://github.com/apache/solr/blob/releases/solr/9.10.1/solr/core/src/java/org/apache/solr/handler/component/SuggestComponent.java#L566-L571].
> * This listener is invoked as a \{{newSearcherListener}} on \{{SolrCore}}'s
> \{{searcherExecutor}}, which is explicitly a *single-threaded* executor (see
> the comment "WARNING: this code assumes a single threaded executor" in
> \{{SolrCore.java}}). Warmup, all \{{newSearcherListener}}s, and searcher
> registration all run sequentially on this one thread.
> * \{{DirectUpdateHandler2.commit()}} blocks the committing thread on
> \{{waitSearcher[0].get()}} — a no-arg, *no-timeout* \{{Future.get()}} —
> whenever \{{waitSearcher=true}}, which is the default for a normal
> \{{commit=true}} request (\{{CommitUpdateCommand.waitSearcher}} defaults to
> \{{true}}).
> * Net effect: an arbitrarily slow \{{buildOnCommit}} rebuild (or any other
> slow \{{newSearcherListener}}) blocks the commit-issuing thread/HTTP request
> indefinitely, bounded only by client/proxy socket timeouts.
> This exact hazard is already flagged in the reference guide ("Some lookup
> implementations may take a long time to build... using buildOnCommit or
> buildOnOptimize, particularly with a high frequency of soft commits is not
> recommended") but the current behavior is worse than "slow" — it can hang a
> commit indefinitely with no internal safeguard.
> Note: \{{SpellCheckComponent}} has an analogous
> \{{buildOnCommit}}/\{{SpellCheckerListener}} pattern and likely shares this
> same architectural risk; filing this ticket scoped to \{{SuggestComponent}}
> only.
> h3. Reproduction
> Two JUnit tests demonstrate this deterministically using a test-only
> dictionary that simulates slow I/O (sleeping a fixed time per term, so total
> build time is bounded and reproducible):
> * \{{SuggestComponentBuildOnCommitBlocksCommitTest}} — with
> \{{buildOnCommit=true}}, \{{commit()}} takes as long as the (artificially
> slow) suggester build.
> * \{{SuggestComponentBuildOnCommitDisabledCommitStaysFastTest}} — same slow
> dictionary, \{{buildOnCommit=false}}: \{{commit()}} stays fast, isolating the
> cause to \{{buildOnCommit}} itself.
> h3. Suggested direction (open to discussion)
> * Run \{{buildOnCommit}}/\{{buildOnOptimize}} suggester rebuilds off the
> shared single-threaded \{{searcherExecutor}} (e.g. a dedicated executor) so
> they don't block warmup/registration/commit for other listeners.
> * And/or decouple the rebuild from the commit's \{{waitSearcher}} path
> entirely, so a slow rebuild degrades to "suggestions are stale a bit longer"
> rather than "commit hangs."
> * And/or add an explicit, documented timeout/async option for
> \{{buildOnCommit}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
