[
https://issues.apache.org/jira/browse/SOLR-18656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amrit Sarkar updated SOLR-18656:
--------------------------------
Attachment: SOLR-18656-test.patch
> SolrCmdDistributor.finish() drops recorded errors when a stream stalls
> ----------------------------------------------------------------------
>
> Key: SOLR-18656
> URL: https://issues.apache.org/jira/browse/SOLR-18656
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Amrit Sarkar
> Priority: Major
> Attachments: SOLR-18656-test.patch
>
>
> {{SolrCmdDistributor.finish()}} catches and logs the {{IOException}} from
> {{blockAndDoRetries()}}:
> {code:java}
> try {
> blockAndDoRetries();
> } catch (IOException e) {
> log.warn("Unable to finish sending updates", e);
> }
> {code}
> {{StreamingSolrClients.blockUntilFinished()}} throws that as soon as one
> client hits {{StallDetection.stallCheck()}}, i.e. no progress with a
> non-empty queue for {{solr.cloud.client.stallTime}}. The stall time is raised
> to 2x {{solr.cloud.client.pollQueueTime}}, so it's 20s server side by
> default. Throwing there skips:
> * waiting on the remaining clients
> * {{doRetriesIfNeeded()}}, so retriable errors are never retried
> * moving errors already collected from the other clients into {{getErrors()}}
> {{getErrors()}} then comes back empty and
> {{DistributedZkUpdateProcessor.doDistribFinish}} treats the batch as
> successful. The client gets a success response, and the replicas that failed
> or stalled aren't put into a lower term, so they don't recover.
> The attached test fails on main. One node accepts the connection and never
> reads, another refuses it. After {{finish()}}, {{getErrors()}} is {{[]}} even
> though the refused node's error was recorded about a second before the stall.
> Proposed fix: catch the {{IOException}} per client in
> {{StreamingSolrClients.blockUntilFinished()}} and report it through that
> client's {{handleError}}. The stall becomes an error for the node that
> stalled, and the other clients are still waited on and retried.
> Related to [SOLR-5939|https://issues.apache.org/jira/browse/SOLR-5939], which
> touches the same error-reporting path.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]