[ 
https://issues.apache.org/jira/browse/SOLR-18656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amrit Sarkar updated SOLR-18656:
--------------------------------
    Attachment: SOLR-18656-test.patch

> SolrCmdDistributor.finish() drops recorded errors when a stream stalls
> ----------------------------------------------------------------------
>
>                 Key: SOLR-18656
>                 URL: https://issues.apache.org/jira/browse/SOLR-18656
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud
>            Reporter: Amrit Sarkar
>            Priority: Major
>         Attachments: SOLR-18656-test.patch
>
>
> {{SolrCmdDistributor.finish()}} catches and logs the {{IOException}} from 
> {{blockAndDoRetries()}}:
> {code:java}
> try {
>   blockAndDoRetries();
> } catch (IOException e) {
>   log.warn("Unable to finish sending updates", e);
> }
> {code}
> {{StreamingSolrClients.blockUntilFinished()}} throws that as soon as one 
> client hits {{StallDetection.stallCheck()}}, i.e. no progress with a 
> non-empty queue for {{solr.cloud.client.stallTime}}. The stall time is raised 
> to 2x {{solr.cloud.client.pollQueueTime}}, so it's 20s server side by 
> default. Throwing there skips:
> * waiting on the remaining clients
> * {{doRetriesIfNeeded()}}, so retriable errors are never retried
> * moving errors already collected from the other clients into {{getErrors()}}
> {{getErrors()}} then comes back empty and 
> {{DistributedZkUpdateProcessor.doDistribFinish}} treats the batch as 
> successful. The client gets a success response, and the replicas that failed 
> or stalled aren't put into a lower term, so they don't recover.
> The attached test fails on main. One node accepts the connection and never 
> reads, another refuses it. After {{finish()}}, {{getErrors()}} is {{[]}} even 
> though the refused node's error was recorded about a second before the stall.
> Proposed fix: catch the {{IOException}} per client in 
> {{StreamingSolrClients.blockUntilFinished()}} and report it through that 
> client's {{handleError}}. The stall becomes an error for the node that 
> stalled, and the other clients are still waited on and retried.
> Related to [SOLR-5939|https://issues.apache.org/jira/browse/SOLR-5939], which 
> touches the same error-reporting path.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to