Amrit Sarkar created SOLR-18656:
-----------------------------------

             Summary: SolrCmdDistributor.finish() drops recorded errors when a 
stream stalls
                 Key: SOLR-18656
                 URL: https://issues.apache.org/jira/browse/SOLR-18656
             Project: Solr
          Issue Type: Bug
          Components: SolrCloud
            Reporter: Amrit Sarkar


{{SolrCmdDistributor.finish()}} catches and logs the {{IOException}} from 
{{blockAndDoRetries()}}:

{code:java}
try {
  blockAndDoRetries();
} catch (IOException e) {
  log.warn("Unable to finish sending updates", e);
}
{code}

{{StreamingSolrClients.blockUntilFinished()}} throws that as soon as one client 
hits {{StallDetection.stallCheck()}}, i.e. no progress with a non-empty queue 
for {{solr.cloud.client.stallTime}}. The stall time is raised to 2x 
{{solr.cloud.client.pollQueueTime}}, so it's 20s server side by default. 
Throwing there skips:
* waiting on the remaining clients
* {{doRetriesIfNeeded()}}, so retriable errors are never retried
* moving errors already collected from the other clients into {{getErrors()}}

{{getErrors()}} then comes back empty and 
{{DistributedZkUpdateProcessor.doDistribFinish}} treats the batch as 
successful. The client gets a success response, and the replicas that failed or 
stalled aren't put into a lower term, so they don't recover.

The attached test fails on main. One node accepts the connection and never 
reads, another refuses it. After {{finish()}}, {{getErrors()}} is {{[]}} even 
though the refused node's error was recorded about a second before the stall.

Proposed fix: catch the {{IOException}} per client in 
{{StreamingSolrClients.blockUntilFinished()}} and report it through that 
client's {{handleError}}. The stall becomes an error for the node that stalled, 
and the other clients are still waited on and retried.

Related to [SOLR-5939|https://issues.apache.org/jira/browse/SOLR-5939], which 
touches the same error-reporting path.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to