Amrit Sarkar created SOLR-18656:
-----------------------------------
Summary: SolrCmdDistributor.finish() drops recorded errors when a
stream stalls
Key: SOLR-18656
URL: https://issues.apache.org/jira/browse/SOLR-18656
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Amrit Sarkar
{{SolrCmdDistributor.finish()}} catches and logs the {{IOException}} from
{{blockAndDoRetries()}}:
{code:java}
try {
blockAndDoRetries();
} catch (IOException e) {
log.warn("Unable to finish sending updates", e);
}
{code}
{{StreamingSolrClients.blockUntilFinished()}} throws that as soon as one client
hits {{StallDetection.stallCheck()}}, i.e. no progress with a non-empty queue
for {{solr.cloud.client.stallTime}}. The stall time is raised to 2x
{{solr.cloud.client.pollQueueTime}}, so it's 20s server side by default.
Throwing there skips:
* waiting on the remaining clients
* {{doRetriesIfNeeded()}}, so retriable errors are never retried
* moving errors already collected from the other clients into {{getErrors()}}
{{getErrors()}} then comes back empty and
{{DistributedZkUpdateProcessor.doDistribFinish}} treats the batch as
successful. The client gets a success response, and the replicas that failed or
stalled aren't put into a lower term, so they don't recover.
The attached test fails on main. One node accepts the connection and never
reads, another refuses it. After {{finish()}}, {{getErrors()}} is {{[]}} even
though the refused node's error was recorded about a second before the stall.
Proposed fix: catch the {{IOException}} per client in
{{StreamingSolrClients.blockUntilFinished()}} and report it through that
client's {{handleError}}. The stall becomes an error for the node that stalled,
and the other clients are still waited on and retried.
Related to [SOLR-5939|https://issues.apache.org/jira/browse/SOLR-5939], which
touches the same error-reporting path.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]