[
https://issues.apache.org/jira/browse/CASSANDRA-21654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Isaac Reath updated CASSANDRA-21654:
------------------------------------
Bug Category: Parent values: Availability(12983)
Complexity: Normal
Discovered By: User Report
Fix Version/s: 4.1.x
Severity: Low
Status: Open (was: Triage Needed)
> Coordinator node experiences read failures on ungraceful shutdown of replica
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-21654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21654
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination
> Reporter: Isaac Reath
> Assignee: Isaac Reath
> Priority: Normal
> Fix For: 4.1.x
>
>
> When a replica node dies, the coordinator node can keep routing reads to it
> until the failure detector convicts the node. During this time, messages will
> continue to collect on the {{OutboundConnection}} of the dead replica until
> it reaches capacity, at which point the {{RequestCallbacks}} calls
> {{ReadCallback#onFailure}} which aborts the entire request, despite there
> being sufficient replicas alive to service the request.
> For RF=3 and CL=QUORUM, a single replica failure causes the whole query to
> abort unless the table is configured with {{speculative_retry = 'ALWAYS'}}.
> For higher replication factors (e.g., rf=5, rf=9), this prevents a read from
> succeeding even with {{speculative_retry = 'ALWAYS'}} and multiple concurrent
> node failures.
> This ticket proposes adding a second trigger for speculative retry beyond
> latency. In the event of an overloaded connection, we propose speculating
> immediately to another replica, letting the query succeed so long as there
> are sufficient replicas to satisfy consistency.
> This issue was originally hit on a production cluster with 18 nodes running
> 4.1.10 with roughly 2k reads/sec/node. I've also been able to reproduce using
> an in-JVM dtest off of cassandra-4.1 HEAD.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
