[
https://issues.apache.org/jira/browse/CASSANDRA-21654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Isaac Reath updated CASSANDRA-21654:
------------------------------------
Test and Documentation Plan: Added unit and dtests. Feature flag is
documented in cassandra.yaml and new metric is documented on the website.
Status: Patch Available (was: In Progress)
> Coordinator node experiences read failures on ungraceful shutdown of replica
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-21654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21654
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination
> Reporter: Isaac Reath
> Assignee: Isaac Reath
> Priority: Normal
> Fix For: 4.1.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When a replica node dies, the coordinator node can keep routing reads to it
> until the failure detector convicts the node. During this time, messages will
> continue to collect on the {{OutboundConnection}} of the dead replica until
> it reaches capacity, at which point the {{RequestCallbacks}} calls
> {{ReadCallback#onFailure}} which aborts the entire request, despite there
> being sufficient replicas alive to service the request.
> For RF=3 and CL=QUORUM, a single replica failure causes the whole query to
> abort unless the table is configured with {{speculative_retry = 'ALWAYS'}}.
> For higher replication factors (e.g., rf=5, rf=9), this prevents a read from
> succeeding even with {{speculative_retry = 'ALWAYS'}} and multiple concurrent
> node failures.
> This ticket proposes adding a second trigger for speculative retry beyond
> latency. In the event of an overloaded connection, we propose speculating
> immediately to another replica, letting the query succeed so long as there
> are sufficient replicas to satisfy consistency.
> This issue was originally hit on a production cluster with 18 nodes running
> 4.1.10 with roughly 2k reads/sec/node. I've also been able to reproduce using
> an in-JVM dtest off of cassandra-4.1 HEAD.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
