[
https://issues.apache.org/jira/browse/CASSANDRA-21654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113013#comment-18113013
]
Isaac Reath commented on CASSANDRA-21654:
-----------------------------------------
I have a branch available here with an initial implementation on 4.1:
https://github.com/isaacreath/cassandra/tree/CASSANDRA-21654-4.1. The
implementation leverages the existing speculation infrastructure to speculate
on overload instead of aborting the request. I think I may want to also update
https://github.com/isaacreath/cassandra/blob/CASSANDRA-21654-4.1/src/java/org/apache/cassandra/net/OutboundConnection.java#L336-L344
to differentiate between an overload to an endpoint vs a global overload. In
the event of a global overload, I think we want to fail fast instead of try
every possible connection.
> Coordinator node experiences read failures on ungraceful shutdown of replica
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-21654
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21654
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination
> Reporter: Isaac Reath
> Assignee: Isaac Reath
> Priority: Normal
> Fix For: 4.1.x
>
>
> When a replica node dies, the coordinator node can keep routing reads to it
> until the failure detector convicts the node. During this time, messages will
> continue to collect on the {{OutboundConnection}} of the dead replica until
> it reaches capacity, at which point the {{RequestCallbacks}} calls
> {{ReadCallback#onFailure}} which aborts the entire request, despite there
> being sufficient replicas alive to service the request.
> For RF=3 and CL=QUORUM, a single replica failure causes the whole query to
> abort unless the table is configured with {{speculative_retry = 'ALWAYS'}}.
> For higher replication factors (e.g., rf=5, rf=9), this prevents a read from
> succeeding even with {{speculative_retry = 'ALWAYS'}} and multiple concurrent
> node failures.
> This ticket proposes adding a second trigger for speculative retry beyond
> latency. In the event of an overloaded connection, we propose speculating
> immediately to another replica, letting the query succeed so long as there
> are sufficient replicas to satisfy consistency.
> This issue was originally hit on a production cluster with 18 nodes running
> 4.1.10 with roughly 2k reads/sec/node. I've also been able to reproduce using
> an in-JVM dtest off of cassandra-4.1 HEAD.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
