[
https://issues.apache.org/jira/browse/CASSANDRA-18766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872980#comment-17872980
]
Aswin Karthik commented on CASSANDRA-18766:
-------------------------------------------
Facing a similar issue:
We upgraded from 4.0.9 to 4.1.5. The speculative retries have increased a lot.
In some clusters, around 80% of requests are being speculatively retried (it
was <1% before the upgrade). % = sum(SpeculativeRetries.Count) /
sum(CoordinatorReadLatency.Count). The symptoms are similar to this resolved
ticket https://issues.apache.org/jira/browse/CASSANDRA-17877.
I am able to reproduce the issue as well
Create 2 clusters
{noformat}
ccm create cluster1 -v 4.1.5 -n 3
ccm create cluster2 -v 4.0.9 -n 3
{noformat}
On each cluster do the following
{noformat}
ccm stress write "duration=2m" -rate "threads=8" -schema
"replication(strategy=SimpleStrategy,replication_factor=2)"
ccm stress "read no-warmup duration=3m" -rate threads=8 -errors
"skip-read-validation"
{noformat}
Checking Speculative retries
{noformat}
# On 4.1.5
$ ccm node1 nodetool -- sjk mx -b
org.apache.cassandra.metrics:type=Table,keyspace=keyspace1,scope=standard1,name=SpeculativeRetries
-mg -f Count
org.apache.cassandra.metrics:type=Table,keyspace=keyspace1,scope=standard1,name=SpeculativeRetries
174252
# On 4.0.9
$ ccm node1 nodetool -- sjk mx -b
org.apache.cassandra.metrics:type=Table,keyspace=keyspace1,scope=standard1,name=SpeculativeRetries
-mg -f Count
org.apache.cassandra.metrics:type=Table,keyspace=keyspace1,scope=standard1,name=SpeculativeRetries
5860
{noformat}
The difference is massive
> high speculative retries on v4.1.3
> ----------------------------------
>
> Key: CASSANDRA-18766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18766
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
> Reporter: Ivans Novikovs
> Priority: Normal
> Fix For: 4.1.x
>
> Attachments: screenshot-1.png, screenshot-2.png, signature.asc,
> signature.asc, v4.0.png, v4.1.png
>
>
> There are up to 10+ times higher speculative retries for reads on 4.1.3
> comparing to 4.0.7 and 4.1.2 when using QUORUM and default setting of 99p.
> On 4.1.3 after upgrade I see speculative retries for up to 35% of all reads
> for specific table. Latency for reads is stable around 500 microseconds.
> java 1.8.0_382 is used
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
