[
https://issues.apache.org/jira/browse/CASSANDRA-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kishan Karunaratne updated CASSANDRA-11804:
-------------------------------------------
Description:
I have a 3-node cluster with a keyspace with RF=3, with some data inserted. I'm
using a DowngradingConsistency retry policy on the client. Performing a query
at CL=ALL with one node blocked (ccm pause/SIGSTOP), the query fails and
returns a ReadTimeout to the client as expected. The following is in the debug
log:
{noformat}
ReadCallback.java:126 - Timed out; received 2 of 3 responses (including data)
{noformat}
Now, the driver automatically retries once, and in this case at QUORUM. The
client receives a ReadTimeout once more: "Cassandra::Errors::ReadTimeoutError:
Operation timed out - received only 1 responses." This ReadTimeout should not
have happened because we still have 2 good nodes/hosts to retrieve data from.
This is in the debug log:
{noformat}
ReadCallback.java:126 - Timed out; received 1 of 2 responses (including data)
{noformat}
The weird part is that the query does occasionally succeed (at QUORUM), I'd say
< 50% of the time. I did the same experiment with two nodes blocked and I get
"Cassandra::Errors::ReadTimeoutError: Operation timed out - received only 0
responses.":
{noformat}
ReadCallback.java:126 - Timed out; received 1 of 3 responses (including data)
ReadCallback.java:126 - Timed out; received 0 of 1 responses
{noformat}
I expect this query to have succeeded at ONE, because there is one good node
left (and it's the one used as the coordinator node). In both cases, I feel
like the coordinator node doesn't count itself as a replica for the CL.
was:
I have a 3-node cluster with a keyspace with RF=3, with some data inserted. I'm
using a DowngradingConsistency retry policy on the client. Performing a query
at CL=ALL with node blocked (ccm pause/SIGSTOP), the query fails and returns a
ReadTimeout to the client as expected. The following is in the debug log:
{noformat}
ReadCallback.java:126 - Timed out; received 2 of 3 responses (including data)
{noformat}
Now, the driver automatically retries once, and in this case at QUORUM. The
client receives a ReadTimeout once more: "Cassandra::Errors::ReadTimeoutError:
Operation timed out - received only 1 responses." This readtimeout should not
have happened because we still have 2 good nodes/hosts to retrieve data from.
This is in the debug log:
{noformat}
ReadCallback.java:126 - Timed out; received 1 of 2 responses (including data)
{noformat}
The weird part is that the query does occasionally succeed (at QUORUM), I'd say
< 50% of the time. I did the same experiment with two nodes blocked and I get
"Cassandra::Errors::ReadTimeoutError: Operation timed out - received only 0
responses.":
{noformat}
ReadCallback.java:126 - Timed out; received 1 of 3 responses (including data)
ReadCallback.java:126 - Timed out; received 0 of 1 responses
{noformat}
I expect this query to have succeeded at ONE, because there is one good node
left (and it's the one used as the coordinator node). In both cases, I feel
like the coordinator node doesn't count itself as a replica for the CL.
> Read request at proper CL returns ReadTimeout occasionally
> ----------------------------------------------------------
>
> Key: CASSANDRA-11804
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11804
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 3.4 | Ruby-driver 3.0-rc2
> Reporter: Kishan Karunaratne
>
> I have a 3-node cluster with a keyspace with RF=3, with some data inserted.
> I'm using a DowngradingConsistency retry policy on the client. Performing a
> query at CL=ALL with one node blocked (ccm pause/SIGSTOP), the query fails
> and returns a ReadTimeout to the client as expected. The following is in the
> debug log:
> {noformat}
> ReadCallback.java:126 - Timed out; received 2 of 3 responses (including data)
> {noformat}
> Now, the driver automatically retries once, and in this case at QUORUM. The
> client receives a ReadTimeout once more:
> "Cassandra::Errors::ReadTimeoutError: Operation timed out - received only 1
> responses." This ReadTimeout should not have happened because we still have 2
> good nodes/hosts to retrieve data from. This is in the debug log:
> {noformat}
> ReadCallback.java:126 - Timed out; received 1 of 2 responses (including data)
> {noformat}
> The weird part is that the query does occasionally succeed (at QUORUM), I'd
> say < 50% of the time. I did the same experiment with two nodes blocked and I
> get "Cassandra::Errors::ReadTimeoutError: Operation timed out - received only
> 0 responses.":
> {noformat}
> ReadCallback.java:126 - Timed out; received 1 of 3 responses (including data)
> ReadCallback.java:126 - Timed out; received 0 of 1 responses
> {noformat}
> I expect this query to have succeeded at ONE, because there is one good node
> left (and it's the one used as the coordinator node). In both cases, I feel
> like the coordinator node doesn't count itself as a replica for the CL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
