Malte Pickhan created CASSANDRA-12947:
-----------------------------------------
Summary: Repair not replicating data
Key: CASSANDRA-12947
URL: https://issues.apache.org/jira/browse/CASSANDRA-12947
Project: Cassandra
Issue Type: Bug
Environment: Cassandra 3.7
Single DC
7 Nodes
RF 3
NetworkTopologyStrategy
OS: Ubuntu
Reporter: Malte Pickhan
We experienced strange behaviour of our C* cluster last week.
In the logs we've seen multiple requests in the logs where we the values read
from Cassandra actually have been 'null'.
When running the query on a node, we sometimes received the result and
sometimes not.
When setting the CL to LOCAL_QUORUM everything was fine. So far nothing
unusual, probably the dataset wasn't replicated to one of the nodes.
When turning on the tracing and running the query there was following
intersting line:
{quote}
Initiating read-repair [SharedPool-Worker-2] | 2016-11-18 10:17:47.528000 |
$PUBLIC_IP | 126 | 127.0.0.1
Digest mismatch: org.apache.cassandra.service.DigestMismatchException:
Mismatch for key DecoratedKey(-5887526567589486157,
3130333031303338383436303937) (db1e86d507513ff12ba95f0eff984b60 vs
d41d8cd98f00b204e9800998ecf8427e) [ReadRepairStage:1]
{quote}
This is probably related to CASSANDRA-12090?
The interesting part is, after that we've run a 'nodetool repair -pr', after
that the behaviour was still the same and the data randomly not available,
depending on which node was hit.
Only after running a 'nodetool repair -pr -full' the issue was gone.
Did we miss something here? The point that's bothering me is that the dataset
was not replicated.
Worth to note is probably that some weeks ago we've hit the bug CASSANDRA-12694
and fixed it by scrubbing some tables.
Any hints/help are appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
