Hi
We had a cluster with 3 Nodes with Replication Factor 2 and we were using read
with consistency Level One.
We recently added a 4th node and changed the replication factor to 3, once this
was done apps reading from DB with CL1 would receive an empty record, Looking
around I was surprised to learn that upon changing the replication factor if
the read request is sent to a node the should own the record according to the
new replication factor while it still doesn't have it yet then an empty record
will be returned because of CL1, the record will be written to that node after
the repair operation is over.
We ran the repair operation which took days in our case (we had to change apps
to CL2 to avoid serious data inconsistencies).
Now the repair operations are over and if I revert to CL1 we are still getting
errors that records do not exist in DB while they do, using CL2 again it works
fine.
Any ideas what I am missing?
Is there a way to validate that the repairs task has actually done what is
needed and that the data is actually now replicated RF3 ?
Could it it be a Cassandra Driver issue? Since if I issue the request in cqlsh
I do get the record but I cannot know if I am hitting the replica that doesn't
hold the record
Thanks for your help
