You have RF=3 and both read & write CL=1, which means you are asking
Cassandra to give up strong consistency in order to gain higher
availability and perhaps slight faster speed, and that's what you get.
If you want to have strong consistency, you will need to make sure (read
CL + write CL) > RF.
On 10/10/2021 11:55, Isaeed Mohanna wrote:
Hi
We had a cluster with 3 Nodes with Replication Factor 2 and we were
using read with consistency Level One.
We recently added a 4^th node and changed the replication factor to 3,
once this was done apps reading from DB with CL1 would receive an
empty record, Looking around I was surprised to learn that upon
changing the replication factor if the read request is sent to a node
the should own the record according to the new replication factor
while it still doesn’t have it yet then an empty record will be
returned because of CL1, the record will be written to that node after
the repair operation is over.
We ran the repair operation which took days in our case (we had to
change apps to CL2 to avoid serious data inconsistencies).
Now the repair operations are over and if I revert to CL1 we are still
getting errors that records do not exist in DB while they do, using
CL2 again it works fine.
Any ideas what I am missing?
Is there a way to validate that the repairs task has actually done
what is needed and that the data is actually now replicated RF3 ?
Could it it be a Cassandra Driver issue? Since if I issue the request
in cqlsh I do get the record but I cannot know if I am hitting the
replica that doesn’t hold the record
Thanks for your help
