Mick, sorry I think I missed your following questions:

- SPECULATIVE_RETRY='ALWAYS'

We saw this issue a couple of times a few years ago. That's why we introduced 
this change. Although at that time, it was on cassandra 1.x version.


- Topology changes:

The only change we did was that we added nodes for 2 months. But that process 
finished 1 month ago. Since then we've run multiple repairs on the table. Other 
than that no other topology changes were done.

________________________________
From: Mick Semb Wever <m...@thelastpickle.com>
Sent: Wednesday, October 24, 2018 4:12:57 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

[ This email has been sent from a source external to Epsilon. Please use 
caution when clicking links or opening attachments. ]

Ninad,

> Here's a bit more information:
>
> -Few rows in this column family can grow quite wide (> 100K columns)
>
> -But we keep seeing this behavior most frequently with rows with just 1 or 
> two columns . The typical behavior is: Machine A adds a new row and a column. 
> 30-60 seconds later Machine B tries to read this row. It doesn't find the 
> row. So the application retries within 500ms. This time it finds the row.


You wrote a lot of useful info in your original post, sorry I missed
it in my first reply.
Only thing there that stands out, apart from short reads that Jeff's
already pointed out, is the use of `speculative_retry='ALWAYS'`. Has
there topology changes in your cluster recently?

Next step would be to try and repeat it with tracing.

regards,
Mick


--
Mick Semb Wever
Australia

The Last Pickle
Apache Cassandra Consulting
http://www.thelastpickle.com

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org


The information contained in this e-mail message and any attachments may be 
privileged and confidential. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that any review, dissemination, distribution or copying 
of this communication is strictly prohibited. If you have received this 
communication in error, please notify the sender immediately by replying to 
this e-mail and delete the message and any attachments from your computer.

Reply via email to