Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

Hello, We're seeing inconsistent data while doing reads on cassandra. Here are the details: It's is a wide column table. The columns can be added my multiple machines, and read by multiple machines. The time between writes and reads are in minutes, but sometimes can be in seconds. Writes

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

caution when clicking links or opening attachments. ] Are you SURE there are no writes to that table coming from another DC? -- Jeff Jirsa On Oct 15, 2018, at 5:34 PM, Naik, Ninad mailto:ninad.n...@epsilon.com>> wrote: Thanks Jeff. We're not doing deletes, but I will take a look at thi

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

Yup. Verified again that this table is being written, read, and replicated to just one data center. From: Naik, Ninad Sent: Monday, October 15, 2018 1:18:43 PM To: user@cassandra.apache.org Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM) Let

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

Thanks Mick. We're on datastax java driver version: 2.1.10.2 and we aren't using the client side timestamps. Anyway, we went ahead and verified that all client machines and cassandra machines are in sync with regards to time. We've also verified that no reads and writes are going to the

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

12, 2018, at 1:50 PM, Naik, Ninad mailto:ninad.n...@epsilon.com>> wrote: Hello, We're seeing inconsistent data while doing reads on cassandra. Here are the details: It's is a wide column table. The columns can be added my multiple machines, and read by multiple machines. The time b

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

it, you need your client clocks to be in sync also. On Sat, Oct 13, 2018 at 1:24 PM Naik, Ninad mailto:ninad.n...@epsilon.com>> wrote: Thanks Maitrayee. I should have mentioned this as one of the things we verified. The clocks on cassandra nodes are in sync. ___

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

on reads (LOCAL_QUORUM) [ This email has been sent from a source external to Epsilon. Please use caution when clicking links or opening attachments. ] We have seen inconsistent read if the clock on the nodes are not in sync. Thank you Sent from my iPhone On Oct 12, 2018, at 1:50 PM, Naik, Ninad

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

), it won’t be backported to 2.1 because it’s a significant change to how reads are executed -- Jeff Jirsa On Oct 13, 2018, at 7:24 PM, Naik, Ninad mailto:ninad.n...@epsilon.com>> wrote: Thanks Maitrayee. I should have mentioned this as one of the things we verified. The clocks on cassandra

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

Mick, sorry I think I missed your following questions: - SPECULATIVE_RETRY='ALWAYS' We saw this issue a couple of times a few years ago. That's why we introduced this change. Although at that time, it was on cassandra 1.x version. - Topology changes: The only change we did was that we added

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

Thanks Mick. Yeah we are planning to try with tracing and by enabling trace level logs for a short duration. I will update this thread with the related details. One other thing we verified is that these partial reads happen all across the cluster. It's not limited to certain cassandra