Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Mick, sorry I think I missed your following questions: - SPECULATIVE_RETRY='ALWAYS' We saw this issue a couple of times a few years ago. That's why we introduced this change. Although at that time, it was on cassandra 1.x version. - Topology changes: The only change we did was that we added nodes for 2 months. But that process finished 1 month ago. Since then we've run multiple repairs on the table. Other than that no other topology changes were done. From: Mick Semb Wever Sent: Wednesday, October 24, 2018 4:12:57 AM To: user@cassandra.apache.org Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM) [ This email has been sent from a source external to Epsilon. Please use caution when clicking links or opening attachments. ] Ninad, > Here's a bit more information: > > -Few rows in this column family can grow quite wide (> 100K columns) > > -But we keep seeing this behavior most frequently with rows with just 1 or > two columns . The typical behavior is: Machine A adds a new row and a column. > 30-60 seconds later Machine B tries to read this row. It doesn't find the > row. So the application retries within 500ms. This time it finds the row. You wrote a lot of useful info in your original post, sorry I missed it in my first reply. Only thing there that stands out, apart from short reads that Jeff's already pointed out, is the use of `speculative_retry='ALWAYS'`. Has there topology changes in your cluster recently? Next step would be to try and repeat it with tracing. regards, Mick -- Mick Semb Wever Australia The Last Pickle Apache Cassandra Consulting http://www.thelastpickle.com - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this e-mail and delete the message and any attachments from your computer.
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Thanks Mick. Yeah we are planning to try with tracing and by enabling trace level logs for a short duration. I will update this thread with the related details. One other thing we verified is that these partial reads happen all across the cluster. It's not limited to certain cassandra nodes. Meanwhile, can following scenarios (however unlikely they might seem) ever happen? - Can a bloom filter malfunction (Ex: For some reason, bloom filter says to skip a SSTable which has the key) - Can the co-ordinator ever send a read request to a wrong replica (Ex: for some reason it calculated a wrong decorated key)? - Can a replica ever drop a read in case of an exception? - Can read timeouts for other wide rows (we've a few rows with > 100k cols) impact the entire cluster to behave in a delayed manner? Thanks again for all of your help ! From: Mick Semb Wever Sent: Wednesday, October 24, 2018 4:12:57 AM To: user@cassandra.apache.org Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM) [ This email has been sent from a source external to Epsilon. Please use caution when clicking links or opening attachments. ] Ninad, > Here's a bit more information: > > -Few rows in this column family can grow quite wide (> 100K columns) > > -But we keep seeing this behavior most frequently with rows with just 1 or > two columns . The typical behavior is: Machine A adds a new row and a column. > 30-60 seconds later Machine B tries to read this row. It doesn't find the > row. So the application retries within 500ms. This time it finds the row. You wrote a lot of useful info in your original post, sorry I missed it in my first reply. Only thing there that stands out, apart from short reads that Jeff's already pointed out, is the use of `speculative_retry='ALWAYS'`. Has there topology changes in your cluster recently? Next step would be to try and repeat it with tracing. regards, Mick -- Mick Semb Wever Australia The Last Pickle Apache Cassandra Consulting http://www.thelastpickle.com - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this e-mail and delete the message and any attachments from your computer.
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Ninad, > Here's a bit more information: > > -Few rows in this column family can grow quite wide (> 100K columns) > > -But we keep seeing this behavior most frequently with rows with just 1 or > two columns . The typical behavior is: Machine A adds a new row and a column. > 30-60 seconds later Machine B tries to read this row. It doesn't find the > row. So the application retries within 500ms. This time it finds the row. You wrote a lot of useful info in your original post, sorry I missed it in my first reply. Only thing there that stands out, apart from short reads that Jeff's already pointed out, is the use of `speculative_retry='ALWAYS'`. Has there topology changes in your cluster recently? Next step would be to try and repeat it with tracing. regards, Mick -- Mick Semb Wever Australia The Last Pickle Apache Cassandra Consulting http://www.thelastpickle.com - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Thanks Mick. We're on datastax java driver version: 2.1.10.2 and we aren't using the client side timestamps. Anyway, we went ahead and verified that all client machines and cassandra machines are in sync with regards to time. We've also verified that no reads and writes are going to the remote data center. Here's a bit more information: -Few rows in this column family can grow quite wide (> 100K columns) -But we keep seeing this behavior most frequently with rows with just 1 or two columns . The typical behavior is: Machine A adds a new row and a column. 30-60 seconds later Machine B tries to read this row. It doesn't find the row. So the application retries within 500ms. This time it finds the row. Thanks. From: Mick Semb Wever Sent: Saturday, October 20, 2018 10:24:53 PM To: user@cassandra.apache.org Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM) [ This email has been sent from a source external to Epsilon. Please use caution when clicking links or opening attachments. ] > Thanks James. Yeah, we're using the datastax java driver. But we're on > version 2.1.10.2. And we are not using the client side timestamps. Just to check Ninad. If you are using Cassandra-2.1 (native protocol v3) and the java driver version 3.0 or above, then you would be using client-side timestamps by default. https://github.com/datastax/java-driver/tree/3.x/manual/query_timestamps With client-side timestamps all client servers and all C* nodes must be kept tightly in-sync, as Elliot said. Monitoring and alerting on any clock skew on any of these machines is important. Also worth checking that any local_quorum requests are not accidentally go to the wrong datacenter. regards, Mick - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org The information contained in this e-mail message and any attachments may be privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this e-mail and delete the message and any attachments from your computer.
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
> Thanks James. Yeah, we're using the datastax java driver. But we're on > version 2.1.10.2. And we are not using the client side timestamps. Just to check Ninad. If you are using Cassandra-2.1 (native protocol v3) and the java driver version 3.0 or above, then you would be using client-side timestamps by default. https://github.com/datastax/java-driver/tree/3.x/manual/query_timestamps With client-side timestamps all client servers and all C* nodes must be kept tightly in-sync, as Elliot said. Monitoring and alerting on any clock skew on any of these machines is important. Also worth checking that any local_quorum requests are not accidentally go to the wrong datacenter. regards, Mick - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Yup. Verified again that this table is being written, read, and replicated to
just one data center.
From: Naik, Ninad
Sent: Monday, October 15, 2018 1:18:43 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Let me double check that.
From: Jeff Jirsa
Sent: Monday, October 15, 2018 11:49:10 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
Are you SURE there are no writes to that table coming from another DC?
--
Jeff Jirsa
On Oct 15, 2018, at 5:34 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Thanks Jeff. We're not doing deletes, but I will take a look at this jira.
From: Jeff Jirsa mailto:jji...@gmail.com>>
Sent: Sunday, October 14, 2018 12:55:17 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
If this is 2.1 AND you do deletes AND you have a non-zero number of failed
writes (timeouts), it’s possibly short reads
3.0 fixes this ( https://issues.apache.org/jira/browse/CASSANDRA-12872 ), it
won’t be backported to 2.1 because it’s a significant change to how reads are
executed
--
Jeff Jirsa
On Oct 13, 2018, at 7:24 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Thanks Maitrayee. I should have mentioned this as one of the things we
verified. The clocks on cassandra nodes are in sync.
From: maitrayee shah
mailto:koolja...@yahoo.com.INVALID>>
Sent: Friday, October 12, 2018 6:40:25 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
We have seen inconsistent read if the clock on the nodes are not in sync.
Thank you
Sent from my iPhone
On Oct 12, 2018, at 1:50 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Hello,
We're seeing inconsistent data while doing reads on cassandra. Here are the
details:
It's is a wide column table. The columns can be added my multiple machines, and
read by multiple machines. The time between writes and reads are in minutes,
but sometimes can be in seconds. Writes happen every 2 minutes.
Now, while reading we're seeing the following cases of inconsistent reads:
* One column was added. If a read was done after the column was added (20
secs to 2 minutes after the write), Cassandra returns no data. As if the key
doesn't exist. If the application retries, it gets the data.
* A few columns exist for a row key. And a new column 'n' was added. Again,
a read happens a few minutes after the write. This time, only the latest column
'n' is returned. In this case the app doesn't know that the data is incomplete
so it doesn't retry. If we manually retry, we see all the columns.
* A few columns exist for a row key. And a new column 'n' is added. When a
read happens after the write, all columns but 'n' are returned.
Here's what we've verified:
* Both writes and reads are using 'LOCAL_QUORUM' consistency level.
* The replication is within local data center. No remote data center is
involved in the read or write.
* During the inconsistent reads, none of the nodes are undergoing GC pauses
* There are no errors in cassandra logs
* Reads always happen after the writes.
A few other details: Cassandra version: 2.1.9 DataStax java driver version:
2.1.10.2 Replication Factor: 3
We don't see this problem in lower environments. We have seen this happen once
or twice last year, but since last few days it's happening quite frequently. On
an average 2 inconsistent reads every minute.
Here's how the table definition looks like:
CREATE TABLE "MY_TABLE" (
key text,
sub_key text,
value text,
PRIMARY KEY ((key), sub_key)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
comment='' AND
dclocal_read_repair_chance=0.10 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.00 AND
default_time_to_live=0 AND
speculative_retry='ALWAYS' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
Please point us in the right direction. Thanks !
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this mess
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Let me double check that.
From: Jeff Jirsa
Sent: Monday, October 15, 2018 11:49:10 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
Are you SURE there are no writes to that table coming from another DC?
--
Jeff Jirsa
On Oct 15, 2018, at 5:34 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Thanks Jeff. We're not doing deletes, but I will take a look at this jira.
From: Jeff Jirsa mailto:jji...@gmail.com>>
Sent: Sunday, October 14, 2018 12:55:17 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
If this is 2.1 AND you do deletes AND you have a non-zero number of failed
writes (timeouts), it’s possibly short reads
3.0 fixes this ( https://issues.apache.org/jira/browse/CASSANDRA-12872 ), it
won’t be backported to 2.1 because it’s a significant change to how reads are
executed
--
Jeff Jirsa
On Oct 13, 2018, at 7:24 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Thanks Maitrayee. I should have mentioned this as one of the things we
verified. The clocks on cassandra nodes are in sync.
From: maitrayee shah
mailto:koolja...@yahoo.com.INVALID>>
Sent: Friday, October 12, 2018 6:40:25 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
We have seen inconsistent read if the clock on the nodes are not in sync.
Thank you
Sent from my iPhone
On Oct 12, 2018, at 1:50 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Hello,
We're seeing inconsistent data while doing reads on cassandra. Here are the
details:
It's is a wide column table. The columns can be added my multiple machines, and
read by multiple machines. The time between writes and reads are in minutes,
but sometimes can be in seconds. Writes happen every 2 minutes.
Now, while reading we're seeing the following cases of inconsistent reads:
* One column was added. If a read was done after the column was added (20
secs to 2 minutes after the write), Cassandra returns no data. As if the key
doesn't exist. If the application retries, it gets the data.
* A few columns exist for a row key. And a new column 'n' was added. Again,
a read happens a few minutes after the write. This time, only the latest column
'n' is returned. In this case the app doesn't know that the data is incomplete
so it doesn't retry. If we manually retry, we see all the columns.
* A few columns exist for a row key. And a new column 'n' is added. When a
read happens after the write, all columns but 'n' are returned.
Here's what we've verified:
* Both writes and reads are using 'LOCAL_QUORUM' consistency level.
* The replication is within local data center. No remote data center is
involved in the read or write.
* During the inconsistent reads, none of the nodes are undergoing GC pauses
* There are no errors in cassandra logs
* Reads always happen after the writes.
A few other details: Cassandra version: 2.1.9 DataStax java driver version:
2.1.10.2 Replication Factor: 3
We don't see this problem in lower environments. We have seen this happen once
or twice last year, but since last few days it's happening quite frequently. On
an average 2 inconsistent reads every minute.
Here's how the table definition looks like:
CREATE TABLE "MY_TABLE" (
key text,
sub_key text,
value text,
PRIMARY KEY ((key), sub_key)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
comment='' AND
dclocal_read_repair_chance=0.10 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.00 AND
default_time_to_live=0 AND
speculative_retry='ALWAYS' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
Please point us in the right direction. Thanks !
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that any review, dissemination, distribution or copying
of this communication is strictly prohibited. If you have received this
communication in error, ple
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Are you SURE there are no writes to that table coming from another DC?
--
Jeff Jirsa
> On Oct 15, 2018, at 5:34 PM, Naik, Ninad wrote:
>
> Thanks Jeff. We're not doing deletes, but I will take a look at this jira.
> From: Jeff Jirsa
> Sent: Sunday, October 14, 2018 12:55:17 PM
> To: user@cassandra.apache.org
> Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
>
> If this is 2.1 AND you do deletes AND you have a non-zero number of failed
> writes (timeouts), it’s possibly short reads
>
> 3.0 fixes this ( https://issues.apache.org/jira/browse/CASSANDRA-12872 ), it
> won’t be backported to 2.1 because it’s a significant change to how reads are
> executed
>
>
> --
> Jeff Jirsa
>
>
> On Oct 13, 2018, at 7:24 PM, Naik, Ninad wrote:
>
>> Thanks Maitrayee. I should have mentioned this as one of the things we
>> verified. The clocks on cassandra nodes are in sync.
>> From: maitrayee shah
>> Sent: Friday, October 12, 2018 6:40:25 PM
>> To: user@cassandra.apache.org
>> Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>>
>> [ This email has been sent from a source external to Epsilon. Please use
>> caution when clicking links or opening attachments. ]
>>
>> We have seen inconsistent read if the clock on the nodes are not in sync.
>>
>>
>> Thank you
>>
>> Sent from my iPhone
>>
>> On Oct 12, 2018, at 1:50 PM, Naik, Ninad wrote:
>>
>>> Hello,
>>>
>>> We're seeing inconsistent data while doing reads on cassandra. Here are the
>>> details:
>>>
>>> It's is a wide column table. The columns can be added my multiple machines,
>>> and read by multiple machines. The time between writes and reads are in
>>> minutes, but sometimes can be in seconds. Writes happen every 2 minutes.
>>>
>>> Now, while reading we're seeing the following cases of inconsistent reads:
>>>
>>> One column was added. If a read was done after the column was added (20
>>> secs to 2 minutes after the write), Cassandra returns no data. As if the
>>> key doesn't exist. If the application retries, it gets the data.
>>> A few columns exist for a row key. And a new column 'n' was added. Again, a
>>> read happens a few minutes after the write. This time, only the latest
>>> column 'n' is returned. In this case the app doesn't know that the data is
>>> incomplete so it doesn't retry. If we manually retry, we see all the
>>> columns.
>>> A few columns exist for a row key. And a new column 'n' is added. When a
>>> read happens after the write, all columns but 'n' are returned.
>>> Here's what we've verified:
>>>
>>> Both writes and reads are using 'LOCAL_QUORUM' consistency level.
>>> The replication is within local data center. No remote data center is
>>> involved in the read or write.
>>> During the inconsistent reads, none of the nodes are undergoing GC pauses
>>> There are no errors in cassandra logs
>>> Reads always happen after the writes.
>>> A few other details: Cassandra version: 2.1.9 DataStax java driver version:
>>> 2.1.10.2 Replication Factor: 3
>>>
>>> We don't see this problem in lower environments. We have seen this happen
>>> once or twice last year, but since last few days it's happening quite
>>> frequently. On an average 2 inconsistent reads every minute.
>>>
>>> Here's how the table definition looks like:
>>>
>>> CREATE TABLE "MY_TABLE" (
>>> key text,
>>> sub_key text,
>>> value text,
>>> PRIMARY KEY ((key), sub_key)
>>> ) WITH
>>> bloom_filter_fp_chance=0.01 AND
>>> caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
>>> comment='' AND
>>> dclocal_read_repair_chance=0.10 AND
>>> gc_grace_seconds=864000 AND
>>> read_repair_chance=0.00 AND
>>> default_time_to_live=0 AND
>>> speculative_retry='ALWAYS' AND
>>> memtable_flush_period_in_ms=0 AND
>>> compaction={'class': 'SizeTieredCompactionStrategy'} AND
>>> compression={'sstable_compression': 'LZ4Compressor'};
>>> Please point us in the right direction. Thanks !
>>>
>>>
>>>
>>> The information contained in this e-mail message and any attachments may b
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Thanks Jeff. We're not doing deletes, but I will take a look at this jira.
From: Jeff Jirsa
Sent: Sunday, October 14, 2018 12:55:17 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
If this is 2.1 AND you do deletes AND you have a non-zero number of failed
writes (timeouts), it’s possibly short reads
3.0 fixes this ( https://issues.apache.org/jira/browse/CASSANDRA-12872 ), it
won’t be backported to 2.1 because it’s a significant change to how reads are
executed
--
Jeff Jirsa
On Oct 13, 2018, at 7:24 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Thanks Maitrayee. I should have mentioned this as one of the things we
verified. The clocks on cassandra nodes are in sync.
From: maitrayee shah
mailto:koolja...@yahoo.com.INVALID>>
Sent: Friday, October 12, 2018 6:40:25 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
We have seen inconsistent read if the clock on the nodes are not in sync.
Thank you
Sent from my iPhone
On Oct 12, 2018, at 1:50 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Hello,
We're seeing inconsistent data while doing reads on cassandra. Here are the
details:
It's is a wide column table. The columns can be added my multiple machines, and
read by multiple machines. The time between writes and reads are in minutes,
but sometimes can be in seconds. Writes happen every 2 minutes.
Now, while reading we're seeing the following cases of inconsistent reads:
* One column was added. If a read was done after the column was added (20
secs to 2 minutes after the write), Cassandra returns no data. As if the key
doesn't exist. If the application retries, it gets the data.
* A few columns exist for a row key. And a new column 'n' was added. Again,
a read happens a few minutes after the write. This time, only the latest column
'n' is returned. In this case the app doesn't know that the data is incomplete
so it doesn't retry. If we manually retry, we see all the columns.
* A few columns exist for a row key. And a new column 'n' is added. When a
read happens after the write, all columns but 'n' are returned.
Here's what we've verified:
* Both writes and reads are using 'LOCAL_QUORUM' consistency level.
* The replication is within local data center. No remote data center is
involved in the read or write.
* During the inconsistent reads, none of the nodes are undergoing GC pauses
* There are no errors in cassandra logs
* Reads always happen after the writes.
A few other details: Cassandra version: 2.1.9 DataStax java driver version:
2.1.10.2 Replication Factor: 3
We don't see this problem in lower environments. We have seen this happen once
or twice last year, but since last few days it's happening quite frequently. On
an average 2 inconsistent reads every minute.
Here's how the table definition looks like:
CREATE TABLE "MY_TABLE" (
key text,
sub_key text,
value text,
PRIMARY KEY ((key), sub_key)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
comment='' AND
dclocal_read_repair_chance=0.10 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.00 AND
default_time_to_live=0 AND
speculative_retry='ALWAYS' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
Please point us in the right direction. Thanks !
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that any review, dissemination, distribution or copying
of this communication is strictly prohibited. If you have received this
communication in error, please notify the sender immediately by replying to
this e-mail and delete the message and any attachments from your computer.
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that any review, dissemination, distribution or copying
of this communication is strictly prohibited. If you have received this
communication in error, please notify the sender immediately by replying to
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
If this is 2.1 AND you do deletes AND you have a non-zero number of failed
writes (timeouts), it’s possibly short reads
3.0 fixes this ( https://issues.apache.org/jira/browse/CASSANDRA-12872 ), it
won’t be backported to 2.1 because it’s a significant change to how reads are
executed
--
Jeff Jirsa
> On Oct 13, 2018, at 7:24 PM, Naik, Ninad wrote:
>
> Thanks Maitrayee. I should have mentioned this as one of the things we
> verified. The clocks on cassandra nodes are in sync.
> From: maitrayee shah
> Sent: Friday, October 12, 2018 6:40:25 PM
> To: user@cassandra.apache.org
> Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
>
> We have seen inconsistent read if the clock on the nodes are not in sync.
>
>
> Thank you
>
> Sent from my iPhone
>
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad wrote:
>
>> Hello,
>>
>> We're seeing inconsistent data while doing reads on cassandra. Here are the
>> details:
>>
>> It's is a wide column table. The columns can be added my multiple machines,
>> and read by multiple machines. The time between writes and reads are in
>> minutes, but sometimes can be in seconds. Writes happen every 2 minutes.
>>
>> Now, while reading we're seeing the following cases of inconsistent reads:
>>
>> One column was added. If a read was done after the column was added (20 secs
>> to 2 minutes after the write), Cassandra returns no data. As if the key
>> doesn't exist. If the application retries, it gets the data.
>> A few columns exist for a row key. And a new column 'n' was added. Again, a
>> read happens a few minutes after the write. This time, only the latest
>> column 'n' is returned. In this case the app doesn't know that the data is
>> incomplete so it doesn't retry. If we manually retry, we see all the columns.
>> A few columns exist for a row key. And a new column 'n' is added. When a
>> read happens after the write, all columns but 'n' are returned.
>> Here's what we've verified:
>>
>> Both writes and reads are using 'LOCAL_QUORUM' consistency level.
>> The replication is within local data center. No remote data center is
>> involved in the read or write.
>> During the inconsistent reads, none of the nodes are undergoing GC pauses
>> There are no errors in cassandra logs
>> Reads always happen after the writes.
>> A few other details: Cassandra version: 2.1.9 DataStax java driver version:
>> 2.1.10.2 Replication Factor: 3
>>
>> We don't see this problem in lower environments. We have seen this happen
>> once or twice last year, but since last few days it's happening quite
>> frequently. On an average 2 inconsistent reads every minute.
>>
>> Here's how the table definition looks like:
>>
>> CREATE TABLE "MY_TABLE" (
>> key text,
>> sub_key text,
>> value text,
>> PRIMARY KEY ((key), sub_key)
>> ) WITH
>> bloom_filter_fp_chance=0.01 AND
>> caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
>> comment='' AND
>> dclocal_read_repair_chance=0.10 AND
>> gc_grace_seconds=864000 AND
>> read_repair_chance=0.00 AND
>> default_time_to_live=0 AND
>> speculative_retry='ALWAYS' AND
>> memtable_flush_period_in_ms=0 AND
>> compaction={'class': 'SizeTieredCompactionStrategy'} AND
>> compression={'sstable_compression': 'LZ4Compressor'};
>> Please point us in the right direction. Thanks !
>>
>>
>>
>> The information contained in this e-mail message and any attachments may be
>> privileged and confidential. If the reader of this message is not the
>> intended recipient or an agent responsible for delivering it to the intended
>> recipient, you are hereby notified that any review, dissemination,
>> distribution or copying of this communication is strictly prohibited. If you
>> have received this communication in error, please notify the sender
>> immediately by replying to this e-mail and delete the message and any
>> attachments from your computer.
>>
>
>
>
> The information contained in this e-mail message and any attachments may be
> privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the intended
> recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If you
> have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Okay, just thought I’d throw it out there. I just so happened to miss that
little tidbit from the upgrade guide and got bit by it.
That is an odd problem, then. I hope you find your answer. Please update
us if you figure it out. Very curious to hear the answer.
On Sat, Oct 13, 2018 at 4:09 PM Naik, Ninad wrote:
> Thanks James. Yeah, we're using the datastax java driver. But we're on
> version 2.1.10.2. And we are not using the client side timestamps.
> --
> *From:* James Carman
> *Sent:* Saturday, October 13, 2018 12:55:51 PM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
> Are you using the Java driver? The default timestamp generator is
> client-side now (since 3.0 I believe), so if you don’t change it, you need
> your client clocks to be in sync also.
> On Sat, Oct 13, 2018 at 1:24 PM Naik, Ninad
> wrote:
>
> Thanks Maitrayee. I should have mentioned this as one of the things we
> verified. The clocks on cassandra nodes are in sync.
> --
> *From:* maitrayee shah
> *Sent:* Friday, October 12, 2018 6:40:25 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
> We have seen inconsistent read if the clock on the nodes are not in sync.
>
>
> Thank you
>
> Sent from my iPhone
>
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad wrote:
>
> Hello,
>
> We're seeing inconsistent data while doing reads on cassandra. Here are
> the details:
>
> It's is a wide column table. The columns can be added my multiple
> machines, and read by multiple machines. The time between writes and reads
> are in minutes, but sometimes can be in seconds. Writes happen every 2
> minutes.
>
> Now, while reading we're seeing the following cases of inconsistent reads:
>
>- One column was added. If a read was done after the column was added
>(20 secs to 2 minutes after the write), Cassandra returns no data. As if
>the key doesn't exist. If the application retries, it gets the data.
>- A few columns exist for a row key. And a new column 'n' was added.
>Again, a read happens a few minutes after the write. This time, only the
>latest column 'n' is returned. In this case the app doesn't know that the
>data is incomplete so it doesn't retry. If we manually retry, we see all
>the columns.
>- A few columns exist for a row key. And a new column 'n' is added.
>When a read happens after the write, all columns but 'n' are returned.
>
> Here's what we've verified:
>
>- Both writes and reads are using 'LOCAL_QUORUM' consistency level.
>- The replication is within local data center. No remote data center
>is involved in the read or write.
>- During the inconsistent reads, none of the nodes are undergoing GC
>pauses
>- There are no errors in cassandra logs
>- Reads always happen after the writes.
>
> A few other details: Cassandra version: 2.1.9 DataStax java driver
> version: 2.1.10.2 Replication Factor: 3
>
> We don't see this problem in lower environments. We have seen this happen
> once or twice last year, but since last few days it's happening quite
> frequently. On an average 2 inconsistent reads every minute.
>
> Here's how the table definition looks like:
>
> CREATE TABLE "MY_TABLE" (
> key text,
> sub_key text,
> value text,
> PRIMARY KEY ((key), sub_key)
> ) WITH
> bloom_filter_fp_chance=0.01 AND
> caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
> comment='' AND
> dclocal_read_repair_chance=0.10 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.00 AND
> default_time_to_live=0 AND
> speculative_retry='ALWAYS' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
>
> Please point us in the right direction. Thanks !
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received t
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Thanks James. Yeah, we're using the datastax java driver. But we're on version
2.1.10.2. And we are not using the client side timestamps.
From: James Carman
Sent: Saturday, October 13, 2018 12:55:51 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
Are you using the Java driver? The default timestamp generator is client-side
now (since 3.0 I believe), so if you don’t change it, you need your client
clocks to be in sync also.
On Sat, Oct 13, 2018 at 1:24 PM Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Thanks Maitrayee. I should have mentioned this as one of the things we
verified. The clocks on cassandra nodes are in sync.
From: maitrayee shah
Sent: Friday, October 12, 2018 6:40:25 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
We have seen inconsistent read if the clock on the nodes are not in sync.
Thank you
Sent from my iPhone
On Oct 12, 2018, at 1:50 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Hello,
We're seeing inconsistent data while doing reads on cassandra. Here are the
details:
It's is a wide column table. The columns can be added my multiple machines, and
read by multiple machines. The time between writes and reads are in minutes,
but sometimes can be in seconds. Writes happen every 2 minutes.
Now, while reading we're seeing the following cases of inconsistent reads:
* One column was added. If a read was done after the column was added (20
secs to 2 minutes after the write), Cassandra returns no data. As if the key
doesn't exist. If the application retries, it gets the data.
* A few columns exist for a row key. And a new column 'n' was added. Again,
a read happens a few minutes after the write. This time, only the latest column
'n' is returned. In this case the app doesn't know that the data is incomplete
so it doesn't retry. If we manually retry, we see all the columns.
* A few columns exist for a row key. And a new column 'n' is added. When a
read happens after the write, all columns but 'n' are returned.
Here's what we've verified:
* Both writes and reads are using 'LOCAL_QUORUM' consistency level.
* The replication is within local data center. No remote data center is
involved in the read or write.
* During the inconsistent reads, none of the nodes are undergoing GC pauses
* There are no errors in cassandra logs
* Reads always happen after the writes.
A few other details: Cassandra version: 2.1.9 DataStax java driver version:
2.1.10.2 Replication Factor: 3
We don't see this problem in lower environments. We have seen this happen once
or twice last year, but since last few days it's happening quite frequently. On
an average 2 inconsistent reads every minute.
Here's how the table definition looks like:
CREATE TABLE "MY_TABLE" (
key text,
sub_key text,
value text,
PRIMARY KEY ((key), sub_key)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
comment='' AND
dclocal_read_repair_chance=0.10 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.00 AND
default_time_to_live=0 AND
speculative_retry='ALWAYS' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
Please point us in the right direction. Thanks !
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that any review, dissemination, distribution or copying
of this communication is strictly prohibited. If you have received this
communication in error, please notify the sender immediately by replying to
this e-mail and delete the message and any attachments from your computer.
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that any review, dissemination, distribution or copying
of this communication is strictly prohibited. If you have received this
communication in error, please notify the sender immediately by replying to
this e-mail and delete the message and any attachments from your computer.
The information con
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Thanks Elliot. We've verified that the cassandra node clocks are in sync. And
we're not using client side timestamps, so client machine clocks wouldn't
matter ?
From: Elliott Sims
Sent: Friday, October 12, 2018 10:16:58 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
I'll second that - we had some weird inconsistent reads for a long time that we
finally tracked to a small number of clients with significant clock skew. Make
very sure all your client (not just C*) machines have tightly-synced clocks.
On Fri, Oct 12, 2018 at 7:40 PM maitrayee shah
wrote:
We have seen inconsistent read if the clock on the nodes are not in sync.
Thank you
Sent from my iPhone
On Oct 12, 2018, at 1:50 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Hello,
We're seeing inconsistent data while doing reads on cassandra. Here are the
details:
It's is a wide column table. The columns can be added my multiple machines, and
read by multiple machines. The time between writes and reads are in minutes,
but sometimes can be in seconds. Writes happen every 2 minutes.
Now, while reading we're seeing the following cases of inconsistent reads:
* One column was added. If a read was done after the column was added (20
secs to 2 minutes after the write), Cassandra returns no data. As if the key
doesn't exist. If the application retries, it gets the data.
* A few columns exist for a row key. And a new column 'n' was added. Again,
a read happens a few minutes after the write. This time, only the latest column
'n' is returned. In this case the app doesn't know that the data is incomplete
so it doesn't retry. If we manually retry, we see all the columns.
* A few columns exist for a row key. And a new column 'n' is added. When a
read happens after the write, all columns but 'n' are returned.
Here's what we've verified:
* Both writes and reads are using 'LOCAL_QUORUM' consistency level.
* The replication is within local data center. No remote data center is
involved in the read or write.
* During the inconsistent reads, none of the nodes are undergoing GC pauses
* There are no errors in cassandra logs
* Reads always happen after the writes.
A few other details: Cassandra version: 2.1.9 DataStax java driver version:
2.1.10.2 Replication Factor: 3
We don't see this problem in lower environments. We have seen this happen once
or twice last year, but since last few days it's happening quite frequently. On
an average 2 inconsistent reads every minute.
Here's how the table definition looks like:
CREATE TABLE "MY_TABLE" (
key text,
sub_key text,
value text,
PRIMARY KEY ((key), sub_key)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
comment='' AND
dclocal_read_repair_chance=0.10 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.00 AND
default_time_to_live=0 AND
speculative_retry='ALWAYS' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
Please point us in the right direction. Thanks !
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that any review, dissemination, distribution or copying
of this communication is strictly prohibited. If you have received this
communication in error, please notify the sender immediately by replying to
this e-mail and delete the message and any attachments from your computer.
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that any review, dissemination, distribution or copying
of this communication is strictly prohibited. If you have received this
communication in error, please notify the sender immediately by replying to
this e-mail and delete the message and any attachments from your computer.
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Are you using the Java driver? The default timestamp generator is
client-side now (since 3.0 I believe), so if you don’t change it, you need
your client clocks to be in sync also.
On Sat, Oct 13, 2018 at 1:24 PM Naik, Ninad wrote:
> Thanks Maitrayee. I should have mentioned this as one of the things we
> verified. The clocks on cassandra nodes are in sync.
> --
> *From:* maitrayee shah
> *Sent:* Friday, October 12, 2018 6:40:25 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
> We have seen inconsistent read if the clock on the nodes are not in sync.
>
>
> Thank you
>
> Sent from my iPhone
>
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad wrote:
>
> Hello,
>
> We're seeing inconsistent data while doing reads on cassandra. Here are
> the details:
>
> It's is a wide column table. The columns can be added my multiple
> machines, and read by multiple machines. The time between writes and reads
> are in minutes, but sometimes can be in seconds. Writes happen every 2
> minutes.
>
> Now, while reading we're seeing the following cases of inconsistent reads:
>
>- One column was added. If a read was done after the column was added
>(20 secs to 2 minutes after the write), Cassandra returns no data. As if
>the key doesn't exist. If the application retries, it gets the data.
>- A few columns exist for a row key. And a new column 'n' was added.
>Again, a read happens a few minutes after the write. This time, only the
>latest column 'n' is returned. In this case the app doesn't know that the
>data is incomplete so it doesn't retry. If we manually retry, we see all
>the columns.
>- A few columns exist for a row key. And a new column 'n' is added.
>When a read happens after the write, all columns but 'n' are returned.
>
> Here's what we've verified:
>
>- Both writes and reads are using 'LOCAL_QUORUM' consistency level.
>- The replication is within local data center. No remote data center
>is involved in the read or write.
>- During the inconsistent reads, none of the nodes are undergoing GC
>pauses
>- There are no errors in cassandra logs
>- Reads always happen after the writes.
>
> A few other details: Cassandra version: 2.1.9 DataStax java driver
> version: 2.1.10.2 Replication Factor: 3
>
> We don't see this problem in lower environments. We have seen this happen
> once or twice last year, but since last few days it's happening quite
> frequently. On an average 2 inconsistent reads every minute.
>
> Here's how the table definition looks like:
>
> CREATE TABLE "MY_TABLE" (
> key text,
> sub_key text,
> value text,
> PRIMARY KEY ((key), sub_key)
> ) WITH
> bloom_filter_fp_chance=0.01 AND
> caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
> comment='' AND
> dclocal_read_repair_chance=0.10 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.00 AND
> default_time_to_live=0 AND
> speculative_retry='ALWAYS' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
>
> Please point us in the right direction. Thanks !
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
Thanks Maitrayee. I should have mentioned this as one of the things we
verified. The clocks on cassandra nodes are in sync.
From: maitrayee shah
Sent: Friday, October 12, 2018 6:40:25 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
[ This email has been sent from a source external to Epsilon. Please use
caution when clicking links or opening attachments. ]
We have seen inconsistent read if the clock on the nodes are not in sync.
Thank you
Sent from my iPhone
On Oct 12, 2018, at 1:50 PM, Naik, Ninad
mailto:ninad.n...@epsilon.com>> wrote:
Hello,
We're seeing inconsistent data while doing reads on cassandra. Here are the
details:
It's is a wide column table. The columns can be added my multiple machines, and
read by multiple machines. The time between writes and reads are in minutes,
but sometimes can be in seconds. Writes happen every 2 minutes.
Now, while reading we're seeing the following cases of inconsistent reads:
* One column was added. If a read was done after the column was added (20
secs to 2 minutes after the write), Cassandra returns no data. As if the key
doesn't exist. If the application retries, it gets the data.
* A few columns exist for a row key. And a new column 'n' was added. Again,
a read happens a few minutes after the write. This time, only the latest column
'n' is returned. In this case the app doesn't know that the data is incomplete
so it doesn't retry. If we manually retry, we see all the columns.
* A few columns exist for a row key. And a new column 'n' is added. When a
read happens after the write, all columns but 'n' are returned.
Here's what we've verified:
* Both writes and reads are using 'LOCAL_QUORUM' consistency level.
* The replication is within local data center. No remote data center is
involved in the read or write.
* During the inconsistent reads, none of the nodes are undergoing GC pauses
* There are no errors in cassandra logs
* Reads always happen after the writes.
A few other details: Cassandra version: 2.1.9 DataStax java driver version:
2.1.10.2 Replication Factor: 3
We don't see this problem in lower environments. We have seen this happen once
or twice last year, but since last few days it's happening quite frequently. On
an average 2 inconsistent reads every minute.
Here's how the table definition looks like:
CREATE TABLE "MY_TABLE" (
key text,
sub_key text,
value text,
PRIMARY KEY ((key), sub_key)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
comment='' AND
dclocal_read_repair_chance=0.10 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.00 AND
default_time_to_live=0 AND
speculative_retry='ALWAYS' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
Please point us in the right direction. Thanks !
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that any review, dissemination, distribution or copying
of this communication is strictly prohibited. If you have received this
communication in error, please notify the sender immediately by replying to
this e-mail and delete the message and any attachments from your computer.
The information contained in this e-mail message and any attachments may be
privileged and confidential. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended recipient,
you are hereby notified that any review, dissemination, distribution or copying
of this communication is strictly prohibited. If you have received this
communication in error, please notify the sender immediately by replying to
this e-mail and delete the message and any attachments from your computer.
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
I'll second that - we had some weird inconsistent reads for a long time
that we finally tracked to a small number of clients with significant clock
skew. Make very sure all your client (not just C*) machines have
tightly-synced clocks.
On Fri, Oct 12, 2018 at 7:40 PM maitrayee shah
wrote:
> We have seen inconsistent read if the clock on the nodes are not in sync.
>
>
> Thank you
>
> Sent from my iPhone
>
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad wrote:
>
> Hello,
>
> We're seeing inconsistent data while doing reads on cassandra. Here are
> the details:
>
> It's is a wide column table. The columns can be added my multiple
> machines, and read by multiple machines. The time between writes and reads
> are in minutes, but sometimes can be in seconds. Writes happen every 2
> minutes.
>
> Now, while reading we're seeing the following cases of inconsistent reads:
>
>- One column was added. If a read was done after the column was added
>(20 secs to 2 minutes after the write), Cassandra returns no data. As if
>the key doesn't exist. If the application retries, it gets the data.
>- A few columns exist for a row key. And a new column 'n' was added.
>Again, a read happens a few minutes after the write. This time, only the
>latest column 'n' is returned. In this case the app doesn't know that the
>data is incomplete so it doesn't retry. If we manually retry, we see all
>the columns.
>- A few columns exist for a row key. And a new column 'n' is added.
>When a read happens after the write, all columns but 'n' are returned.
>
> Here's what we've verified:
>
>- Both writes and reads are using 'LOCAL_QUORUM' consistency level.
>- The replication is within local data center. No remote data center
>is involved in the read or write.
>- During the inconsistent reads, none of the nodes are undergoing GC
>pauses
>- There are no errors in cassandra logs
>- Reads always happen after the writes.
>
> A few other details: Cassandra version: 2.1.9 DataStax java driver
> version: 2.1.10.2 Replication Factor: 3
>
> We don't see this problem in lower environments. We have seen this happen
> once or twice last year, but since last few days it's happening quite
> frequently. On an average 2 inconsistent reads every minute.
>
> Here's how the table definition looks like:
>
> CREATE TABLE "MY_TABLE" (
> key text,
> sub_key text,
> value text,
> PRIMARY KEY ((key), sub_key)
> ) WITH
> bloom_filter_fp_chance=0.01 AND
> caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
> comment='' AND
> dclocal_read_repair_chance=0.10 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.00 AND
> default_time_to_live=0 AND
> speculative_retry='ALWAYS' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
>
> Please point us in the right direction. Thanks !
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>
>
Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
We have seen inconsistent read if the clock on the nodes are not in sync.
Thank you
Sent from my iPhone
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad wrote:
>
> Hello,
>
> We're seeing inconsistent data while doing reads on cassandra. Here are the
> details:
>
> It's is a wide column table. The columns can be added my multiple machines,
> and read by multiple machines. The time between writes and reads are in
> minutes, but sometimes can be in seconds. Writes happen every 2 minutes.
>
> Now, while reading we're seeing the following cases of inconsistent reads:
>
> One column was added. If a read was done after the column was added (20 secs
> to 2 minutes after the write), Cassandra returns no data. As if the key
> doesn't exist. If the application retries, it gets the data.
> A few columns exist for a row key. And a new column 'n' was added. Again, a
> read happens a few minutes after the write. This time, only the latest column
> 'n' is returned. In this case the app doesn't know that the data is
> incomplete so it doesn't retry. If we manually retry, we see all the columns.
> A few columns exist for a row key. And a new column 'n' is added. When a read
> happens after the write, all columns but 'n' are returned.
> Here's what we've verified:
>
> Both writes and reads are using 'LOCAL_QUORUM' consistency level.
> The replication is within local data center. No remote data center is
> involved in the read or write.
> During the inconsistent reads, none of the nodes are undergoing GC pauses
> There are no errors in cassandra logs
> Reads always happen after the writes.
> A few other details: Cassandra version: 2.1.9 DataStax java driver version:
> 2.1.10.2 Replication Factor: 3
>
> We don't see this problem in lower environments. We have seen this happen
> once or twice last year, but since last few days it's happening quite
> frequently. On an average 2 inconsistent reads every minute.
>
> Here's how the table definition looks like:
>
> CREATE TABLE "MY_TABLE" (
> key text,
> sub_key text,
> value text,
> PRIMARY KEY ((key), sub_key)
> ) WITH
> bloom_filter_fp_chance=0.01 AND
> caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
> comment='' AND
> dclocal_read_repair_chance=0.10 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.00 AND
> default_time_to_live=0 AND
> speculative_retry='ALWAYS' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
> Please point us in the right direction. Thanks !
>
>
>
> The information contained in this e-mail message and any attachments may be
> privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the intended
> recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If you
> have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
