I'll second that - we had some weird inconsistent reads for a long time
that we finally tracked to a small number of clients with significant clock
skew. Make very sure all your client (not just C*) machines have
tightly-synced clocks.
On Fri, Oct 12, 2018 at 7:40 PM maitrayee shah <koolja...@yahoo.com.invalid>
wrote:
> We have seen inconsistent read if the clock on the nodes are not in sync.
>
>
> Thank you
>
> Sent from my iPhone
>
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad <ninad.n...@epsilon.com> wrote:
>
> Hello,
>
> We're seeing inconsistent data while doing reads on cassandra. Here are
> the details:
>
> It's is a wide column table. The columns can be added my multiple
> machines, and read by multiple machines. The time between writes and reads
> are in minutes, but sometimes can be in seconds. Writes happen every 2
> minutes.
>
> Now, while reading we're seeing the following cases of inconsistent reads:
>
> - One column was added. If a read was done after the column was added
> (20 secs to 2 minutes after the write), Cassandra returns no data. As if
> the key doesn't exist. If the application retries, it gets the data.
> - A few columns exist for a row key. And a new column 'n' was added.
> Again, a read happens a few minutes after the write. This time, only the
> latest column 'n' is returned. In this case the app doesn't know that the
> data is incomplete so it doesn't retry. If we manually retry, we see all
> the columns.
> - A few columns exist for a row key. And a new column 'n' is added.
> When a read happens after the write, all columns but 'n' are returned.
>
> Here's what we've verified:
>
> - Both writes and reads are using 'LOCAL_QUORUM' consistency level.
> - The replication is within local data center. No remote data center
> is involved in the read or write.
> - During the inconsistent reads, none of the nodes are undergoing GC
> pauses
> - There are no errors in cassandra logs
> - Reads always happen after the writes.
>
> A few other details: Cassandra version: 2.1.9 DataStax java driver
> version: 2.1.10.2 Replication Factor: 3
>
> We don't see this problem in lower environments. We have seen this happen
> once or twice last year, but since last few days it's happening quite
> frequently. On an average 2 inconsistent reads every minute.
>
> Here's how the table definition looks like:
>
> CREATE TABLE "MY_TABLE" (
> key text,
> sub_key text,
> value text,
> PRIMARY KEY ((key), sub_key)
> ) WITH
> bloom_filter_fp_chance=0.010000 AND
> caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
> comment='' AND
> dclocal_read_repair_chance=0.100000 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.000000 AND
> default_time_to_live=0 AND
> speculative_retry='ALWAYS' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
>
> Please point us in the right direction. Thanks !
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>
>
