Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

[James Carman]

Okay, just thought I’d throw it out there.  I just so happened to miss that
little tidbit from the upgrade guide and got bit by it.

That is an odd problem, then.  I hope you find your answer.  Please update
us if you figure it out.  Very curious to hear the answer.

On Sat, Oct 13, 2018 at 4:09 PM Naik, Ninad  wrote:

> Thanks James. Yeah, we're using the datastax java driver. But we're on
> version 2.1.10.2. And we are not using the client side timestamps.
> --
> *From:* James Carman 
> *Sent:* Saturday, October 13, 2018 12:55:51 PM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
> Are you using the Java driver? The default timestamp generator is
> client-side now (since 3.0 I believe), so if you don’t change it, you need
> your client clocks to be in sync also.
> On Sat, Oct 13, 2018 at 1:24 PM Naik, Ninad 
> wrote:
>
> Thanks Maitrayee. I should have mentioned this as one of the things we
> verified. The clocks on cassandra nodes are in sync.
> --
> *From:* maitrayee shah 
> *Sent:* Friday, October 12, 2018 6:40:25 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
> We have seen inconsistent read if the clock on the nodes are not in sync.
>
>
> Thank you
>
> Sent from my iPhone
>
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad  wrote:
>
> Hello,
>
> We're seeing inconsistent data while doing reads on cassandra. Here are
> the details:
>
> It's is a wide column table. The columns can be added my multiple
> machines, and read by multiple machines. The time between writes and reads
> are in minutes, but sometimes can be in seconds. Writes happen every 2
> minutes.
>
> Now, while reading we're seeing the following cases of inconsistent reads:
>
>- One column was added. If a read was done after the column was added
>(20 secs to 2 minutes after the write), Cassandra returns no data. As if
>the key doesn't exist. If the application retries, it gets the data.
>- A few columns exist for a row key. And a new column 'n' was added.
>Again, a read happens a few minutes after the write. This time, only the
>latest column 'n' is returned. In this case the app doesn't know that the
>data is incomplete so it doesn't retry. If we manually retry, we see all
>the columns.
>- A few columns exist for a row key. And a new column 'n' is added.
>When a read happens after the write, all columns but 'n' are returned.
>
> Here's what we've verified:
>
>- Both writes and reads are using 'LOCAL_QUORUM' consistency level.
>- The replication is within local data center. No remote data center
>is involved in the read or write.
>- During the inconsistent reads, none of the nodes are undergoing GC
>pauses
>- There are no errors in cassandra logs
>- Reads always happen after the writes.
>
> A few other details: Cassandra version: 2.1.9 DataStax java driver
> version: 2.1.10.2 Replication Factor: 3
>
> We don't see this problem in lower environments. We have seen this happen
> once or twice last year, but since last few days it's happening quite
> frequently. On an average 2 inconsistent reads every minute.
>
> Here's how the table definition looks like:
>
> CREATE TABLE "MY_TABLE" (
>   key text,
>   sub_key text,
>   value text,
>   PRIMARY KEY ((key), sub_key)
> ) WITH
>   bloom_filter_fp_chance=0.01 AND
>   caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
>   comment='' AND
>   dclocal_read_repair_chance=0.10 AND
>   gc_grace_seconds=864000 AND
>   read_repair_chance=0.00 AND
>   default_time_to_live=0 AND
>   speculative_retry='ALWAYS' AND
>   memtable_flush_period_in_ms=0 AND
>   compaction={'class': 'SizeTieredCompactionStrategy'} AND
>   compression={'sstable_compression': 'LZ4Compressor'};
>
> Please point us in the right direction. Thanks !
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> 

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

[Naik, Ninad]

Thanks James. Yeah, we're using the datastax java driver. But we're on version 
2.1.10.2. And we are not using the client side timestamps.


From: James Carman 
Sent: Saturday, October 13, 2018 12:55:51 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)


[ This email has been sent from a source external to Epsilon. Please use 
caution when clicking links or opening attachments. ]

Are you using the Java driver? The default timestamp generator is client-side 
now (since 3.0 I believe), so if you don’t change it, you need your client 
clocks to be in sync also.
On Sat, Oct 13, 2018 at 1:24 PM Naik, Ninad 
mailto:ninad.n...@epsilon.com>> wrote:

Thanks Maitrayee. I should have mentioned this as one of the things we 
verified. The clocks on cassandra nodes are in sync.


From: maitrayee shah 
Sent: Friday, October 12, 2018 6:40:25 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)


[ This email has been sent from a source external to Epsilon. Please use 
caution when clicking links or opening attachments. ]

We have seen inconsistent read if the clock on the nodes are not in sync.


Thank you

Sent from my iPhone

On Oct 12, 2018, at 1:50 PM, Naik, Ninad 
mailto:ninad.n...@epsilon.com>> wrote:


Hello,

We're seeing inconsistent data while doing reads on cassandra. Here are the 
details:

It's is a wide column table. The columns can be added my multiple machines, and 
read by multiple machines. The time between writes and reads are in minutes, 
but sometimes can be in seconds. Writes happen every 2 minutes.

Now, while reading we're seeing the following cases of inconsistent reads:

  *   One column was added. If a read was done after the column was added (20 
secs to 2 minutes after the write), Cassandra returns no data. As if the key 
doesn't exist. If the application retries, it gets the data.
  *   A few columns exist for a row key. And a new column 'n' was added. Again, 
a read happens a few minutes after the write. This time, only the latest column 
'n' is returned. In this case the app doesn't know that the data is incomplete 
so it doesn't retry. If we manually retry, we see all the columns.
  *   A few columns exist for a row key. And a new column 'n' is added. When a 
read happens after the write, all columns but 'n' are returned.

Here's what we've verified:

  *   Both writes and reads are using 'LOCAL_QUORUM' consistency level.
  *   The replication is within local data center. No remote data center is 
involved in the read or write.
  *   During the inconsistent reads, none of the nodes are undergoing GC pauses
  *   There are no errors in cassandra logs
  *   Reads always happen after the writes.

A few other details: Cassandra version: 2.1.9 DataStax java driver version: 
2.1.10.2 Replication Factor: 3

We don't see this problem in lower environments. We have seen this happen once 
or twice last year, but since last few days it's happening quite frequently. On 
an average 2 inconsistent reads every minute.

Here's how the table definition looks like:

CREATE TABLE "MY_TABLE" (
  key text,
  sub_key text,
  value text,
  PRIMARY KEY ((key), sub_key)
) WITH
  bloom_filter_fp_chance=0.01 AND
  caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
  comment='' AND
  dclocal_read_repair_chance=0.10 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.00 AND
  default_time_to_live=0 AND
  speculative_retry='ALWAYS' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};


Please point us in the right direction. Thanks !



The information contained in this e-mail message and any attachments may be 
privileged and confidential. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that any review, dissemination, distribution or copying 
of this communication is strictly prohibited. If you have received this 
communication in error, please notify the sender immediately by replying to 
this e-mail and delete the message and any attachments from your computer.



The information contained in this e-mail message and any attachments may be 
privileged and confidential. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that any review, dissemination, distribution or copying 
of this communication is strictly prohibited. If you have received this 
communication in error, please notify the sender immediately by replying to 
this e-mail and delete the message and any attachments from your computer.

The information contained in this e-mail message and any attachments may be 
privileged and 

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

[Naik, Ninad]

Thanks Elliot. We've verified that the cassandra node clocks are in sync. And 
we're not using client side timestamps, so client machine clocks wouldn't 
matter ?


From: Elliott Sims 
Sent: Friday, October 12, 2018 10:16:58 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)


[ This email has been sent from a source external to Epsilon. Please use 
caution when clicking links or opening attachments. ]

I'll second that - we had some weird inconsistent reads for a long time that we 
finally tracked to a small number of clients with significant clock skew.  Make 
very sure all your client (not just C*) machines have tightly-synced clocks.

On Fri, Oct 12, 2018 at 7:40 PM maitrayee shah  
wrote:
We have seen inconsistent read if the clock on the nodes are not in sync.


Thank you

Sent from my iPhone

On Oct 12, 2018, at 1:50 PM, Naik, Ninad 
mailto:ninad.n...@epsilon.com>> wrote:


Hello,

We're seeing inconsistent data while doing reads on cassandra. Here are the 
details:

It's is a wide column table. The columns can be added my multiple machines, and 
read by multiple machines. The time between writes and reads are in minutes, 
but sometimes can be in seconds. Writes happen every 2 minutes.

Now, while reading we're seeing the following cases of inconsistent reads:

  *   One column was added. If a read was done after the column was added (20 
secs to 2 minutes after the write), Cassandra returns no data. As if the key 
doesn't exist. If the application retries, it gets the data.
  *   A few columns exist for a row key. And a new column 'n' was added. Again, 
a read happens a few minutes after the write. This time, only the latest column 
'n' is returned. In this case the app doesn't know that the data is incomplete 
so it doesn't retry. If we manually retry, we see all the columns.
  *   A few columns exist for a row key. And a new column 'n' is added. When a 
read happens after the write, all columns but 'n' are returned.

Here's what we've verified:

  *   Both writes and reads are using 'LOCAL_QUORUM' consistency level.
  *   The replication is within local data center. No remote data center is 
involved in the read or write.
  *   During the inconsistent reads, none of the nodes are undergoing GC pauses
  *   There are no errors in cassandra logs
  *   Reads always happen after the writes.

A few other details: Cassandra version: 2.1.9 DataStax java driver version: 
2.1.10.2 Replication Factor: 3

We don't see this problem in lower environments. We have seen this happen once 
or twice last year, but since last few days it's happening quite frequently. On 
an average 2 inconsistent reads every minute.

Here's how the table definition looks like:

CREATE TABLE "MY_TABLE" (
  key text,
  sub_key text,
  value text,
  PRIMARY KEY ((key), sub_key)
) WITH
  bloom_filter_fp_chance=0.01 AND
  caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
  comment='' AND
  dclocal_read_repair_chance=0.10 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.00 AND
  default_time_to_live=0 AND
  speculative_retry='ALWAYS' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};


Please point us in the right direction. Thanks !



The information contained in this e-mail message and any attachments may be 
privileged and confidential. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that any review, dissemination, distribution or copying 
of this communication is strictly prohibited. If you have received this 
communication in error, please notify the sender immediately by replying to 
this e-mail and delete the message and any attachments from your computer.

The information contained in this e-mail message and any attachments may be 
privileged and confidential. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that any review, dissemination, distribution or copying 
of this communication is strictly prohibited. If you have received this 
communication in error, please notify the sender immediately by replying to 
this e-mail and delete the message and any attachments from your computer.

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

[James Carman]

Are you using the Java driver? The default timestamp generator is
client-side now (since 3.0 I believe), so if you don’t change it, you need
your client clocks to be in sync also.
On Sat, Oct 13, 2018 at 1:24 PM Naik, Ninad  wrote:

> Thanks Maitrayee. I should have mentioned this as one of the things we
> verified. The clocks on cassandra nodes are in sync.
> --
> *From:* maitrayee shah 
> *Sent:* Friday, October 12, 2018 6:40:25 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)
>
>
> [ This email has been sent from a source external to Epsilon. Please use
> caution when clicking links or opening attachments. ]
> We have seen inconsistent read if the clock on the nodes are not in sync.
>
>
> Thank you
>
> Sent from my iPhone
>
> On Oct 12, 2018, at 1:50 PM, Naik, Ninad  wrote:
>
> Hello,
>
> We're seeing inconsistent data while doing reads on cassandra. Here are
> the details:
>
> It's is a wide column table. The columns can be added my multiple
> machines, and read by multiple machines. The time between writes and reads
> are in minutes, but sometimes can be in seconds. Writes happen every 2
> minutes.
>
> Now, while reading we're seeing the following cases of inconsistent reads:
>
>- One column was added. If a read was done after the column was added
>(20 secs to 2 minutes after the write), Cassandra returns no data. As if
>the key doesn't exist. If the application retries, it gets the data.
>- A few columns exist for a row key. And a new column 'n' was added.
>Again, a read happens a few minutes after the write. This time, only the
>latest column 'n' is returned. In this case the app doesn't know that the
>data is incomplete so it doesn't retry. If we manually retry, we see all
>the columns.
>- A few columns exist for a row key. And a new column 'n' is added.
>When a read happens after the write, all columns but 'n' are returned.
>
> Here's what we've verified:
>
>- Both writes and reads are using 'LOCAL_QUORUM' consistency level.
>- The replication is within local data center. No remote data center
>is involved in the read or write.
>- During the inconsistent reads, none of the nodes are undergoing GC
>pauses
>- There are no errors in cassandra logs
>- Reads always happen after the writes.
>
> A few other details: Cassandra version: 2.1.9 DataStax java driver
> version: 2.1.10.2 Replication Factor: 3
>
> We don't see this problem in lower environments. We have seen this happen
> once or twice last year, but since last few days it's happening quite
> frequently. On an average 2 inconsistent reads every minute.
>
> Here's how the table definition looks like:
>
> CREATE TABLE "MY_TABLE" (
>   key text,
>   sub_key text,
>   value text,
>   PRIMARY KEY ((key), sub_key)
> ) WITH
>   bloom_filter_fp_chance=0.01 AND
>   caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
>   comment='' AND
>   dclocal_read_repair_chance=0.10 AND
>   gc_grace_seconds=864000 AND
>   read_repair_chance=0.00 AND
>   default_time_to_live=0 AND
>   speculative_retry='ALWAYS' AND
>   memtable_flush_period_in_ms=0 AND
>   compaction={'class': 'SizeTieredCompactionStrategy'} AND
>   compression={'sstable_compression': 'LZ4Compressor'};
>
> Please point us in the right direction. Thanks !
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>
>
>
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
>

Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)

[Naik, Ninad]

Thanks Maitrayee. I should have mentioned this as one of the things we 
verified. The clocks on cassandra nodes are in sync.


From: maitrayee shah 
Sent: Friday, October 12, 2018 6:40:25 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra: Inconsistent data on reads (LOCAL_QUORUM)


[ This email has been sent from a source external to Epsilon. Please use 
caution when clicking links or opening attachments. ]

We have seen inconsistent read if the clock on the nodes are not in sync.


Thank you

Sent from my iPhone

On Oct 12, 2018, at 1:50 PM, Naik, Ninad 
mailto:ninad.n...@epsilon.com>> wrote:


Hello,

We're seeing inconsistent data while doing reads on cassandra. Here are the 
details:

It's is a wide column table. The columns can be added my multiple machines, and 
read by multiple machines. The time between writes and reads are in minutes, 
but sometimes can be in seconds. Writes happen every 2 minutes.

Now, while reading we're seeing the following cases of inconsistent reads:

  *   One column was added. If a read was done after the column was added (20 
secs to 2 minutes after the write), Cassandra returns no data. As if the key 
doesn't exist. If the application retries, it gets the data.
  *   A few columns exist for a row key. And a new column 'n' was added. Again, 
a read happens a few minutes after the write. This time, only the latest column 
'n' is returned. In this case the app doesn't know that the data is incomplete 
so it doesn't retry. If we manually retry, we see all the columns.
  *   A few columns exist for a row key. And a new column 'n' is added. When a 
read happens after the write, all columns but 'n' are returned.

Here's what we've verified:

  *   Both writes and reads are using 'LOCAL_QUORUM' consistency level.
  *   The replication is within local data center. No remote data center is 
involved in the read or write.
  *   During the inconsistent reads, none of the nodes are undergoing GC pauses
  *   There are no errors in cassandra logs
  *   Reads always happen after the writes.

A few other details: Cassandra version: 2.1.9 DataStax java driver version: 
2.1.10.2 Replication Factor: 3

We don't see this problem in lower environments. We have seen this happen once 
or twice last year, but since last few days it's happening quite frequently. On 
an average 2 inconsistent reads every minute.

Here's how the table definition looks like:

CREATE TABLE "MY_TABLE" (
  key text,
  sub_key text,
  value text,
  PRIMARY KEY ((key), sub_key)
) WITH
  bloom_filter_fp_chance=0.01 AND
  caching='{"keys":"ALL", "rows_per_partition":"NONE"}' AND
  comment='' AND
  dclocal_read_repair_chance=0.10 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.00 AND
  default_time_to_live=0 AND
  speculative_retry='ALWAYS' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};


Please point us in the right direction. Thanks !



The information contained in this e-mail message and any attachments may be 
privileged and confidential. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that any review, dissemination, distribution or copying 
of this communication is strictly prohibited. If you have received this 
communication in error, please notify the sender immediately by replying to 
this e-mail and delete the message and any attachments from your computer.

The information contained in this e-mail message and any attachments may be 
privileged and confidential. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that any review, dissemination, distribution or copying 
of this communication is strictly prohibited. If you have received this 
communication in error, please notify the sender immediately by replying to 
this e-mail and delete the message and any attachments from your computer.

Re: Tracing in cassandra

[Nitan Kainth]

Using partition key is best way to query 

Sent from my iPhone

> On Oct 12, 2018, at 6:49 PM, Abdul Patel  wrote:
> 
> Yes with range queries its timing out, one question was the where condition 
> is primary key rather than clustering key.
> 
>> On Friday, October 12, 2018, Nitan Kainth  wrote:
>> Did it still timeout?
>> 
>> Sent from my iPhone
>> 
>>> On Oct 12, 2018, at 1:11 PM, Abdul Patel  wrote:
>>> 
>>> With limit 11 this is query..
>>> Select * from table where status=0 and tojen(user_id)  >=token(126838) and 
>>> token(user_id)  <= token>> 
 On Friday, October 12, 2018, Abdul Patel  wrote:
 Let me try with limit 11 ..we have 18 node cluster ..no nodes down..
 
> On Friday, October 12, 2018, Nitan Kainth  wrote:
> Try query with partition key selection in where clause. But time for 
> limit 11 shouldn’t fail. Are all nodes up? Do you see any corruption in 
> ay sstable?
> 
> Sent from my iPhone
> 
>> On Oct 12, 2018, at 11:40 AM, Abdul Patel  wrote:
>> 
>> Sean,
>> 
>> here it is :
>> CREATE TABLE Keyspave.tblname (
>> user_id bigint,
>> session_id text,
>> application_guid text,
>> last_access_time timestamp,
>> login_time timestamp,
>> status int,
>> terminated_by text,
>> update_time timestamp,
>> PRIMARY KEY (user_id, session_id)
>> ) WITH CLUSTERING ORDER BY (session_id ASC)
>> 
>> also they see timeouts with limit 11 as well, so is it better to remove 
>> with limit option ? or whats best to query such schema?
>> 
>>> On Fri, Oct 12, 2018 at 11:05 AM Durity, Sean R 
>>>  wrote:
>>> Cross-partition = multiple partitions
>>> 
>>>  
>>> 
>>> Simple example:
>>> 
>>> Create table customer (
>>> 
>>> Customerid int,
>>> 
>>> Name text,
>>> 
>>> Lastvisit date,
>>> 
>>> Phone text,
>>> 
>>> Primary key (customerid) );
>>> 
>>>  
>>> 
>>> Query
>>> 
>>> Select customerid from customer limit 5000;
>>> 
>>>  
>>> 
>>> The query is asking for 5000 different partitions to be selected across 
>>> the cluster. This is a very EXPENSIVE query for Cassandra, especially 
>>> as the number of nodes goes up. Typically, you want to query a single 
>>> partition. Read timeouts are usually caused by queries that are 
>>> selecting many partitions or a very large partition. That is why a 
>>> schema for the involved table could help.
>>> 
>>>  
>>> 
>>>  
>>> 
>>> Sean Durity
>>> 
>>>  
>>> 
>>> From: Abdul Patel  
>>> Sent: Friday, October 12, 2018 10:04 AM
>>> To: user@cassandra.apache.org
>>> Subject: [EXTERNAL] Re: Tracing in cassandra
>>> 
>>>  
>>> 
>>> Cpuld you elaborate cross partition query?
>>> 
>>> On Friday, October 12, 2018, Durity, Sean R 
>>>  wrote:
>>> 
>>> I suspect you are doing a cross-partition query, which will not scale 
>>> well (as you can see). What is the schema for the table involved?
>>> 
>>>  
>>> 
>>>  
>>> 
>>> Sean Durity
>>> 
>>>  
>>> 
>>> From: Abdul Patel  
>>> Sent: Thursday, October 11, 2018 5:54 PM
>>> To: a...@instaclustr.com
>>> Cc: user@cassandra.apache.org
>>> Subject: [EXTERNAL] Re: Tracing in cassandra
>>> 
>>>  
>>> 
>>> Query :
>>> 
>>> SELECT * FROM keysoace.tablenameWHERE user_id = 390797583 LIMIT 5000; 
>>> 
>>> -Error: ReadTimeout: Error from server: code=1200 [Coordinator node 
>>> timed out waiting for replica nodes' responses] message="Operation 
>>> timed out - received only 0 responses." info={'received_responses': 0, 
>>> 'required_responses': 1, 'consistency': 'ONE'}
>>> 
>>>  
>>> 
>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | 
>>> e70bd7c0-cd9e-11e8-8e99-15807bff4dfd |  
>>>   Parsing SELECT * FROM 
>>> keysoace.tablenameWHERE user_id = 390797583 LIMIT 5000; | 10.54.145.32 
>>> |   4020 |   Native-Transport-Requests-3
>>> 
>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | 
>>> e70bfed0-cd9e-11e8-8e99-15807bff4dfd |  
>>> 
>>>  Preparing statement | 10.54.145.32 |   
>>> 5065 |   Native-Transport-Requests-3
>>> 
>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | 
>>> e70c25e0-cd9e-11e8-8e99-15807bff4dfd |  
>>>
>>> Executing single-partition query on roles | 10.54.145.32 |   
>>> 6171 |   ReadStage-2
>>> 
>>> 

Re: Advantage over Cassandra in Kubernetes

[Goutham reddy]

Thanks Ben for the detailed insight. Btw we are planning to set up
Kubernetes Cassandra cluster in Development. We wanted to know what are the
possible problems we face if we go by Kubernetes

On Thu, Oct 11, 2018 at 4:06 PM Ben Bromhead  wrote:

> This is a fairly high-level question which could end up going quite deep,
> but below is a quick summary off the top of my head.
>
> You can get a few advantages when running Cassandra in Kubernetes,
> particularly:
>
>- Easy discovery and network connectivity with other services running
>on K8s
>- Reproducible, repeatable operations and deployments
>- A cloud-independent approach to container orchestration, that is
>supported by all major cloud providers.
>- Easy backups, deployments, scaling etc via statefulsets or an
>operator (see https://github.com/instaclustr/cassandra-operator).
>
> There are also some rough edges with running Cassandra on Kubernetes:
>
>- Failure domain placement with statefulsets is still challenging
>(v1.12 goes a long way to fixing this -
>
> https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/
>)
>- Getting resource constraints correct and working out scheduling in
>constrained environments can be maddening.
>- Only a few, small deployments (that I'm aware of) are running
>Cassandra in Kubernetes in production. So you will be breaking new ground
>and encounter problems that haven't been solved before.
>- The Cassandra examples in the official Kubernetes documentation is
>not something you want to take into production.
>
> Cheers
>
> Ben
>
> On Thu, Oct 11, 2018 at 6:50 PM Goutham reddy 
> wrote:
>
>> Hi,
>> We are in the process of setting up Cassandra cluster with high
>> availability. So the debate is installing Cassandra in Kubernetes cluster.
>> Can someone  throw some light, what advantages can I get when created
>> Cassandra cluster inside Kubernetes cluster. Any comments are highly
>> appreciated:)
>>
>> Thanks and Regards,
>> Goutham Reddy Aenugu.
>> --
>> Regards
>> Goutham Reddy
>>
> --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692
> Reliability at Scale
> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>
-- 
Regards
Goutham Reddy