sstabledump and liveness_info question

[Xiangfei Ni]

Hi Expert,
  I have questions about sstabledump
  1,create table and insert data:
create table test2(a text,b text,primary key (a));
insert into test2(a,b) values ('a','b');
  2,flush data to disk
  3,sstabledump the sstable
[cassandra@concar-cassandra01] 
/opt/database/apache-cassandra-data/stresscql/test2-a9d1a1306adb11e8a14077a1580c0a40
 $ /opt/database/apache-cassandra/tools/bin/sstabledump ./mc-1-big-Data.db
[
  {
"partition" : {
  "key" : [ "a" ],
  "position" : 0
},
"rows" : [
  {
"type" : "row",
"position" : 15,
"liveness_info" : { "tstamp" : "2018-06-08T05:20:39.932553Z" },
"cells" : [
  { "name" : "b", "value" : "b" }
]
  }
]
  }
  4,update data:
update test2 set b='2' where a='a';
  5,flush data to disk
  6,sstabledump the sstable
[cassandra@concar-cassandra01] 
/opt/database/apache-cassandra-data/stresscql/test2-a9d1a1306adb11e8a14077a1580c0a40
 $ /opt/database/apache-cassandra/tools/bin/sstabledump ./mc-2-big-Data.db
[
  {
"partition" : {
  "key" : [ "a" ],
  "position" : 0
},
"rows" : [
  {
"type" : "row",
"position" : 15,
"cells" : [
  { "name" : "b", "value" : "2", "tstamp" : 
"2018-06-08T05:22:17.959558Z" }
]
  }
]
  }
]

My questions are:
1,why there is no liveness_info element in the second sstable dump?
2,why the output format is not the same?The second dump has the "tstamp" : 
"2018-06-08T05:22:17.959558Z" in the cell element but there is no in the first 
output?
3,is there detail instruction about sstableoutput?

Re: Cassandra 3.0.X migarte to VPC

[kurt greaves]

>
> I meant migrating to gosspsnitch during adding new dc. New dc will be
> empty so all the data will be streamed based on snitch property chosen


Should work fine on the new DC, as long as the original DC is using a
snitch that supports datacenters - then just don't mix and match snitches
within a DC.​

Re: Schema version mismatch

[Nandakishore Tokala]

FYI

this is the info I got after research on schema version

what is the schema version in Cassandra and for what are all the changes
schema version is changed?

schema version is a UUID used to set a baseline version of the schema. It
tracks if a schema migration is required. Schema version changes when there
is any schema changes made to the cluster like adding or dropping
tables/column/keyspaces i.e CREATE, DROP, ALTER etc.

why do we see schema version mismatch other than upgrades time?

It can be seen when schema changes are made i.e DDL statements are issues
concurrently.
If a node is down and schema changes are being made.
NTP is not in sync
If the nodes in the cluster are running with mixed  Apache versions.
Cassandra schema updates assume that schema changes are done one-at-a-time.
If you make multiple changes at the same time, you can cause some nodes to
end up with a different schema, than others.

Can we solve the schema version mismatch without restart the Cassandra nodes

When you only have one node in disagreement, the easiest way is simply to
run nodetool resetlocalschema, which will make that node forget it's schema
and request it again from the other nodes.

When you have more than one node in disagreement, it becomes more
difficult, because you can't control which node will respond to the schema
request, so one of the nodes with the wrong schema could send it back to
the node where you ran resetlocalschema, and then it will still be in
disagreement with the majority.

In this case, you need to shut down all the nodes that have the incorrect
schema, then start them up one node at a time and after each node comes up,
run nodetool resetlocalschema. Check that the schema is now in agreement
for all the nodes that are currently running, and then repeat the process
for each remaining node that has the incorrect schema.

On Thu, Jun 7, 2018 at 4:54 PM, Nandakishore Tokala <
nandakishore.tok...@gmail.com> wrote:

> Thanks Justin
>
> On Thu, Jun 7, 2018 at 4:27 PM, Justin Cameron 
> wrote:
>
>> You may run into schema mismatch issues if you make multiple schema
>> alterations in a very short period of time (e.g. if you're programmatically
>> modifying tables 50x a second). You'll be better off making schema changes
>> in advance. If you need to make dynamic changes then you could check the
>> version to make sure that the previous change has applied before applying
>> the next one.
>>
>> AFAIK the only way to resolve schema mismatch is a rolling restart
>>
>> On Fri, 8 Jun 2018 at 03:03 Nandakishore Tokala <
>> nandakishore.tok...@gmail.com> wrote:
>>
>>> what is the schema version in Cassandra and for what are all the changes
>>> schema version is changed?
>>>
>>> why do we see schema version mismatch other than upgrades time?
>>> Can we solve the schema version mismatch without restart the
>>> Cassandra nodes
>>>
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Nanda Kishore
>>>
>> --
>>
>>
>> *Justin Cameron*Senior Software Engineer
>>
>>
>> 
>>
>>
>> This email has been sent on behalf of Instaclustr Pty. Limited
>> (Australia) and Instaclustr Inc (USA).
>>
>> This email and any attachments may contain confidential and legally
>> privileged information.  If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>>
>
>
>
> --
> Thanks & Regards,
> Nanda Kishore
>



-- 
Thanks & Regards,
Nanda Kishore

Re: Schema version mismatch

[Nandakishore Tokala]

Thanks Justin

On Thu, Jun 7, 2018 at 4:27 PM, Justin Cameron 
wrote:

> You may run into schema mismatch issues if you make multiple schema
> alterations in a very short period of time (e.g. if you're programmatically
> modifying tables 50x a second). You'll be better off making schema changes
> in advance. If you need to make dynamic changes then you could check the
> version to make sure that the previous change has applied before applying
> the next one.
>
> AFAIK the only way to resolve schema mismatch is a rolling restart
>
> On Fri, 8 Jun 2018 at 03:03 Nandakishore Tokala <
> nandakishore.tok...@gmail.com> wrote:
>
>> what is the schema version in Cassandra and for what are all the changes
>> schema version is changed?
>>
>> why do we see schema version mismatch other than upgrades time?
>> Can we solve the schema version mismatch without restart the
>> Cassandra nodes
>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Nanda Kishore
>>
> --
>
>
> *Justin Cameron*Senior Software Engineer
>
>
> 
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>



-- 
Thanks & Regards,
Nanda Kishore

Re: Schema version mismatch

[Justin Cameron]

You may run into schema mismatch issues if you make multiple schema
alterations in a very short period of time (e.g. if you're programmatically
modifying tables 50x a second). You'll be better off making schema changes
in advance. If you need to make dynamic changes then you could check the
version to make sure that the previous change has applied before applying
the next one.

AFAIK the only way to resolve schema mismatch is a rolling restart

On Fri, 8 Jun 2018 at 03:03 Nandakishore Tokala <
nandakishore.tok...@gmail.com> wrote:

> what is the schema version in Cassandra and for what are all the changes
> schema version is changed?
>
> why do we see schema version mismatch other than upgrades time?
> Can we solve the schema version mismatch without restart the
> Cassandra nodes
>
>
>
>
> --
> Thanks & Regards,
> Nanda Kishore
>
-- 


*Justin Cameron*Senior Software Engineer





This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Schema version mismatch

[Nandakishore Tokala]

what is the schema version in Cassandra and for what are all the changes
schema version is changed?

why do we see schema version mismatch other than upgrades time?
Can we solve the schema version mismatch without restart the Cassandra nodes



-- 
Thanks & Regards,
Nanda Kishore

Re: Cassandra 3.0.X migarte to VPC

[Nitan Kainth]

I meant migrating to gosspsnitch during adding new dc. New dc will be empty so 
all the data will be streamed based on snitch property chosen

Sent from my iPhone

> On Jun 7, 2018, at 12:23 PM, Jeff Jirsa  wrote:
> 
> I don't know about "new snitch protocol" (there is no snitch protocol), but 
> in the case of ec2snitch and gossipingpropertyfilesnitch (do not extrapolate 
> this to any other combination), you can switch them 1:1 before the move, and 
> everything should be fine, as long as care is taken to make the property file 
> match the aws region/az discovered.
> 
> If there were other snitches involved (especially SimpleSnitch), things get 
> much more complicated. 
> 
> 
> 
> 
>> On Thu, Jun 7, 2018 at 9:18 AM, Nitan Kainth  wrote:
>> Jeff, 
>> 
>> In this case, if Riccardo is adding new DC, then he can pickup new snitch 
>> protocol right?
>> 
>>> On Thu, Jun 7, 2018 at 12:15 PM, Jeff Jirsa  wrote:
>>> 
>>> 
>>> 
 On Thu, Jun 7, 2018 at 9:12 AM, Nitan Kainth  wrote:
 Riccardo,
 
 Simplest method can be is to add VPC as an additional datacenter to 
 existing cluster; once New DC has synced up all data, just switch over 
 your application. Only caveat is that there should be enough network 
 bandwidth between EC2 and VPC.
 
 Horizontal scaling will help to some extent. 
 
 Changing snitch will cause data movement but for new DC, I believe you can 
 chose. Leaving others comment.
>>> 
>>> If you change the snitch, and the data ownership changes, you run the risk 
>>> of violating at least consistency guarantees, if not outright losing data.
>>> 
>>> Changing the snitch is only something you should attempt if you understand 
>>> the implications. Cassandra does NOT re-stream data to the right places 
>>> if/when you change a snitch. 
>>> 
>>>  
 
 With EBS, use right instance where IOPs are as high as possible. 
 Generally, SSDs IOPs are 75K; what is best for you depends on workload. I 
 have seen 20K IOPs supporting production loads in reasonable cluster.
>>> 
>>> This is very much workload dependent. I know of people running thousands of 
>>> nodes / petabytes of EBS on 4T GP2 volumes which provide 10k iops, and they 
>>> RARELY go above 1k iops during normal use patterns.
>>> 
>>> -  Jeff
>>> 
>>>  
 
> On Thu, Jun 7, 2018 at 6:20 AM, Riccardo Ferrari  
> wrote:
> Dear list,
> 
> We have a small cluster on AWS EC2-Classic and we are planning to move it 
> to a VPC. 
> 
> I know this has been discussed few times already including here:
> https://mail-archives.apache.org/mod_mbox/cassandra-user/201406.mbox/%3cca+vsrlopop7th8nx20aoz3as75g2jrjm3ryx119deklynhq...@mail.gmail.com%3E
>  - Thanks Alain RODRIGUEZ!
> or here:
> http://grokbase.com/t/cassandra/user/1498y74hn7/moving-cassandra-from-ec2-classic-into-vpc
>  - Thanks Ben Bromhead!
> 
> However I can't find anyone suggesting ClassicLink to perform such 
> migration, maybe because was recent addition compared to those posts. So 
> current status is:
> 
> 3 nodes : m1.xlarge (ephemeral storage)
> Cassandra 3.0.8
> Few keyspaces with {'class': 'SimpleStrategy', 'replication_factor': '3'}
> endpoint_snitch: Ec2Snitch
> connectivity on private ips (local_address, rpc_address, no 
> broadcast_address)
> average load pretty high (we need to scale up)
> vnodes
> My idea is to migrate:
> Add intances on to the existing cluster (1 by 1):
> Same cassandra version
> Same cluster name
> Seed list from ec2 classic list
> Run repair and cleanup
> Update seed list (cassandar config and services)
> Decommision old instances (1 by 1)
> Run repair and cleanup again
> 
> Question time:
> What is the best approach, in 2018, for migrating a cluster from 
> EC2Classic to VPC with 0 downtime?
> Should we scale up before or during the migration? I know I should expect 
> some load from data streaming, at the same time we're adding capacity.
> We would like to migrato to GossipingPropertyFileSnitch, can we mix them?
> Any gothca?
> We are thinking to move to EBS (due to cluster size and snapshotting 
> caoabilities). Any hardware reccomendation?
> Any suggestion is much appreciated!
> 
> 
> 
 
>>> 
>> 
> 

Re: Cassandra 3.0.X migarte to VPC

[Jeff Jirsa]

I don't know about "new snitch protocol" (there is no snitch protocol), but
in the case of ec2snitch and gossipingpropertyfilesnitch (do not
extrapolate this to any other combination), you can switch them 1:1 before
the move, and everything should be fine, as long as care is taken to make
the property file match the aws region/az discovered.

If there were other snitches involved (especially SimpleSnitch), things get
much more complicated.




On Thu, Jun 7, 2018 at 9:18 AM, Nitan Kainth  wrote:

> Jeff,
>
> In this case, if Riccardo is adding new DC, then he can pickup new snitch
> protocol right?
>
> On Thu, Jun 7, 2018 at 12:15 PM, Jeff Jirsa  wrote:
>
>>
>>
>>
>> On Thu, Jun 7, 2018 at 9:12 AM, Nitan Kainth 
>> wrote:
>>
>>> Riccardo,
>>>
>>> Simplest method can be is to add VPC as an additional datacenter to
>>> existing cluster; once New DC has synced up all data, just switch over your
>>> application. Only caveat is that there should be enough network bandwidth
>>> between EC2 and VPC.
>>>
>>> Horizontal scaling will help to some extent.
>>>
>>> Changing snitch will cause data movement but for new DC, I believe you
>>> can chose. Leaving others comment.
>>>
>>
>> If you change the snitch, and the data ownership changes, you run the
>> risk of violating at least consistency guarantees, if not outright losing
>> data.
>>
>> Changing the snitch is only something you should attempt if you
>> understand the implications. Cassandra does NOT re-stream data to the right
>> places if/when you change a snitch.
>>
>>
>>
>>>
>>> With EBS, use right instance where IOPs are as high as possible.
>>> Generally, SSDs IOPs are 75K; what is best for you depends on workload. I
>>> have seen 20K IOPs supporting production loads in reasonable cluster.
>>>
>>
>> This is very much workload dependent. I know of people running thousands
>> of nodes / petabytes of EBS on 4T GP2 volumes which provide 10k iops, and
>> they RARELY go above 1k iops during normal use patterns.
>>
>> -  Jeff
>>
>>
>>
>>>
>>> On Thu, Jun 7, 2018 at 6:20 AM, Riccardo Ferrari 
>>> wrote:
>>>
 Dear list,

 We have a small cluster on AWS EC2-Classic and we are planning to move
 it to a VPC.

 I know this has been discussed few times already including here:
 https://mail-archives.apache.org/mod_mbox/cassandra-user/201
 406.mbox/%3CCA+VSrLopop7Th8nX20aOZ3As75g2jrJm3ryX119dekLYNHq
 f...@mail.gmail.com%3E - Thanks Alain RODRIGUEZ!
 or here:
 http://grokbase.com/t/cassandra/user/1498y74hn7/moving-cassa
 ndra-from-ec2-classic-into-vpc - Thanks Ben Bromhead!

 However I can't find anyone suggesting ClassicLink to perform such
 migration, maybe because was recent addition compared to those posts. So
 current status is:


- 3 nodes : m1.xlarge (ephemeral storage)
- Cassandra 3.0.8
- Few keyspaces with {'class': 'SimpleStrategy',
'replication_factor': '3'}
- endpoint_snitch: Ec2Snitch
- connectivity on private ips (local_address, rpc_address, no
broadcast_address)
- average load pretty high (we need to scale up)
- vnodes

 My idea is to migrate:

- Add intances on to the existing cluster (1 by 1):
   - Same cassandra version
   - Same cluster name
   - Seed list from ec2 classic list
- Run repair and cleanup
- Update seed list (cassandar config and services)
- Decommision old instances (1 by 1)
- Run repair and cleanup again


 Question time:

- What is the best approach, in 2018, for migrating a cluster from
EC2Classic to VPC with 0 downtime?
- Should we scale up before or during the migration? I know I
should expect some load from data streaming, at the same time we're 
 adding
capacity.
- We would like to migrato to GossipingPropertyFileSnitch, can we
mix them?
- Any gothca?
- We are thinking to move to EBS (due to cluster size and
snapshotting caoabilities). Any hardware reccomendation?

 Any suggestion is much appreciated!




>>>
>>
>

Re: Cassandra 3.0.X migarte to VPC

[Nitan Kainth]

Jeff,

In this case, if Riccardo is adding new DC, then he can pickup new snitch
protocol right?

On Thu, Jun 7, 2018 at 12:15 PM, Jeff Jirsa  wrote:

>
>
>
> On Thu, Jun 7, 2018 at 9:12 AM, Nitan Kainth 
> wrote:
>
>> Riccardo,
>>
>> Simplest method can be is to add VPC as an additional datacenter to
>> existing cluster; once New DC has synced up all data, just switch over your
>> application. Only caveat is that there should be enough network bandwidth
>> between EC2 and VPC.
>>
>> Horizontal scaling will help to some extent.
>>
>> Changing snitch will cause data movement but for new DC, I believe you
>> can chose. Leaving others comment.
>>
>
> If you change the snitch, and the data ownership changes, you run the risk
> of violating at least consistency guarantees, if not outright losing data.
>
> Changing the snitch is only something you should attempt if you understand
> the implications. Cassandra does NOT re-stream data to the right places
> if/when you change a snitch.
>
>
>
>>
>> With EBS, use right instance where IOPs are as high as possible.
>> Generally, SSDs IOPs are 75K; what is best for you depends on workload. I
>> have seen 20K IOPs supporting production loads in reasonable cluster.
>>
>
> This is very much workload dependent. I know of people running thousands
> of nodes / petabytes of EBS on 4T GP2 volumes which provide 10k iops, and
> they RARELY go above 1k iops during normal use patterns.
>
> -  Jeff
>
>
>
>>
>> On Thu, Jun 7, 2018 at 6:20 AM, Riccardo Ferrari 
>> wrote:
>>
>>> Dear list,
>>>
>>> We have a small cluster on AWS EC2-Classic and we are planning to move
>>> it to a VPC.
>>>
>>> I know this has been discussed few times already including here:
>>> https://mail-archives.apache.org/mod_mbox/cassandra-user/201
>>> 406.mbox/%3CCA+VSrLopop7Th8nX20aOZ3As75g2jrJm3ryX119dekLYNHq
>>> f...@mail.gmail.com%3E - Thanks Alain RODRIGUEZ!
>>> or here:
>>> http://grokbase.com/t/cassandra/user/1498y74hn7/moving-cassa
>>> ndra-from-ec2-classic-into-vpc - Thanks Ben Bromhead!
>>>
>>> However I can't find anyone suggesting ClassicLink to perform such
>>> migration, maybe because was recent addition compared to those posts. So
>>> current status is:
>>>
>>>
>>>- 3 nodes : m1.xlarge (ephemeral storage)
>>>- Cassandra 3.0.8
>>>- Few keyspaces with {'class': 'SimpleStrategy',
>>>'replication_factor': '3'}
>>>- endpoint_snitch: Ec2Snitch
>>>- connectivity on private ips (local_address, rpc_address, no
>>>broadcast_address)
>>>- average load pretty high (we need to scale up)
>>>- vnodes
>>>
>>> My idea is to migrate:
>>>
>>>- Add intances on to the existing cluster (1 by 1):
>>>   - Same cassandra version
>>>   - Same cluster name
>>>   - Seed list from ec2 classic list
>>>- Run repair and cleanup
>>>- Update seed list (cassandar config and services)
>>>- Decommision old instances (1 by 1)
>>>- Run repair and cleanup again
>>>
>>>
>>> Question time:
>>>
>>>- What is the best approach, in 2018, for migrating a cluster from
>>>EC2Classic to VPC with 0 downtime?
>>>- Should we scale up before or during the migration? I know I should
>>>expect some load from data streaming, at the same time we're adding
>>>capacity.
>>>- We would like to migrato to GossipingPropertyFileSnitch, can we
>>>mix them?
>>>- Any gothca?
>>>- We are thinking to move to EBS (due to cluster size and
>>>snapshotting caoabilities). Any hardware reccomendation?
>>>
>>> Any suggestion is much appreciated!
>>>
>>>
>>>
>>>
>>
>

Re: Cassandra 3.0.X migarte to VPC

[Jeff Jirsa]

On Thu, Jun 7, 2018 at 9:12 AM, Nitan Kainth  wrote:

> Riccardo,
>
> Simplest method can be is to add VPC as an additional datacenter to
> existing cluster; once New DC has synced up all data, just switch over your
> application. Only caveat is that there should be enough network bandwidth
> between EC2 and VPC.
>
> Horizontal scaling will help to some extent.
>
> Changing snitch will cause data movement but for new DC, I believe you can
> chose. Leaving others comment.
>

If you change the snitch, and the data ownership changes, you run the risk
of violating at least consistency guarantees, if not outright losing data.

Changing the snitch is only something you should attempt if you understand
the implications. Cassandra does NOT re-stream data to the right places
if/when you change a snitch.



>
> With EBS, use right instance where IOPs are as high as possible.
> Generally, SSDs IOPs are 75K; what is best for you depends on workload. I
> have seen 20K IOPs supporting production loads in reasonable cluster.
>

This is very much workload dependent. I know of people running thousands of
nodes / petabytes of EBS on 4T GP2 volumes which provide 10k iops, and they
RARELY go above 1k iops during normal use patterns.

-  Jeff



>
> On Thu, Jun 7, 2018 at 6:20 AM, Riccardo Ferrari 
> wrote:
>
>> Dear list,
>>
>> We have a small cluster on AWS EC2-Classic and we are planning to move it
>> to a VPC.
>>
>> I know this has been discussed few times already including here:
>> https://mail-archives.apache.org/mod_mbox/cassandra-user/201
>> 406.mbox/%3CCA+VSrLopop7Th8nX20aOZ3As75g2jrJm3ryX119dekLYNHq
>> f...@mail.gmail.com%3E - Thanks Alain RODRIGUEZ!
>> or here:
>> http://grokbase.com/t/cassandra/user/1498y74hn7/moving-
>> cassandra-from-ec2-classic-into-vpc - Thanks Ben Bromhead!
>>
>> However I can't find anyone suggesting ClassicLink to perform such
>> migration, maybe because was recent addition compared to those posts. So
>> current status is:
>>
>>
>>- 3 nodes : m1.xlarge (ephemeral storage)
>>- Cassandra 3.0.8
>>- Few keyspaces with {'class': 'SimpleStrategy',
>>'replication_factor': '3'}
>>- endpoint_snitch: Ec2Snitch
>>- connectivity on private ips (local_address, rpc_address, no
>>broadcast_address)
>>- average load pretty high (we need to scale up)
>>- vnodes
>>
>> My idea is to migrate:
>>
>>- Add intances on to the existing cluster (1 by 1):
>>   - Same cassandra version
>>   - Same cluster name
>>   - Seed list from ec2 classic list
>>- Run repair and cleanup
>>- Update seed list (cassandar config and services)
>>- Decommision old instances (1 by 1)
>>- Run repair and cleanup again
>>
>>
>> Question time:
>>
>>- What is the best approach, in 2018, for migrating a cluster from
>>EC2Classic to VPC with 0 downtime?
>>- Should we scale up before or during the migration? I know I should
>>expect some load from data streaming, at the same time we're adding
>>capacity.
>>- We would like to migrato to GossipingPropertyFileSnitch, can we mix
>>them?
>>- Any gothca?
>>- We are thinking to move to EBS (due to cluster size and
>>snapshotting caoabilities). Any hardware reccomendation?
>>
>> Any suggestion is much appreciated!
>>
>>
>>
>>
>

Re: Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.

[Nitan Kainth]

Check logs in your nodes, there could be GC or other issues causing
timeouts. This can happen for various reasons, network blips, node down,
node over loaded etc.

On Thu, Jun 7, 2018 at 6:16 AM, Saurav Suman  wrote:

> Hi Team,
>
>
> I am getting below error in all the nodes of cassandra . Can anyone let me
> know what could be the cause. I am using apache cassandra 3.0.10.
>
>
>
> Jun 06, 2018 2:59:04 PM com.google.common.cache.LocalCache$Segment$1 run
> WARNING: Exception thrown during refresh
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out
> - received only 0 responses.
> at com.google.common.util.concurrent.AbstractFuture$
> Sync.getValue(AbstractFuture.java:299)
> at com.google.common.util.concurrent.AbstractFuture$
> Sync.get(AbstractFuture.java:286)
> at com.google.common.util.concurrent.AbstractFuture.get(
> AbstractFuture.java:116)
> at com.google.common.util.concurrent.Uninterruptibles.
> getUninterruptibly(Uninterruptibles.java:137)
> at com.google.common.cache.LocalCache$Segment.
> getAndRecordStats(LocalCache.java:2348)
> at com.google.common.cache.LocalCache$Segment$1.run(
> LocalCache.java:2331)
> at com.google.common.util.concurrent.MoreExecutors$
> DirectExecutor.execute(MoreExecutors.java:457)
> at com.google.common.util.concurrent.ExecutionList.
> executeListener(ExecutionList.java:156)
> at com.google.common.util.concurrent.ExecutionList.
> execute(ExecutionList.java:145)
> at com.google.common.util.concurrent.AbstractFuture.
> setException(AbstractFuture.java:202)
> at com.google.common.util.concurrent.Futures$
> ChainingListenableFuture.run(Futures.java:902)
> at com.google.common.util.concurrent.MoreExecutors$
> DirectExecutor.execute(MoreExecutors.java:457)
> at com.google.common.util.concurrent.ExecutionList.
> executeListener(ExecutionList.java:156)
> at com.google.common.util.concurrent.ExecutionList.
> execute(ExecutionList.java:145)
> at com.google.common.util.concurrent.ListenableFutureTask.done(
> ListenableFutureTask.java:91)
> at java.util.concurrent.FutureTask.finishCompletion(
> FutureTask.java:384)
> at java.util.concurrent.FutureTask.setException(FutureTask.java:251)
> at java.util.concurrent.FutureTask.run(FutureTask.java:271)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at org.apache.cassandra.concurrent.NamedThreadFactory.
> lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: 
> org.apache.cassandra.exceptions.ReadTimeoutException:
> Operation timed out - received only 0 responses.
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(
> CassandraRoleManager.java:516)
> at org.apache.cassandra.auth.CassandraRoleManager.isSuper(
> CassandraRoleManager.java:305)
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:52)
> at org.apache.cassandra.auth.AuthenticatedUser.isSuper(
> AuthenticatedUser.java:71)
> at org.apache.cassandra.auth.CassandraAuthorizer.authorize(
> CassandraAuthorizer.java:76)
> at org.apache.cassandra.auth.PermissionsCache$1$1.call(
> PermissionsCache.java:136)
> at org.apache.cassandra.auth.PermissionsCache$1$1.call(
> PermissionsCache.java:131)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 4 more
>
>
>
> *Saurav Suman*
> BigData Consultant
> Email- saurav.su...@widas.in
> 
> *WidasConcepts India Pvt. Ltd*
> UNIT II, SJR HUB, Sarjapur Road
> Bellandur, Bangalore 560103
>
>

Re: Cassandra 3.0.X migarte to VPC

[Nitan Kainth]

Riccardo,

Simplest method can be is to add VPC as an additional datacenter to
existing cluster; once New DC has synced up all data, just switch over your
application. Only caveat is that there should be enough network bandwidth
between EC2 and VPC.

Horizontal scaling will help to some extent.

Changing snitch will cause data movement but for new DC, I believe you can
chose. Leaving others comment.

With EBS, use right instance where IOPs are as high as possible. Generally,
SSDs IOPs are 75K; what is best for you depends on workload. I have seen
20K IOPs supporting production loads in reasonable cluster.

On Thu, Jun 7, 2018 at 6:20 AM, Riccardo Ferrari  wrote:

> Dear list,
>
> We have a small cluster on AWS EC2-Classic and we are planning to move it
> to a VPC.
>
> I know this has been discussed few times already including here:
> https://mail-archives.apache.org/mod_mbox/cassandra-user/
> 201406.mbox/%3CCA+VSrLopop7Th8nX20aOZ3As75g2jrJm
> 3ryx119deklynhq...@mail.gmail.com%3E - Thanks Alain RODRIGUEZ!
> or here:
> http://grokbase.com/t/cassandra/user/1498y74hn7/moving-cassandra-from-ec2-
> classic-into-vpc - Thanks Ben Bromhead!
>
> However I can't find anyone suggesting ClassicLink to perform such
> migration, maybe because was recent addition compared to those posts. So
> current status is:
>
>
>- 3 nodes : m1.xlarge (ephemeral storage)
>- Cassandra 3.0.8
>- Few keyspaces with {'class': 'SimpleStrategy', 'replication_factor':
>'3'}
>- endpoint_snitch: Ec2Snitch
>- connectivity on private ips (local_address, rpc_address, no
>broadcast_address)
>- average load pretty high (we need to scale up)
>- vnodes
>
> My idea is to migrate:
>
>- Add intances on to the existing cluster (1 by 1):
>   - Same cassandra version
>   - Same cluster name
>   - Seed list from ec2 classic list
>- Run repair and cleanup
>- Update seed list (cassandar config and services)
>- Decommision old instances (1 by 1)
>- Run repair and cleanup again
>
>
> Question time:
>
>- What is the best approach, in 2018, for migrating a cluster from
>EC2Classic to VPC with 0 downtime?
>- Should we scale up before or during the migration? I know I should
>expect some load from data streaming, at the same time we're adding
>capacity.
>- We would like to migrato to GossipingPropertyFileSnitch, can we mix
>them?
>- Any gothca?
>- We are thinking to move to EBS (due to cluster size and snapshotting
>caoabilities). Any hardware reccomendation?
>
> Any suggestion is much appreciated!
>
>
>
>

Re: Single Host: Fix "Unknown CF" issue

[mm]

Hi,

we will follow the recommendation not to use materialized views.

Thanks a lot to both of you!
You helped me a lot.

Oh and besides: We are also using the lagom framework :) So we will also 
be able to regenerate a Read-Side if have to.


greetings,
Michael


On 07.06.2018 13:45, Evelyn Smith wrote:

Hey Michael,

In the case that you have a production cluster set up with multiple
nodes, assuming you have rf>1 it’s easier to just replace the broken
node and restore it’s data. (For future reference)

I wasn’t sure if view was referring to materialised view at the time
although Pradeeps comment along with your own suggest it might (I
didn’t get a chance to look through the code to confirm if view was
MV or something else and I’m not that familiar with the code base).

As far as the choice of using Materialised Views, they aren’t being
deprecated they are currently marked as experimental and most people
strongly advise you to not use them. If you can avoid it don’t do
it. They’re associated with a lot of bugs and scalability issues.
Also they’re just hard to do right if you aren’t exceptionally
familiar with Cassandra.

Regards,
Evelyn.


On 7 Jun 2018, at 3:05 am, Pradeep Chhetri 
wrote:

Hi Michael,

We have faced the same situation as yours in our production
environment where we suddenly got "Unknown CF Exception" for
materialized views too. We are using Lagom apps with cassandra for
persistence. In our case, since these views can be regenerated from
the original events, we were able to safely recover.

Few suggestions from my operations experience:

1) Upgrade your cassandra cluster to 3.11.2 because there are lots
of bug fixes specific to materialized views.
2) Never let your application create/update/delete cassandra
table/materialized views. Always create them manually to make sure
that only connection is doing the operation.

Regards,
Pradeep

On Wed, Jun 6, 2018 at 9:44 PM,  wrote:
Hi Evelyn,

thanks a lot for your detailed response message.

The data is not important. We've already wiped the data and created
a new cassandra installation. The data re-import task is already
running. We've lost the data for a couple of months but in this case
this does not matter.

Nevertheless we will try what you told us - just to be
smarter/faster if this happens in production (where we will setup a
cassandra cluster with multiple cassandra nodes anyway). I will drop
you a note when we are done.

Hmmm... the problem is within a "View". Are this the materialized
views?

I'm asking this because:
* Someone on the internet (stackoverflow if a recall correctly)
mentioned that using materialized views are to be deprecated.
* I had been on a datastax workshop in Zurich a couple of days ago
where a datastax employee told me that we should not use
materialized views - it is better to create & fill all tables
directly.

Would you also recommend not to use materialized views? As this
problem is related to a view - maybe we could avoid this problem
simply by following this recommendation.

Thanks a lot again!

Greetings,
Michael

On 06.06.2018 16:48, Evelyn Smith wrote:
Hi Michael,

So I looked at the code, here are some stages of your error message:
1. at


org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292)

[apache-cassandra-3.11.0.jar:3.11.0
At this step Cassandra is running through the keyspaces in it’s
schema turning off compactions for all tables before it starts
rerunning the commit log (so it isn’t an issue with the commit
log).
2. at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127)
~[apache-cassandra-3.11.0.jar:3.11.0]
Loading key space related to the column family that is erroring out
3. at org.apache.cassandra.db.Keyspace.(Keyspace.java:324)
~[apache-cassandra-3.11.0.jar:3.11.0]
Cassandra has initialised the column family and is reloading the
view
4. at


org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204)

~[apache-cassandra-3.11.0.jar:3.11.0]
At this point I haven’t had enough time to tell if Cassandra is
requesting info on a column specifically or still requesting
information on a column family. Regardless, given we already rule
out
issues with the SSTables and their directory and Cassandra is yet to
start processing the commit log this to me suggests it’s something
wrong in one of the system keyspaces storing the schema information.

There should definitely be a way to resolve this with zero data loss
by either:
1. Fixing the issue in the system keyspace SSTables (hard)
2. Rerunning the commit log on a new Cassandra node that has been
restored from the current one (I’m not sure if this is possible
but
I’ll figure it out tomorrow)

The alternative is if you are ok with losing the commitlog then you
can backup the data and restore it to a new node (or the same node
but
with everything blown away). This isn’t a trivial process though
I’ve done it a few times.

How important is the data?

Happy to come back to this tomorrow (need some sleep)

Regards,

Re: Single Host: Fix "Unknown CF" issue

[Evelyn Smith]

Hey Michael,

In the case that you have a production cluster set up with multiple nodes, 
assuming you have rf>1 it’s easier to just replace the broken node and restore 
it’s data. (For future reference)

I wasn’t sure if view was referring to materialised view at the time although 
Pradeeps comment along with your own suggest it might (I didn’t get a chance to 
look through the code to confirm if view was MV or something else and I’m not 
that familiar with the code base).

As far as the choice of using Materialised Views, they aren’t being deprecated 
they are currently marked as experimental and most people strongly advise you 
to not use them. If you can avoid it don’t do it. They’re associated with a lot 
of bugs and scalability issues. Also they’re just hard to do right if you 
aren’t exceptionally familiar with Cassandra.

Regards,
Evelyn.

> On 7 Jun 2018, at 3:05 am, Pradeep Chhetri  wrote:
> 
> Hi Michael,
> 
> We have faced the same situation as yours in our production environment where 
> we suddenly got "Unknown CF Exception" for materialized views too. We are 
> using Lagom apps with cassandra for persistence. In our case, since these 
> views can be regenerated from the original events, we were able to safely 
> recover.
> 
> Few suggestions from my operations experience:
> 
> 1) Upgrade your cassandra cluster to 3.11.2 because there are lots of bug 
> fixes specific to materialized views.
> 2) Never let your application create/update/delete cassandra 
> table/materialized views. Always create them manually to make sure that only 
> connection is doing the operation.
> 
> Regards,
> Pradeep
> 
> 
> 
> On Wed, Jun 6, 2018 at 9:44 PM, mailto:m...@vis.at>> wrote:
> Hi Evelyn,
> 
> thanks a lot for your detailed response message.
> 
> The data is not important. We've already wiped the data and created a new 
> cassandra installation. The data re-import task is already running. We've 
> lost the data for a couple of months but in this case this does not matter.
> 
> Nevertheless we will try what you told us - just to be smarter/faster if this 
> happens in production (where we will setup a cassandra cluster with multiple 
> cassandra nodes anyway). I will drop you a note when we are done.
> 
> Hmmm... the problem is within a "View". Are this the materialized views?
> 
> I'm asking this because:
> * Someone on the internet (stackoverflow if a recall correctly) mentioned 
> that using materialized views are to be deprecated.
> * I had been on a datastax workshop in Zurich a couple of days ago where a 
> datastax employee told me that we should not use materialized views - it is 
> better to create & fill all tables directly.
> 
> Would you also recommend not to use materialized views? As this problem is 
> related to a view - maybe we could avoid this problem simply by following 
> this recommendation.
> 
> Thanks a lot again!
> 
> Greetings,
> Michael
> 
> 
> 
> 
> On 06.06.2018 16:48, Evelyn Smith wrote:
> Hi Michael,
> 
> So I looked at the code, here are some stages of your error message:
> 1. at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292)
> [apache-cassandra-3.11.0.jar:3.11.0
>  At this step Cassandra is running through the keyspaces in it’s
> schema turning off compactions for all tables before it starts
> rerunning the commit log (so it isn’t an issue with the commit log).
> 2. at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>  Loading key space related to the column family that is erroring out
> 3. at org.apache.cassandra.db.Keyspace.(Keyspace.java:324)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>  Cassandra has initialised the column family and is reloading the view
> 4. at
> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
>  At this point I haven’t had enough time to tell if Cassandra is
> requesting info on a column specifically or still requesting
> information on a column family. Regardless, given we already rule out
> issues with the SSTables and their directory and Cassandra is yet to
> start processing the commit log this to me suggests it’s something
> wrong in one of the system keyspaces storing the schema information.
> 
> There should definitely be a way to resolve this with zero data loss
> by either:
> 1. Fixing the issue in the system keyspace SSTables (hard)
> 2. Rerunning the commit log on a new Cassandra node that has been
> restored from the current one (I’m not sure if this is possible but
> I’ll figure it out tomorrow)
> 
> The alternative is if you are ok with losing the commitlog then you
> can backup the data and restore it to a new node (or the same node but
> with everything blown away). This isn’t a trivial process though
> I’ve done it a few times.
> 
> How important is the data?
> 
> Happy to come back to this tomorrow (need some sleep)
> 
> Regards,
> Eevee.
> 
> On 5 Jun 2018, at 7:32 pm, m...@vis.at 

Cassandra 3.0.X migarte to VPC

[Riccardo Ferrari]

Dear list,

We have a small cluster on AWS EC2-Classic and we are planning to move it
to a VPC.

I know this has been discussed few times already including here:
https://mail-archives.apache.org/mod_mbox/cassandra-user/201406.mbox/%3cca+vsrlopop7th8nx20aoz3as75g2jrjm3ryx119deklynhq...@mail.gmail.com%3E
- Thanks Alain RODRIGUEZ!
or here:
http://grokbase.com/t/cassandra/user/1498y74hn7/moving-cassandra-from-ec2-classic-into-vpc
- Thanks Ben Bromhead!

However I can't find anyone suggesting ClassicLink to perform such
migration, maybe because was recent addition compared to those posts. So
current status is:


   - 3 nodes : m1.xlarge (ephemeral storage)
   - Cassandra 3.0.8
   - Few keyspaces with {'class': 'SimpleStrategy', 'replication_factor':
   '3'}
   - endpoint_snitch: Ec2Snitch
   - connectivity on private ips (local_address, rpc_address, no
   broadcast_address)
   - average load pretty high (we need to scale up)
   - vnodes

My idea is to migrate:

   - Add intances on to the existing cluster (1 by 1):
  - Same cassandra version
  - Same cluster name
  - Seed list from ec2 classic list
   - Run repair and cleanup
   - Update seed list (cassandar config and services)
   - Decommision old instances (1 by 1)
   - Run repair and cleanup again


Question time:

   - What is the best approach, in 2018, for migrating a cluster from
   EC2Classic to VPC with 0 downtime?
   - Should we scale up before or during the migration? I know I should
   expect some load from data streaming, at the same time we're adding
   capacity.
   - We would like to migrato to GossipingPropertyFileSnitch, can we mix
   them?
   - Any gothca?
   - We are thinking to move to EBS (due to cluster size and snapshotting
   caoabilities). Any hardware reccomendation?

Any suggestion is much appreciated!

Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.

[Saurav Suman]

Hi Team,


I am getting below error in all the nodes of cassandra . Can anyone let me know 
what could be the cause. I am using apache cassandra 3.0.10.



Jun 06, 2018 2:59:04 PM com.google.common.cache.LocalCache$Segment$1 run
WARNING: Exception thrown during refresh
java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at 
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at 
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at 
com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
at 
com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348)
at com.google.common.cache.LocalCache$Segment$1.run(LocalCache.java:2331)
at 
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at 
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at 
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at 
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at 
com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:902)
at 
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at 
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at 
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at 
com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91)
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384)
at java.util.concurrent.FutureTask.setException(FutureTask.java:251)
at java.util.concurrent.FutureTask.run(FutureTask.java:271)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: 
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - 
received only 0 responses.
at 
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:516)
at 
org.apache.cassandra.auth.CassandraRoleManager.isSuper(CassandraRoleManager.java:305)
at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:52)
at 
org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71)
at 
org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:76)
at 
org.apache.cassandra.auth.PermissionsCache$1$1.call(PermissionsCache.java:136)
at 
org.apache.cassandra.auth.PermissionsCache$1$1.call(PermissionsCache.java:131)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 4 more



Saurav Suman
BigData Consultant
Email- saurav.su...@widas.in

WidasConcepts India Pvt. Ltd
UNIT II, SJR HUB, Sarjapur Road
Bellandur, Bangalore 560103