RE: Adding new node to cluster

[SEAN_R_DURITY]

What snitch are you using? You may need to do some work on your topology file 
(or rackdc) to make sure you have the topology you want. Also, it is possible 
you may need to restart OpsCenter agents and/or your browser to see the nodes 
represented properly in OpsCenter.


Sean Durity – Cassandra Admin, Home Depot

From: Batranut Bogdan [mailto:batra...@yahoo.com]
Sent: Tuesday, February 17, 2015 10:20 AM
To: user@cassandra.apache.org; reynald.bourtembo...@esrf.fr
Subject: Re: Adding new node to cluster

Hello,

I know that UN is good, but what troubles me is the addition of the own node's 
ip in it's yaml seeds section.

On Tuesday, February 17, 2015 3:40 PM, Reynald Bourtembourg 
reynald.bourtembo...@esrf.frmailto:reynald.bourtembo...@esrf.fr wrote:

Hi Bogdan

In nodetool status:

  *   UJ: means your node is Up and Joining
  *   UN: means your node is Up and in Normal state
UN in nodetool is good ;-)

On 17/02/2015 13:56, Batranut Bogdan wrote:
Hello all,

I have an existing cluster. When adding a new node, I saw that Opscenter saw 
the node in an unknown cluster. In the yaml, the cluster name is the same. So i 
have stopped the node and added it's ip address in the list of seeds. Now 
Opscenter sees my node. But nodetool status now sees it as UN, instead of UJ 
when it first started. One other mension is that even if I stop the node, 
remove it's ip from the list of seeds, Opscenter sees the node in the known 
clustre but nodetool sees it as UN. I am not sure what the implications of 
adding a node's ip in it's seed list are and I think that for the existing 
nodes I have might done the same. Eg. started with it's ip in the seed list but 
after removing it and having to restart the nodes for whatever reason, I did 
not see any changes.

Is my cluster ok, or what do I need to do to bring the cluster to a good state?

Thank you.





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Two problems with Cassandra

[SEAN_R_DURITY]

Full table scans are not the best use case for Cassandra. Without some kind of 
pagination, the node taking the request (the coordinator node) will try to 
assemble the data from all nodes to return to the client. With a dataset of any 
decent size, it will overwhelm the single node.

Pagination is supported in newer versions of Cassandra (2.0.x+, I think) and 
some drivers. You can see there is other discussion on the list about the best 
ways to split your workload and do some parallel processing. Something I 
haven’t seen mentioned recently (but probably discussed before I joined the 
list) is setting up a separate, analytics DC. There you could integrate with 
hadoop or spark or just size your nodes differently to handle an analytics type 
workload.

We have found that it is better to use a list of known keys and pull back rows 
(aka partitions) individually for any table scan type operations. However, we 
are usually able to generate the list of keys outside of Cassandra…


Sean Durity – Cassandra Admin, Home Depot

From: Pavel Velikhov [mailto:pavel.velik...@gmail.com]
Sent: Thursday, February 12, 2015 4:23 AM
To: user@cassandra.apache.org
Subject: Re: Two problems with Cassandra


On Feb 12, 2015, at 12:37 AM, Robert Coli 
rc...@eventbrite.commailto:rc...@eventbrite.com wrote:

On Wed, Feb 11, 2015 at 2:22 AM, Pavel Velikhov 
pavel.velik...@gmail.commailto:pavel.velik...@gmail.com wrote:
  2. While trying to update the full dataset with a simple transformation 
(again via python driver), single node and clustered Cassandra run out of 
memory no matter what settings I try, even I put a lot of sleeps into the mix. 
However simpler transformations (updating just one column, specially when there 
is a lot of processing overhead) work just fine.

What does a simple transformation mean here? Assuming a reasonable sized 
heap, OOM sounds like you're trying to update a large number of large 
partitions in a single operation.

In general, in Cassandra, you're best off interacting with a single or small 
number of partitions in any given interaction.

=Rob


Hi Robert!

  Simple transformation is changing just a single column value (for I usually 
do it for the whole dataset).
  But when I was running out of memory, I was reading in 5 columns and updating 
3. Some of them could be big, but I need to check and rerun this case.
  (I worked around this by dumping to files and then scanning the files and 
updating the database, but this stinks!)

  I don’t quite understand the fundamentals of Cassandra - if I’m just doing 
one scan with a reasonable number of columns that I fetch, and I’m updating at 
the same time, what’s happening there? Why eat up so much memory and die?



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Adding new node to cluster

[SEAN_R_DURITY]

SimpleSnitch is not rack aware. You would want to choose seed nodes and then 
not change them. Seed nodes apparently don’t bootstrap. All nodes need the same 
seeds in the yaml file. Here is more info: 
http://www.datastax.com/documentation/cassandra/2.0/cassandra/initialize/initializeSingleDS.html



Sean Durity – Cassandra Admin, Big Data Team
To engage the team, create a 
requesthttps://portal.homedepot.com/sites/bigdata/SitePages/Big%20Data%20Engagement%20Request.aspx

From: Batranut Bogdan [mailto:batra...@yahoo.com]
Sent: Tuesday, February 17, 2015 3:28 PM
To: user@cassandra.apache.org; reynald.bourtembo...@esrf.fr
Subject: Re: Adding new node to cluster

Hello,

I use SimpleSnitch. All the nodes are in the sane datacenter. Not sure if all 
are in the same rack.

On Tuesday, February 17, 2015 8:53 PM, 
sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com 
sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com wrote:

What snitch are you using? You may need to do some work on your topology file 
(or rackdc) to make sure you have the topology you want. Also, it is possible 
you may need to restart OpsCenter agents and/or your browser to see the nodes 
represented properly in OpsCenter.


Sean Durity – Cassandra Admin, Home Depot

From: Batranut Bogdan [mailto:batra...@yahoo.com]
Sent: Tuesday, February 17, 2015 10:20 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org; 
reynald.bourtembo...@esrf.frmailto:reynald.bourtembo...@esrf.fr
Subject: Re: Adding new node to cluster

Hello,

I know that UN is good, but what troubles me is the addition of the own node's 
ip in it's yaml seeds section.

On Tuesday, February 17, 2015 3:40 PM, Reynald Bourtembourg 
reynald.bourtembo...@esrf.frmailto:reynald.bourtembo...@esrf.fr wrote:

Hi Bogdan

In nodetool status:

  *   UJ: means your node is Up and Joining
  *   UN: means your node is Up and in Normal state
UN in nodetool is good ;-)

On 17/02/2015 13:56, Batranut Bogdan wrote:
Hello all,

I have an existing cluster. When adding a new node, I saw that Opscenter saw 
the node in an unknown cluster. In the yaml, the cluster name is the same. So i 
have stopped the node and added it's ip address in the list of seeds. Now 
Opscenter sees my node. But nodetool status now sees it as UN, instead of UJ 
when it first started. One other mension is that even if I stop the node, 
remove it's ip from the list of seeds, Opscenter sees the node in the known 
clustre but nodetool sees it as UN. I am not sure what the implications of 
adding a node's ip in it's seed list are and I think that for the existing 
nodes I have might done the same. Eg. started with it's ip in the seed list but 
after removing it and having to restart the nodes for whatever reason, I did 
not see any changes.

Is my cluster ok, or what do I need to do to bring the cluster to a good state?

Thank you.





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its 

RE: Run Mixed Workload using two instances on one node

[SEAN_R_DURITY]

Yes, for over 2 years.

As for #2 - you would keep all CFs in both DCs. But, maybe only do RF=2 in OLTP 
and 3 in reporting. Not sure of all your requirements. Writes are fast and 
cheap in Cassandra, so I wouldn’t be concerned with “extra” writes in the OLTP 
DC.


Sean Durity – Cassandra Admin, Big Data Team
From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Tuesday, March 17, 2015 1:29 PM
To: user@cassandra.apache.org
Subject: Re: Run Mixed Workload using two instances on one node

Thanks Sean. Are you using 2 Cassandra instances on single node in PRODUCTION 
environment?

Yes. We considered having separate virtual DC for OLTP and Reporting something 
similar to the approach mentioned at 
http://www.datastax.com/docs/datastax_enterprise3.1/solutions/dse_search_cluster
 .
The approach mentions dedicated nodes for each workload.

In Virtual DC approach, we have following concerns:
1. We can't afford dedicated Reporting nodes for small customers. Bigger 
clusters may have dedicated reporting nodes.
2. When OLTP DC replicates transaction data to Reporting DC in real time, what 
would trigger population of other reporting tables in Reporting DC. I am not 
sure whether triggers are stable yet?

Anuj


On Tuesday, 17 March 2015 6:58 PM, 
sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com 
sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com wrote:

We run two cassandra nodes on the same host for a use case that requires a 
random ordered ring and a byte ordered ring. It is technically feasible. 
However, it makes administration of the rings a bit tougher (different ports 
for one, etc.). OpsCenter agents can only connect to one of the rings at a 
time. (Perhaps we could run 2 agents also, on different ports…)

Have you considered using a separate, logical Data Center for the reporting use 
cases? OLTP clients use the OLTP DC; reporting clients use the Reporting DC. 
The reporting CFs could have smaller replication factor on the OLTP DC, if 
needed, to keep writes and data size minimized there.


Sean Durity – Cassandra Admin, Big Data Team
From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Tuesday, March 17, 2015 1:36 AM
To: Ali Akhtar; user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: Run Mixed Workload using two instances on one node

I understand that 2 instances on one node looks a weird solution. But can have 
dedicated reporting nodes for big customers but not for small customers.

My questions would be:
1. What is the technical reasoning? What problems you foresee  if we use 2 C* 
instances on one node in production? We have ample HW on each server and mostly 
it's under-utilized. We just want that heavy reporting must not impact OLTP and 
both OLTP and reporting should be individually scalable.
2. I think we dont need Elastic Search. We just need a plain Reporting DB which 
can reply to reporting queries.We can create our own CF as indexes. We dont 
need overhead of another 3PP for our current reporting needs.

Thanks
Anuj







On Tuesday, 17 March 2015 9:59 AM, Ali Akhtar 
ali.rac...@gmail.commailto:ali.rac...@gmail.com wrote:

I don't think its recommended to have two instances on the same node.
Have you considered using something like elasticsearch for the reports? Its 
designed for that sort of thing.
On Mar 17, 2015 8:07 AM, Anuj Wadehra 
anujw_2...@yahoo.co.inmailto:anujw_2...@yahoo.co.in wrote:

Hi,

We are trying to Decouple our Reporting DB from OLTP. Need urgent help on the 
feasibility of proposed solution for PRODUCTION.

Use Case: Currently, our OLTP and Reporting application and DB are same. Some 
CF are used for both OLTP and Reporting while others are solely used for 
Reporting.Every business transaction synchronously updates the main OLTP CF and 
asynchronously updates other Reporting CFs.

Problem Statement:
1. Decouple Reporting and OLTP such that Reporting load can't impact  OLTP 
performance.
2. Scaling of Reporting  and OLTP modules must be independent
3. OLTP client should not update all Reporting CFs. We generate Data Records on 
File sytem/shared disk.Reporting should use these Records to create Reporting 
DB.
4. Small customers may do OLTP and Reporting on same 3-node cluster. Bigger 
customers can be given an option to have dedicated OLTP and Reporting nodes. 
So, standard Hardware box should be usable for 3 deployments (OLTP,Reporting or 
OLTP+Reporting)

Note: Reporting is ad-hoc, may involve full table scans and does not involve 
Analytics. Data size is huge 2TB (OLTP+Reporting) per node.

Hardware : Standard deployment -3 node cluster with each node having 24 cores, 
64GB RAM, 400GB * 6 SSDs in RAID5

Proposed Solution:
1. Split OLTP and Reporting clients into two application components.
2. For small deployments where more than 3 nodes are not required:
A. Install 2 Cassandra instances on each node one for OLTP and other for 
Reporting
B. To distribute I/O load in 2:1 --Remove RAID5 (as Cassandra offers 

RE: Run Mixed Workload using two instances on one node

[SEAN_R_DURITY]

We run two cassandra nodes on the same host for a use case that requires a 
random ordered ring and a byte ordered ring. It is technically feasible. 
However, it makes administration of the rings a bit tougher (different ports 
for one, etc.). OpsCenter agents can only connect to one of the rings at a 
time. (Perhaps we could run 2 agents also, on different ports…)

Have you considered using a separate, logical Data Center for the reporting use 
cases? OLTP clients use the OLTP DC; reporting clients use the Reporting DC. 
The reporting CFs could have smaller replication factor on the OLTP DC, if 
needed, to keep writes and data size minimized there.


Sean Durity – Cassandra Admin, Big Data Team
From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Tuesday, March 17, 2015 1:36 AM
To: Ali Akhtar; user@cassandra.apache.org
Subject: Re: Run Mixed Workload using two instances on one node

I understand that 2 instances on one node looks a weird solution. But can have 
dedicated reporting nodes for big customers but not for small customers.

My questions would be:
1. What is the technical reasoning? What problems you foresee  if we use 2 C* 
instances on one node in production? We have ample HW on each server and mostly 
it's under-utilized. We just want that heavy reporting must not impact OLTP and 
both OLTP and reporting should be individually scalable.
2. I think we dont need Elastic Search. We just need a plain Reporting DB which 
can reply to reporting queries.We can create our own CF as indexes. We dont 
need overhead of another 3PP for our current reporting needs.

Thanks
Anuj







On Tuesday, 17 March 2015 9:59 AM, Ali Akhtar 
ali.rac...@gmail.commailto:ali.rac...@gmail.com wrote:

I don't think its recommended to have two instances on the same node.
Have you considered using something like elasticsearch for the reports? Its 
designed for that sort of thing.
On Mar 17, 2015 8:07 AM, Anuj Wadehra 
anujw_2...@yahoo.co.inmailto:anujw_2...@yahoo.co.in wrote:

Hi,

We are trying to Decouple our Reporting DB from OLTP. Need urgent help on the 
feasibility of proposed solution for PRODUCTION.

Use Case: Currently, our OLTP and Reporting application and DB are same. Some 
CF are used for both OLTP and Reporting while others are solely used for 
Reporting.Every business transaction synchronously updates the main OLTP CF and 
asynchronously updates other Reporting CFs.

Problem Statement:
1. Decouple Reporting and OLTP such that Reporting load can't impact  OLTP 
performance.
2. Scaling of Reporting  and OLTP modules must be independent
3. OLTP client should not update all Reporting CFs. We generate Data Records on 
File sytem/shared disk.Reporting should use these Records to create Reporting 
DB.
4. Small customers may do OLTP and Reporting on same 3-node cluster. Bigger 
customers can be given an option to have dedicated OLTP and Reporting nodes. 
So, standard Hardware box should be usable for 3 deployments (OLTP,Reporting or 
OLTP+Reporting)

Note: Reporting is ad-hoc, may involve full table scans and does not involve 
Analytics. Data size is huge 2TB (OLTP+Reporting) per node.

Hardware : Standard deployment -3 node cluster with each node having 24 cores, 
64GB RAM, 400GB * 6 SSDs in RAID5

Proposed Solution:
1. Split OLTP and Reporting clients into two application components.
2. For small deployments where more than 3 nodes are not required:
A. Install 2 Cassandra instances on each node one for OLTP and other for 
Reporting
B. To distribute I/O load in 2:1 --Remove RAID5 (as Cassandra offers 
replication) and assign 4 disks as JBod for OLTP and 2 disks for Reporting
C. RAM is abundant and often under-utilized , so assign 8GB each for 2 
Cassandra instance
D. To make sure that Reporting is not able to overload CPU, tune 
concurrent_reads,concurrent_writes
OLTP client will only write to OLTP DB and generate DB record. Reporting client 
will poll FS and populate Reporting DB in required format.
3. Larger customers can have Reporting clients and DB on dedicated physical 
nodes with all resources.

Key Questions:
Is it ok to run 2 Cassandra instances on one node in Production system and 
limit CPU Usage,Disk I/O and RAM as suggested above?
Any other solution for above mentioned problem statement?



Thanks
Anuj





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy 

RE: upgrade from 1.0.12 to 1.1.12

[SEAN_R_DURITY]

Yes, run upgradesstables on all nodes - unless you already force major 
compactions on all tables. I run them on a few nodes at a time to minimize 
impact to performance. The upgrade is not complete until upgradesstables 
completes on all nodes. Then you are safe to resume any streaming operations 
(repairs and bootstraps).


Sean Durity – Cassandra Admin, Big Data Team
To engage the team, create a request

-Original Message-
From: Jason Wee [mailto:peich...@gmail.com]
Sent: Wednesday, March 25, 2015 10:59 AM
To: user@cassandra.apache.org
Subject: Re: upgrade from 1.0.12 to 1.1.12

hmm... okay.

one more question
https://github.com/apache/cassandra/blob/cassandra-1.1.12/NEWS.txt  I upgraded 
directly to 1.1.12 , do I need to run nodetool upgradesstables as stipulated in 
version 1.1.3 ?

jason

On Wed, Mar 25, 2015 at 1:04 AM, Jonathan Haddad j...@jonhaddad.com wrote:
 Streaming is repair, adding  removing nodes.  In general it's a bad
 idea to do any streaming op when you've got an upgrade in progress.

 On Tue, Mar 24, 2015 at 3:14 AM Jason Wee peich...@gmail.com wrote:

 Hello,

 Reading this documentation
 http://www.datastax.com/docs/1.1/install/upgrading

 If you are upgrading to Cassandra 1.1.9 from a version earlier than
 1.1.7, all nodes must be upgraded before any streaming can take place.
 Until you upgrade all nodes, you cannot add version 1.1.7 nodes or
 later to a 1.1.7 or earlier cluster.

 Does this apply for upgrade to cassandra 1.1.12 ?

 What is cassandra streaming ? Is repair (nodetool or background),
 hinted handoff, antientropy consider streaming? if yes, how do we
 prevent streaming after a node is upgraded to 1.1.12 in a 1.0.12
 cluster environment?

 Thanks.

 Jason



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Running Cassandra on mixed OS

[SEAN_R_DURITY]

This is not for the long haul, but in order to accomplish an OS upgrade across 
the cluster, without taking an outage.

Sean Durity

From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Monday, March 02, 2015 1:15 PM
To: user@cassandra.apache.org
Subject: Re: Running Cassandra on mixed OS

I would really not recommend this.  There's enough issues that can come up with 
a distributed database that can make it hard to pinpoint problems.

In an ideal world, every machine would be completely identical.  Don't set 
yourself up for fail.  Pin the OS  all packages to specific versions.

On Mon, Mar 2, 2015 at 6:44 AM 
sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com wrote:
Cassandra 1.2.13+/2.0.12

Have any of you run a single Cassandra cluster on a mix of OS (Red Hat 5 and 6, 
for example), but with the same JVM? Any issues or concerns? If there are 
problems, how do you handle OS upgrades?



Sean R. Durity



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: sstables remain after compaction

[SEAN_R_DURITY]

In my experience, you do not want to stay on 1.1 very long. 1.08 was very 
stable. 1.1 can get bad in a hurry. 1.2 (with many things moved off-heap) is 
very much better.


Sean Durity – Cassandra Admin, Big Data Team

From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: Monday, March 02, 2015 2:01 PM
To: user@cassandra.apache.org
Subject: Re: sstables remain after compaction

On Sat, Feb 28, 2015 at 5:39 PM, Jason Wee 
peich...@gmail.commailto:peich...@gmail.com wrote:
Hi Rob, sorry for the late response, festive season here. cassandra version is 
1.0.8 and thank you, I will read on the READ_STAGE threads.

1.0.8 is pretty seriously old in 2015. I would upgrade to at least 1.2.x (via 
1.1.x) ASAP. Your cluster will be much happier, in general.

=Rob




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Turning on internal security with no downtime

[SEAN_R_DURITY]

Cassandra 1.2.19

We would like to turn on Cassandra's internal security (PasswordAuthenticator 
and CassandraAuthorizer) on the ring (away from AllowAll). (Clients are already 
passing credentials in their connections.) However, I know all nodes have to be 
switched to those before the basic security objects (system_auth) are created. 
So, an outage would be required to change all the nodes, let system_auth get 
created, alter system_auth for replication strategy, create all the 
users/permissions, repair system_auth.

For DataStax, there is a TransitionalAuthorizer that allows the system_auth to 
get created, but doesn't really require passwords. So, with a double, rolling 
bounce, you can implement security with no downtime. Anything like that for 
open source? Any other ways you have activated security without downtime?



Sean R. Durity





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: What is 'Read Reuqests' on OpsCenter exaclty?

[SEAN_R_DURITY]

Just to add to this. It seems that reads done for authentication and 
authorization (using the built-in security classes) are included in the read 
request counts.


Sean Durity – Cassandra Admin, Big Data Team
To engage the team, create a 
requesthttps://portal.homedepot.com/sites/bigdata/SitePages/Big%20Data%20Engagement%20Request.aspx

From: Bongseo Jang [mailto:grayce...@gmail.com]
Sent: Friday, April 24, 2015 6:13 AM
To: user@cassandra.apache.org
Subject: Re: What is 'Read Reuqests' on OpsCenter exaclty?

I can see two numbers matches when the redundant selects are removed with 1 
node/1 replica settings.

On 24 April 2015 at 19:09, Bongseo Jang 
grayce...@gmail.commailto:grayce...@gmail.com wrote:
You're quite right. I missed important thing first.

I found a mistake in my program while making test case. It turns out that the 
original program has 3~4 selects for non-existing row keys plus a select for 
existing row key. It was intended to do nothing but for next tests. My original 
test  counted only selects for existing row key but opscenter graph showed real 
number of request. That's it

Thank you again ~

On 24 April 2015 at 16:01, Carlos Rolo 
r...@pythian.commailto:r...@pythian.com wrote:
Let me try to reproduce your test and get back wiith some results.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: 
linkedin.com/in/carlosjuzarterolohttp://linkedin.com/in/carlosjuzarterolo
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.comhttp://www.pythian.com/

On Fri, Apr 24, 2015 at 2:35 AM, Bongseo Jang 
grayce...@gmail.commailto:grayce...@gmail.com wrote:
Thanks a lot Carlos, Sebastian :-)

My test was with 1 node/1 replica settings, on which I assumed client request = 
read request on the graph. Because there seems no read_repair and already 
CL=ONE in my case, I need more explanation, don't I? Or can any other internals 
be still involved?

Do you have more suggestions? I want to design new test narrowing the gap on 
the suggestions.

On 24 April 2015 at 00:23, Sebastian Estevez 
sebastian.este...@datastax.commailto:sebastian.este...@datastax.com wrote:

Carlos is right:

Read Requests - The number of read requests per second on the coordinator 
nodes, analogous to client reads. Monitoring the number of requests over a 
given time period reveals system read workload and usage patterns.

Avg - The average of values recorded during a time interval.

A future version of OpsC will include tooltips with these descriptions for 
better clarity.
On Apr 23, 2015 6:30 AM, Carlos Rolo 
r...@pythian.commailto:r...@pythian.com wrote:
Probably it takes in account the read repair, plus a read that have consistency 
!= 1 will produce reads on other machines (which are taken in account). I don't 
know the internals of opscenter but I would assume that this is the case.
If you want to test it further, disable read_repair, and make all your reads 
with CL=ONE. Then your client and Opscenter should match.
PS: Speculative_retry could also send reads over to more machines.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: 
linkedin.com/in/carlosjuzarterolohttp://linkedin.com/in/carlosjuzarterolo
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 
x1649tel:%2B1%20613%20565%208696%20x1649
www.pythian.comhttp://www.pythian.com/

On Thu, Apr 23, 2015 at 10:34 AM, Bongseo Jang 
grayce...@gmail.commailto:grayce...@gmail.com wrote:
I have cassandra 2.1 + OpsCenter 5.1.1 and test them.

When I monitored with opscenter 'read requests' graph, it seems the number on 
the graph is not what I expected, the number of client requests or responses.

I recorded actual number of client request and compare it with graph, then 
found they're different. The number on the graph is about 4 times larger than 
what the client claimed.

So, my question is what 'Read Reuqests' on OpsCenter counts exaclty ?

Thanks !

--
Regards,
Jang.

 a sound mind in a sound body



--





--
Regards,
Jang.

 a sound mind in a sound body



--





--
Regards,
Jang.

 a sound mind in a sound body



--
Regards,
Jang.

 a sound mind in a sound body



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any 

RE: Insert Vs Updates - Both create tombstones

[SEAN_R_DURITY]

I think you have over-simplified it just a bit here, though I may be wrong.

In order to get a tombstone on a TTL row or column, some kind of read has to 
occur. The tombstones don’t magically appear (remember, a tombstone is a 
special kind of insert). So, I think it takes at least two compactions to 
actually get rid of the data. One compaction to create tombstones, and a second 
one (after gc_grace_seconds) to rewrite the good data and leave out expired 
tombstones. And it has to happen on all replica nodes.

(I suppose it is possible that other kinds of reads might create TTL-based 
tombstones as an optimization, but I don’t know that this exists.)

Someone closer to the source code may correct me, but this is my understanding 
of how it works.

Sean Durity – Cassandra Admin, Big Data Team
To engage the team, create a 
requesthttps://portal.homedepot.com/sites/bigdata/SitePages/Big%20Data%20Engagement%20Request.aspx

From: Walsh, Stephen [mailto:stephen.wa...@aspect.com]
Sent: Thursday, May 14, 2015 6:37 AM
To: user@cassandra.apache.org
Subject: RE: Insert Vs Updates - Both create tombstones

Thanks ☺

I think you might have got your T’s and V’s mixed up ?

So we insert V2 @ T2, then insert V1 @ T1 where T1 is earlier to T2 = V2

Should it not be the  other way around?

So we insert V1 @ T1, then insert V2 @ T2 where T2 is earlier to T2 = V2


So in a tombstone manor over 5 seconds we are looking like this

Second 1
Insert V1, T1 with TTL =5

Second 2
V1, T1 (TTL 4)
Insert V1, T2 with TTL= 5

Second 3
V1, T1 (TTL 3)
V1, T2 (TTL 4)
Insert V1, T3 with TTL= 5

Second 3
V1, T1 (TTL 2)
V1, T2 (TTL 3)
V1, T3 (TTL 4)
Insert V1, T4 with TTL= 5

Second 4
V1, T1 (TTL 1)
V1, T2 (TTL 2)
V1, T3 (TTL 3)
V1, T4 (TTL 4)
Insert V1, T5 with TTL= 5

Second 5
V1, T1 (Tombstoned)
V1, T2 (TTL 1)
V1, T3 (TTL 2)
V1, T4 (TTL 3)
V1, T5 (TTL 4)

Second 6
V1, T1 (Tombstoned)
V1, T2 (Tombstoned)
V1, T3 (TTL 1)
V1, T4 (TTL 2)
V1, T5 (TTL 3)

Second 7
V1, T1 (Tombstoned)
V1, T2 (Tombstoned)
V1, T3 (Tombstoned)
V1, T4 (TTL 1)
V1, T5 (TTL 2)


Second 8
V1, T1 (Tombstoned)
V1, T2 (Tombstoned)
V1, T3 (Tombstoned)
V1, T4 (Tombstoned)
V1, T5 (TTL 1)

Second 8
V1, T1 (Tombstoned)
V1, T2 (Tombstoned)
V1, T3 (Tombstoned)
V1, T4 (Tombstoned)
V1, T5 (Tombstoned)

Second 9
(Minor Compaction run to clean up tombstones)


And if I did an “update“, the result would be the same.
And like you mentioned, if I did a query at “second 4”, the query would be 
based of 5 versions of V1 to query against, and the highest T value would be 
returned.




From: Peer, Oded [mailto:oded.p...@rsa.com]
Sent: 14 May 2015 11:12
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: RE: Insert Vs Updates - Both create tombstones

If this how you update then you are not creating tombstones.

If you used UPDATE it’s the same behavior. You are simply inserting a new value 
for the cell which does not create a tombstone.
When you modify data by using either the INSERT or the UPDATE command the value 
is stored along with a timestamp indicating the timestamp of the value.
Assume timestamp T1 is before T2 (T1  T2) and you stored value V2 with 
timestamp T2. Then you store V1 with timestamp T1.
Now you have two values of V in the DB: V2,T2, V1,T1
When you read the value of V from the DB you read both V2,T2, V1,T1, which 
may be in different sstables, Cassandra resolves the conflict by comparing the 
timestamp and returns V2.
Compaction will later take care and remove V1,T1 from the DB.


From: Walsh, Stephen [mailto:stephen.wa...@aspect.com]
Sent: Thursday, May 14, 2015 11:39 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: RE: Insert Vs Updates - Both create tombstones

Thank you,

We are updating the entire row (all columns) each second via the “insert” 
command.
So if we did updates – no tombstones would be created?
But because we are doing inserts- we are creating tombstones for each column 
each insert?


From: Ali Akhtar [mailto:ali.rac...@gmail.com]
Sent: 13 May 2015 12:10
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: Insert Vs Updates - Both create tombstones

Sorry, wrong thread. Disregard the above

On Wed, May 13, 2015 at 4:08 PM, Ali Akhtar 
ali.rac...@gmail.commailto:ali.rac...@gmail.com wrote:
If specifying 'using' timestamp, the docs say to provide microseconds, but 
where are these microseconds obtained from? I have regular java.util.Date 
objects, I can get the time in milliseconds (i.e the unix timestamp), how would 
I convert that to microseconds?

On Wed, May 13, 2015 at 3:45 PM, Peer, Oded 
oded.p...@rsa.commailto:oded.p...@rsa.com wrote:
Under the assumption that when you update the columns you also update the TTL 
for the columns then a tombstone won’t be created for those columns.
Remember that TTL is set on columns (or “cells”), not on rows, so your 
description of updating a row is slightly misleading. If every query updates 
different columns then different columns might expire 

RE: C*1.2 on JVM 1.7

[SEAN_R_DURITY]

I have run plenty of 1.2.x Cassandra versions on the Oracle JVM 1.7. I have 
used both 1.7.0_40 and 1.7.0_72 with no issues. Also have 3.2.7 DSE running on 
1.7.0_72 in PR with no issues.


Sean Durity – Cassandra Admin, Big Data Team
To engage the team, create a 
requesthttps://portal.homedepot.com/sites/bigdata/SitePages/Big%20Data%20Engagement%20Request.aspx

From: cass savy [mailto:casss...@gmail.com]
Sent: Friday, May 15, 2015 1:22 PM
To: user@cassandra.apache.org
Subject: C*1.2 on JVM 1.7



Has anybody run DSE 3.2.6(C*1.2.16) on JRE 1.7. I know its recommended that we 
have to get to JVM version 7 to get to C*2.0 and higher.

We are experiencing latency issues during rolling upgrade from 1.2 to 2.0. 
Hence cannot get o C*1.2, but plan to just upgrade JRE from 1.6 to 1.7. I 
wanted to know if anyof you have C*1.2 on JRE 1.7 in PROD and have run into 
issues.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Availability testing of Cassandra nodes

[SEAN_R_DURITY]

I do two types of node monitoring. On each host, we have a process monitor 
looking for the cassandra process. If it goes down, it will get restarted (if a 
flag is set appropriately).

Secondly, from a remote host, I have an hourly check of all nodes where I 
essentially log in to each node and execute nodetool info. If that returns an 
error, then the node is probably “up,” but hung. (Or the flag above is not set 
properly and the host was bounced/patched, but cassandra did not start.) I 
email details to the support team to investigate.


Sean Durity

From: Jiri Horky [mailto:ho...@avast.com]
Sent: Thursday, April 09, 2015 4:32 AM
To: user@cassandra.apache.org; java-driver-u...@lists.datastax.com
Subject: Re: Availability testing of Cassandra nodes

Hi Jack,

it seems there is a some misunderstanding. There are two things. One is that 
the Cassandra works for application, which may (and should) be true even if 
some of the nodes are actually down. The other thing is that even in this case 
you want to be notified that there are faulty Cassandra nodes.

Now I am trying to tackle the later case, I am not having issues with how 
client-side load balancing works.

Jirka H.
On 04/09/2015 07:15 AM, Ajay wrote:
Adding Java driver forum.
Even we like to know more on this.
-
Ajay

On Wed, Apr 8, 2015 at 8:15 PM, Jack Krupansky 
jack.krupan...@gmail.commailto:jack.krupan...@gmail.com wrote:
Just a couple of quick comments:

1. The driver is supposed to be doing availability and load balancing already.
2. If your cluster is lightly loaded, it isn't necessary to be so precise with 
load balancing.
3. If your cluster is heavily loaded, it won't help. Solution is to expand your 
cluster so that precise balancing of requests (beyond what the driver does) is 
not required.

Is there anything special about your use case that you feel is worth the extra 
treatment?

If you are having problems with the driver balancing requests and properly 
detecting available nodes or see some room for improvement, make sure to the 
issues so that they can be fixed.


-- Jack Krupansky

On Wed, Apr 8, 2015 at 10:31 AM, Jiri Horky 
ho...@avast.commailto:ho...@avast.com wrote:
Hi all,

we are thinking of how to best proceed with availability testing of
Cassandra nodes. It is becoming more and more apparent that it is rather
complex task. We thought that we should try to read and write to each
cassandra node to monitoring keyspace with a unique value with low
TTL. This helps to find an issue but it also triggers flapping of
unaffected hosts, as the key of the value which is beining inserted
sometimes belongs to an affected host and sometimes not. Now, we could
calculate the right value to insert so we can be sure it will hit the
host we are connecting to, but then, you have replication factor and
consistency level, so you can not be really sure that it actually tests
ability of the given host to write values.

So we ended up thinking that the best approach is to connect to each
individual host, read some system keyspace (which might be on a
different disk drive...), which should be local, and then check several
JMX values that could indicate an error + JVM statitics (full heap, gc
overhead). Moreover, we will more monitor our applications that are
using cassandra (with mostly datastax driver) and try to get fail node
information from them.

How others do the testing?

Jirka H.






The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: C* 2.0.15 - java.lang.NegativeArraySizeException

[SEAN_R_DURITY]

In my experience, you don’t want to do streaming operations (repairs or 
bootstraps) with mixed Cassandra versions. Upgrade the ring to the new version, 
and then add nodes (or add the nodes at the current version, and then upgrade).


Sean Durity

From: Aiman Parvaiz [mailto:ai...@flipagram.com]
Sent: Tuesday, June 09, 2015 1:29 PM
To: user@cassandra.apache.org
Subject: Re: C* 2.0.15 - java.lang.NegativeArraySizeException

Quick update, saw the same error on another new node, again the node isn't 
really misbehaving uptill now.

Thanks

On Mon, Jun 8, 2015 at 9:48 PM, Aiman Parvaiz 
ai...@flipagram.commailto:ai...@flipagram.com wrote:
Hi everyone
I am running C* 2.0.9 and decided to do a rolling upgrade. Added a node of C* 
2.0.15 in the existing cluster and saw this twice:

Jun  9 02:27:20 prod-cass23.localdomain cassandra: 2015-06-09 02:27:20,658 INFO 
CompactionExecutor:4 CompactionTask.runMayThrow - Compacting 
[SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-37-Data.db'),
 
SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-40-Data.db'),
 
SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-42-Data.db'),
 
SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-38-Data.db'),
 
SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-39-Data.db'),
 
SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-44-Data.db')]



Jun  9 02:27:20 prod-cass23.localdomain cassandra: 2015-06-09 02:27:20,669 
ERROR CompactionExecutor:4 CassandraDaemon.uncaughtException - Exception in 
thread Thread[CompactionExecutor:4,1,main]
Jun  9 02:27:20 prod-cass23.localdomain java.lang.NegativeArraySizeException
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:335)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:462)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:448)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:432)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.io.sstable.SSTableReader.getAncestors(SSTableReader.java:1366)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.io.sstable.SSTableMetadata.createCollector(SSTableMetadata.java:134)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.db.compaction.CompactionTask.createCompactionWriter(CompactionTask.java:316)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:162)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
Jun  9 02:27:20 prod-cass23.localdomain at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
Jun  9 02:27:20 prod-cass23.localdomain at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
Jun  9 02:27:20 prod-cass23.localdomain at 
java.util.concurrent.FutureTask.run(FutureTask.java:262)
Jun  9 02:27:20 prod-cass23.localdomain at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Jun  9 02:27:20 prod-cass23.localdomain at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Jun  9 02:27:20 prod-cass23.localdomain at 
java.lang.Thread.run(Thread.java:745)
Jun  9 02:27:47 prod-cass23.localdomain cassandra: 2015-06-09 02:27:47,725 INFO 
main StorageService.setMode - JOINING: Starting to bootstrap...

As you can see this happened first time even before Joining. Second occasion 
stack trace:

Jun  9 02:32:15 prod-cass23.localdomain cassandra: 2015-06-09 02:32:15,097 
ERROR CompactionExecutor:6 CassandraDaemon.uncaughtException - Exception in 
thread Thread[CompactionExecutor:6,1,main]
Jun  9 02:32:15 prod-cass23.localdomain java.lang.NegativeArraySizeException
Jun  9 02:32:15 prod-cass23.localdomain at 
org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:335)
Jun  9 02:32:15 prod-cass23.localdomain at 

RE: nodetool repair

[SEAN_R_DURITY]

It seems to me that running repair on any given node may also induce repairs to 
related replica nodes. For example, if I run repair on node A and node B has 
some replicas, data might stream from A to B (assuming A has newer/more data). 
Now, that does NOT mean that node B will be fully repaired. You still need to 
run repair -pr on all nodes before gc_grace_seconds.

You can run repairs on multiple nodes at the same time. However, you might end 
up with a large amount of streaming, if many repairs are needed. So, you should 
be aware of a performance impact.

I run weekly repairs on one node at a time, if possible. On, larger rings, 
though, I run repairs on multiple nodes staggered by a few hours. Once your 
routine maintenance is established, repairs will not run for very long. But, if 
you have a large ring that hasn’t been repaired, those first repairs may take 
days (but should get faster as you get further through the ring).


Sean Durity

From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: Friday, June 19, 2015 3:56 AM
To: user@cassandra.apache.org
Subject: Re: nodetool repair

Hi,

This is not necessarily true. Repair will induce compactions only if you have 
entropy in your cluster. If not it will just read your data to compare all the 
replica of each piece of data (using indeed cpu and disk IO).

If there is some data missing it will repair it. Though, due to merkle tree 
size, you will generally stream more data than just the data needed. To limit 
this downside and the compactions amount, use range repairs -- 
http://www.datastax.com/dev/blog/advanced-repair-techniques.

About tombstones, they will be evicted only after gc_grace_period and only if 
all the parts of the row are part of the compaction.

C*heers,

Alain

2015-06-19 9:08 GMT+02:00 arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com:
Yes compactions will remove tombstones

On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Perfect thank you.
So making a weekly nodetool repair -pr”  on all nodes one after the other will 
repair my cluster. That is great.

If it does a compaction, does it mean that it would also clean up my tombstone 
from my LeveledCompactionStrategy tables at the same time?

Thanks for your help.

On 19 Jun 2015, at 07:56 , arun sirimalla 
arunsi...@gmail.commailto:arunsi...@gmail.com wrote:

Hi Jean,

Running nodetool repair on a node will repair only that node in the cluster. It 
is recommended to run nodetool repair on one node at a time.

Few things to keep in mind while running repair
   1. Running repair will trigger compactions
   2. Increase in CPU utilization.


Run node tool repair with -pr option, so that it will repair only the range 
that node is responsible for.

On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Thanks Jonathan.

But I need to know the following:

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?
If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

Kind regards

On 18 Jun 2015, at 19:19 , Jonathan Haddad 
j...@jonhaddad.commailto:j...@jonhaddad.com wrote:

If you're using DSE, you can schedule it automatically using the repair 
service.  If you're open source, check out Spotify cassandra reaper, it'll 
manage it for you.

https://github.com/spotify/cassandra-reaper



On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay 
jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com 
wrote:
Hi,

I want to make on a regular base repairs on my cluster as suggested by the 
documentation.
I want to do this in a way that the cluster is still responding to read 
requests.
So I understand that I should not use the -par switch for that as it will do 
the repair in parallel and consume all available resources.

If you issue a “nodetool repair” on one node will it repair all the nodes in 
the cluster or only the one on which we issue the command?

If it repairs only one node, do I have to wait that the nodetool repair ends, 
and only then issue another “nodetool repair” on the next node?

If we had down time periods I would issue a nodetool -par, but we don’t have 
down time periods.

Sorry for the stupid questions.
Thanks for your help.




--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html





--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the 

RE: C* 2.0.15 - java.lang.NegativeArraySizeException

[SEAN_R_DURITY]

Right. I have had very few problems running mixed versions for normal 
operations (as long as the versions are “close”). During upgrades, I turn off 
repairs. Adding/replacing nodes is very infrequent for me, so not much of a 
consideration. We upgrade as quickly as we can, however, to protect against 
issues. Our clusters are not huge, though.

NOTE: the above applies to version 1.0.8, 1.2.x and 2.0.x. The 1.1.x versions 
are painful and problematic.

Sean Durity

From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: Tuesday, June 09, 2015 6:19 PM
To: user@cassandra.apache.org
Subject: Re: C* 2.0.15 - java.lang.NegativeArraySizeException

On Tue, Jun 9, 2015 at 2:35 PM, Aiman Parvaiz 
ai...@flipagram.commailto:ai...@flipagram.com wrote:
Thanks Sean, in this scenario also I would end up running 2 versions of 
Cassandra as I am planning to do a rolling upgrade and hence zero downtime. 
Upgrading in place one node at a time would lead to running 2 versions, please 
let me know if I am missing something here.

Running a cluster with nodes of two different versions during a rolling 
upgrade, for the duration of the upgrade, is (mostly) supported [1].

Modifying cluster topology (adding or removing or replacing nodes) during such 
an upgrade is not.

It is a fair statement that with very large clusters and very slow 
upgradesstables, the ability of any operator to operate in this manner 
approaches not-possible. I don't know how people with truly huge clusters deal 
with this race.

=Rob
 [1] some operations are not supported possible with some combinations of 
versions. for example one cannot repair in some cases.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Cassandra 2.2, 3.0, and beyond

[SEAN_R_DURITY]

With 3.0, what happens to existing Thrift-based tables (with dynamic column 
names, etc.)?

Sean Durity

From: Jonathan Ellis [mailto:jbel...@gmail.com]
Sent: Wednesday, June 10, 2015 10:30 AM
To: user
Subject: Cassandra 2.2, 3.0, and beyond


As you know, we've split our post-2.1 release into two pieces, with 2.2 to be 
released in July (rc1 out Mondayhttp://cassandra.apache.org/download/) and 
3.0 in September.


2.2 will include Windows support, commitlog 
compressionhttps://issues.apache.org/jira/browse/CASSANDRA-6809, JSON 
supporthttps://issues.apache.org/jira/browse/CASSANDRA-7970, role-based 
authorizationhttp://www.datastax.com/dev/blog/role-based-access-control-in-cassandra,
 bootstrap-aware leveled 
compactionhttps://issues.apache.org/jira/browse/CASSANDRA-7460, and 
user-defined 
functionshttp://christopher-batey.blogspot.com/2015/05/cassandra-aggregates-min-max-avg-group.html.


3.0 will include a major storage engine 
rewritehttps://issues.apache.org/jira/browse/CASSANDRA-8099 and materialized 
viewshttps://issues.apache.org/jira/browse/CASSANDRA-6477.


We're splitting things up this way because we don't want to block the features 
that are already complete while waiting for 8099 (the new storage engine).  
Releasing them now as 2.2 reduces the risk for users (2.2 has a lot in common 
with 2.1) and allows us to stabilize that independently of the upheaval from 
8099.


After 3.0, we'll take this even further: we will release 3.x versions monthly.  
Even releases will include both bugfixes and new features; odd releases will be 
bugfix-only.  You may have heard this referred to as tick-tock releases, 
after Intel's policy of changing process and architecture 
independentlyhttp://www.intel.com/content/www/us/en/silicon-innovations/intel-tick-tock-model-general.html.


The primary goal is to improve release quality.  Our current major dot zero 
releases require another five or six months to make them stable enough for 
production.  This is directly related to how we pile features in for 9 to 12 
months and release all at once.  The interactions between the new features are 
complex and not always obvious.  2.1 was no exception, despite DataStax hiring 
a full time test engineering team specifically for Apache Cassandra.


We need to try something different.  Tick-tock releases will dramatically 
reduce the number of features in each version, which will necessarily improve 
our ability to quickly track down any regressions.  And pausing every other 
month to focus on bug fixes will help ensure that we don't accumulate issues 
faster than we can fix them.


Tick-tock will also prevent situations like the one we are in now with 8099 
delaying everything else.  Users will get to test new features almost 
immediately.


To get there, we are investing significant effort in making trunk always 
releasable, with the goal that each release, or at least each odd-numbered 
bugfix release, should be usable in production.  We’ve extended our continuous 
integration server to make it easy for contributors to run tests against 
feature 
brancheshttp://www.datastax.com/dev/blog/cassandra-testing-improvements-for-developer-convenience-and-confidence
 before merging to trunk and we’re working on more test 
infrastructurehttps://docs.google.com/document/d/1Seku0vPwChbnH3uYYxon0UO-b6LDtSqluZiH--sWWi0
 and 
procedureshttps://docs.google.com/document/d/1ptr47UQ56N80jqL_O6AlE67b0STyn_cVp2k5DTv-OMc
 to improve release quality.  You can see how this is coming along in our May 
retrospectivehttps://docs.google.com/document/d/1GtuYRocdr9luNdwmm8wE84uC5Wr6TvewFbQtqoAFVeU/edit.


We are also extending our backwards compatibility policy to cover all 3.x 
releases: you will be able to upgrade seamlessly from 3.1 to 3.7, for instance, 
including cross-version repair.  We will not introduce any extra upgrade 
requirements or remove deprecated features until 4.0, no sooner than a year 
after 3.0.


Under normal conditions, we will not release 3.x.y stability releases for x  
0.  That is, we will have a traditional 3.0.y stability series, but the 
odd-numbered bugfix-only releases will fill that role for the tick-tock series 
-- recognizing that occasionally we will need to be flexible enough to release 
an emergency fix in the case of a critical bug or security vulnerability.


We do recognize that it will take some time for tick-tock releases to deliver 
production-level stability, which is why we will continue to deliver 2.2.y and 
3.0.y bugfix releases.  (But if we do demonstrate that tick-tock can deliver 
the stability we want, there will be no need for a 4.0.y bugfix series, only 
4.x tick-tock.)

After 2.2.0 is released, 2.0 will reach end-of-life as planned.  After 3.0.0 is 
released, 2.1 will also reach end of life.  This is earlier than expected, but 
2.2 will be very close to as stable as 2.1 and users will be well served by 
upgrading.  We will maintain the 2.2 stability series until 4.0 is released, 
and 3.0 for 

RE: Support for ad-hoc query

[SEAN_R_DURITY]

I will note here that the limitations on ad-hoc querying (and aggregates) make 
it much more difficult to deal with data quality problems, QA testing, and 
similar efforts, especially where people are used to a more relational, ad-hoc 
model. We have often had to extract data from Cassandra to Hadoop for querying 
by hive.

Example: “We found a few records with incorrect data. How many more records 
like that are out there?”


Sean Durity

From: Peter Lin [mailto:wool...@gmail.com]
Sent: Wednesday, June 10, 2015 8:17 AM
To: user@cassandra.apache.org
Subject: Re: Support for ad-hoc query


I'll second Jack's detailed response and add that you really should do some 
discovery to figure out what kinds of queries you may need to support.
It might not be possible and often that is the case, but it's worth while to 
ask the end users what kind of reports they need to run. Allowing arbitrary 
ad-hoc queries is a known anti-pattern for cassandra. If the system needs to 
query multiple cf to derive/calculate some result, using Cassandra alone isn't 
going to do it. You'll need some other system to give you better query 
capabilities like Hive.
If you need data warehouse like features, look at http://www.kylin.io/ . They 
are doing some interesting things.
peter

On Wed, Jun 10, 2015 at 7:58 AM, Jack Krupansky 
jack.krupan...@gmail.commailto:jack.krupan...@gmail.com wrote:
Knowing your queries in advance is a hard-core requirement for effective 
deployment of Cassandra. Ad hoc queries are a very clear anti-pattern for 
Cassandra. DSE Search does provide support for advanced, complex, and ad hoc 
queries. Stratio and TupleJump Stargate can also be used.

Back to the question of what you mean by ad hoc queries:

1. Do you expect real-time results, like sub-second, or are these long-running 
queries that might take seconds, 10 seconds or more, or even minutes to run?
2. Will they be very rare or quite frequent - how much load do you expect them 
to place on the cluster?
3. How complex do you expect them to be - how many clauses and operators?
4. What is their net cardinality - are they selecting just a few rows or many 
rows?
5. Do they have individual query clauses that select many rows even if the net 
combination of all select clauses is not so many rows?

The requirement to perform advanced, complex, and ad hoc queries using DSE 
Search or the other techniques will almost certainly require that you use 
moderately more capable hardware, especially more RAM, for each node, and 
probably more nodes as well to reduce the row count per node since ad hoc 
queries will tend to be compute-intensive based on number of rows on the node.

Yes, it can be done. No, it is not free or cheap. And, no, it does not come out 
of the box for a non-DSE Cassandra release. And, yes, you must address this 
requirement before deployment, not after deployment.


-- Jack Krupansky

On Wed, Jun 10, 2015 at 1:18 AM, Srinivasa T N 
seen...@gmail.commailto:seen...@gmail.com wrote:
Thanks guys for the inputs.
By ad-hoc queries I mean that I don't know the queries during cf design time.  
The data may be from single cf or multiple cf.  (This feature maybe required if 
I want to do analysis on the data stored in cassandra, do you have any better 
ideas)?
Regards,
Seenu.

On Tue, Jun 9, 2015 at 5:57 PM, Peter Lin 
wool...@gmail.commailto:wool...@gmail.com wrote:

what do you mean by ad-hoc queries?
Do you mean simple queries against a single column family aka table?
Or do you mean MDX style queries that looks at multiple tables?
if it's MDX style queries, many people extract data from Cassandra into a data 
warehouse that support multi-dimensional cubes. This works well when the 
extracted data is a small subset and fits neatly in a data warehouse.
As others have stated, Cassandra isn't great at ad-hoc. For MDX style queries, 
Cassandra wasn't designed for it. One thing we've done for our own project is 
to combine solr with our own fuzzy index to make ad-hoc queries against a 
single table more friendly.


On Tue, Jun 9, 2015 at 2:38 AM, Srinivasa T N 
seen...@gmail.commailto:seen...@gmail.com wrote:
Hi All,
   I have an web application running with my backend data stored in cassandra.  
Now I want to do some analysis on the data stored which requires some ad-hoc 
queries fired on cassandra.  How can I do the same?
Regards,
Seenu.







The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home 

RE: Periodic Anti-Entropy repair

[SEAN_R_DURITY]

We use nodetool repair -pr on each node through the week. Basically I have a 
cron job that checks a schedule on each host to see “should I run repair 
today?” If yes, it kicks off repair.


Sean Durity

From: Tiwari, Tarun [mailto:tarun.tiw...@kronos.com]
Sent: Monday, May 25, 2015 12:41 AM
To: user@cassandra.apache.org
Subject: RE: Periodic Anti-Entropy repair

We have encountered issues of very long running nodetool repair when we ran it 
node by node on really large dataset. It even kept on running for a week in 
some cases.
IMO the strategy you are choosing of repairing nodes by –st and –et is good one 
and does the same work in small increments logs of which can be analyzed easily.

In addition my suggestion would be to use –h option to connect to the node from 
outside, and take care of the fact that node tool ring will give even –ve token 
ranges in the ‘for’ loop. You can go from -2^63 to first ring value, then from 
(there+1) to next token value. Better not use i+=2 because token values are not 
necessarily even numbers.

Regards,
Tarun

From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Sunday, May 24, 2015 6:31 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: Periodic Anti-Entropy repair

You should use nodetool repair -pr on every node to make sure that each range 
is repaired only once.


Thanks
Anuj Wadehra

Sent from Yahoo Mail on 
Androidhttps://overview.mail.yahoo.com/mobile/?.src=Android


From:Brice Argenson bargen...@gmail.commailto:bargen...@gmail.com
Date:Sat, 23 May, 2015 at 12:31 am
Subject:Periodic Anti-Entropy repair
Hi everyone,

We are currently migrating from DSE to Apache Cassandra and we would like to 
put in place an automatic and periodic nodetool repair execution to replace the 
one executed by OpsCenter.

I wanted to create a script / service that would run something like that:

token_rings = `nodetool ring | awk '{print $8}’`
for(int i = 0; i  token_rings.length; i += 2) {
   `nodetool repair -st token_rings[i] -et token_rings[i+1]`
}

That script / service would run every week (our GCGrace is 10 days) and would 
repair all the ranges of the ring one by one.

I also looked a bit on Google and I found that script: 
https://github.com/BrianGallew/cassandra_range_repair
It seems to do something equivalent but it also seems to run the repair node by 
node instead of the complete ring.
From my understanding, that would mean that the script has to be run for every 
node of the cluster and that all token ranges would be repair as many time as 
the number of replicas containing it.


Is there something I misunderstand?
Which approach is better?
How do you handle your Periodic Anti-Entropy Repairs?


Thanks a lot!






The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: How to measure disk space used by a keyspace?

[SEAN_R_DURITY]

That’s ok for a single node, but to answer the question, “how big is my table 
across the cluster?” it would be much better if the cluster could provide an 
answer.

Sean Durity

From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Monday, June 29, 2015 8:15 AM
To: user
Subject: Re: How to measure disk space used by a keyspace?

If you're looking to measure actual disk space, I'd use the du command, 
assuming you're on a linux: http://linuxconfig.org/du-1-manual-page

On Mon, Jun 29, 2015 at 2:26 AM shahab 
shahab.mok...@gmail.commailto:shahab.mok...@gmail.com wrote:
Hi,

Probably this question has been already asked in the mailing list, but I 
couldn't find it.

The question is how to measure disk-space used by a keyspace, column family 
wise, excluding snapshots?

best,
/Shahab



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Multiple cassandra instances per physical node

[SEAN_R_DURITY]

We run 2 nodes (from 2 different rings) on the same physical host. One is for a 
random ring; the other is byteordered to support some alphabetic range queries. 
Each instance has its own binary install, data directory and ports. One 
limitation - with one install of OpsCenter agent, it can only connect to one of 
the rings. We haven’t tried two OpsCenter agent installs, yet.


Sean Durity

From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Thursday, May 21, 2015 5:26 PM
To: user@cassandra.apache.org
Subject: Re: Multiple cassandra instances per physical node

Yep, that would be one way to handle it.
On Thu, May 21, 2015 at 2:07 PM Dan Kinder 
dkin...@turnitin.commailto:dkin...@turnitin.com wrote:
@James Rothering yeah I was thinking of container in a broad sense: either full 
virtual machines, docker containers, straight LXC, or whatever else would allow 
the Cassandra nodes to have their own IPs and bind to default ports.

@Jonathan Haddad thanks for the blog post. To ensure the same host does not 
replicate its own data, would I basically need the nodes on a single host to be 
labeled as one rack? (Assuming I use vnodes)

On Thu, May 21, 2015 at 1:02 PM, Sebastian Estevez 
sebastian.este...@datastax.commailto:sebastian.este...@datastax.com wrote:
JBOD -- just a bunch of disks, no raid.


All the best,



[Image removed by sender. datastax_logo.png]http://www.datastax.com/

Sebastián Estévez

Solutions Architect | 954 905 8615tel:954%20905%208615 | 
sebastian.este...@datastax.commailto:sebastian.este...@datastax.com

[Image removed by sender. 
linkedin.png]https://www.linkedin.com/company/datastax[Image removed by 
sender. facebook.png]https://www.facebook.com/datastax[Image removed by 
sender. twitter.png]https://twitter.com/datastax[Image removed by sender. 
g+.png]https://plus.google.com/+Datastax/about[Image removed by 
sender.]http://feeds.feedburner.com/datastax

[Image removed by sender.]http://cassandrasummit-datastax.com/

DataStax is the fastest, most scalable distributed database technology, 
delivering Apache Cassandra to the world’s most innovative enterprises. 
Datastax is built to be agile, always-on, and predictably scalable to any size. 
With more than 500 customers in 45 countries, DataStax is the database 
technology and transactional backbone of choice for the worlds most innovative 
companies such as Netflix, Adobe, Intuit, and eBay.

On Thu, May 21, 2015 at 4:00 PM, James Rothering 
jrother...@codojo.memailto:jrother...@codojo.me wrote:
Hmmm ... Not familiar with JBOD. Is that just RAID-0?

Also ... wrt  the container talk, is that a Docker container you're talking 
about?



On Thu, May 21, 2015 at 12:48 PM, Jonathan Haddad 
j...@jonhaddad.commailto:j...@jonhaddad.com wrote:
If you run it in a container with dedicated IPs it'll work just fine.  Just be 
sure you aren't using the same machine to replicate it's own data.

On Thu, May 21, 2015 at 12:43 PM Manoj Khangaonkar 
khangaon...@gmail.commailto:khangaon...@gmail.com wrote:
+1.
I agree we need to be able to run multiple server instances on one physical 
machine. This is especially necessary in development and test environments 
where one is experimenting and needs a cluster, but do not have access to 
multiple physical machines.
If you google , you  can find a few blogs that talk about how to do this.

But it is less than ideal. We need to be able to do it by changing ports in 
cassandra.yaml. ( The way it is done easily with Hadoop or Apache Kafka or 
Redis and many other distributed systems)

regards



On Thu, May 21, 2015 at 10:32 AM, Dan Kinder 
dkin...@turnitin.commailto:dkin...@turnitin.com wrote:
Hi, I'd just like some clarity and advice regarding running multiple cassandra 
instances on a single large machine (big JBOD array, plenty of CPU/RAM).

First, I am aware this was not Cassandra's original design, and doing this 
seems to unreasonably go against the commodity hardware intentions of 
Cassandra's design. In general it seems to be recommended against (at least as 
far as I've heard from @Rob Coli and others).

However maybe this term commodity is changing... my hardware/ops team argues 
that due to cooling, power, and other datacenter costs, having slightly larger 
nodes (=32G RAM, =24 CPU, =8 disks JBOD) is actually a better price point. 
Now, I am not a hardware guy, so if this is not actually true I'd love to hear 
why, otherwise I pretty much need to take them at their word.

Now, Cassandra features seemed to have improved such that JBOD works fairly 
well, but especially with memory/GC this seems to be reaching its limit. One 
Cassandra instance can only scale up so much.

So my question is: suppose I take a 12 disk JBOD and run 2 Cassandra nodes 
(each with 5 data disks, 1 commit log disk) and either give each its own 
container  IP or change the listen ports. Will this work? What are the risks? 
Will/should Cassandra support this better in the future?


--
http://khangaonkar.blogspot.com/





--
Dan 

RE: Can I run upgrade sstables on many nodes on one time

[SEAN_R_DURITY]

Yes, you should run upgradesstables on each node. If the sstable structure has 
changed, you will need this completed before you can do streaming operations 
like repairs or adding nodes.

As for running in parallel, that should be fine. It is a “within the node” 
operation that pounds I/O (but is capped by compaction threshold). You need to 
look at the level of activity from normal operations, though. If Cassandra is 
running without much stress/sweat, go ahead and run 2 at once. (Conservatively, 
that’s all I would do on 6 nodes.) If the cluster is inactive, let it fly on 
all nodes.


Sean Durity
Lead Cassandra Admin, Big Data Team

From: Ola Nowak [mailto:ola.nowa...@gmail.com]
Sent: Thursday, August 13, 2015 5:30 AM
To: user@cassandra.apache.org
Subject: Can I run upgrade sstables on many nodes on one time

Hi all,
I'm trying to update my 6 node cluster from 2.0.11 to 2.1.8.
I'm following this update procedure: 
http://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgradeCassandraDetails.html
 and the point 8 says: If you are upgrading from a major version (for example, 
from Cassandra 1.2 to 2.0) or a major point release (for example, from 
Cassandra 2.0 to 2.1), upgrade the SSTables on each node.
$ nodetool upgradesstables
As far as I understood it correctly I should run nodetool upgradesstables on 
every node after upgrading the version on each node. Is that right?
As it is a really time consuming operation I wonder if I could run 
upgradesstables on multiple nodes at one time ( parallelly)?
Regards,
Ola




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: limit the size of data type LIST

[SEAN_R_DURITY]

This sounds like something you do on the client side BEFORE you insert. Or are 
you wanting to limit the size of the list coming out to the client?


Sean Durity
Lead Cassandra Admin, Big Data Team

From: yuankui [mailto:kui.y...@fraudmetrix.cn]
Sent: Thursday, August 13, 2015 9:06 AM
To: user@cassandra.apache.org
Subject: limit the size of data type LIST

hi, friends

I am design a message history table

CREATE TABLE message_history (
user_name text PRIMARY KEY,
time timestamp,
message_details listtext,
);

so that I can query a user's message via primary key `user_name` at once.

but the item in `message_details` list may be very long so that I want to limit 
the list size of the message_details list.

is there a way to solve this?

like a redis operation `LTRIM` - 
http://redis.readthedocs.org/en/latest/list/ltrim.html





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Manual Indexing With Buckets

[SEAN_R_DURITY]

It is a bit hard to follow. Perhaps you could include your proposed schema 
(annotated with your size predictions) to spur more discussion. To me, it 
sounds a bit convoluted. Why is a batch so big (up to 100 million rows)? Is a 
row in the primary only associated with one batch?


Sean Durity - Cassandra Admin, Big Data Team
To engage the team, create a 
requesthttps://portal.homedepot.com/sites/bigdata/SitePages/Big%20Data%20Engagement%20Request.aspx

From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Friday, July 24, 2015 3:57 AM
To: user@cassandra.apache.org
Subject: Re: Manual Indexing With Buckets

Can anyone take this one?

Thanks
Anuj

Sent from Yahoo Mail on 
Androidhttps://overview.mail.yahoo.com/mobile/?.src=Android


From:Anuj Wadehra anujw_2...@yahoo.co.inmailto:anujw_2...@yahoo.co.in
Date:Thu, 23 Jul, 2015 at 10:57 pm
Subject:Manual Indexing With Buckets
We have a primary table and we need search capability by batchid column. So we 
are creating a manual index for search by batch id. We are using buckets to 
restrict a row size in batch id index table to 50mb. As batch size may vary 
drastically ( ie one batch id may be associated to 100k row keys in primary 
table while other may be associated with 100million row keys), we are creating 
a metadata table to track the approximate data while insertions for a batch in 
primary table, so that batch id index table has dynamic no of buckets/rows. As 
more data is inserted for a batch in primary table, new set of 10 buckets are 
added. At any point in time, clients will write to latest 10 buckets created 
for a batch od index in round robin  to avoid hotspots.

Comments required on the following:
1. I want to know any suggestios on above design?

2. Whats the best approach for updating/deleting from index table. When a row 
is manually purged from primary table, we dont know where that row key exists 
in x number of buckets created for its batch id?

Thanks
Anuj

Sent from Yahoo Mail on 
Androidhttps://overview.mail.yahoo.com/mobile/?.src=Android







The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Need company to support Cassandra on Windows

[SEAN_R_DURITY]

I highly recommend DataStax support, although we have not done Windows. They 
are in a growth phase, so I expect there will be growing pains with support. 
However, they have some top notch folks in place.

Sean Durity – Lead Cassandra Admin
From: Troy Collinsworth [mailto:troycollinswo...@gmail.com]
Sent: Tuesday, October 27, 2015 9:58 AM
To: user@cassandra.apache.org
Subject: Need company to support Cassandra on Windows

Searching for a well established company that can provide consulting and 
operations support for private multi-dc production Cassandra cluster on Windows 
OS. New project. OS is hosting mandate.

Troy Collinsworth
585-576-8761



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Is replication possible with already existing data?

[SEAN_R_DURITY]

How many nodes are you planning to add? How many replicas do you want? In 
general, there shouldn't be a problem adding nodes and then altering the 
keyspace to change replication. You will want to run repairs to stream the data 
to the new replicas. You shouldn't need downtime or data migration -- this is 
the beauty of Cassandra.


Sean Durity – Lead Cassandra Admin

-Original Message-
From: Ajay Garg [mailto:ajaygargn...@gmail.com]
Sent: Wednesday, October 07, 2015 11:56 AM
To: user@cassandra.apache.org
Subject: Is replication possible with already existing data?

Hi All.

We have a scenario, where till now we had been using a plain, simple single 
node, with the keyspace created using ::

CREATE KEYSPACE our_db WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '1'}  AND durable_writes = true;


We now plan to introduce replication (in the true sense) in our scheme of 
things, but cannot afford to lose any data.
We, however can take a bit of downtime, and do any data-migration if required 
(we have already done data-migration once in the past, when we moved our plain, 
simple single node from one physical machine to another).


So,

a)
Is it possible at all to introduce replication in our scenario?
If yes, what needs to be done to NOT LOSE our current existing data?

b)
Also, will "NetworkTopologyStrategy" work in our scenario (since 
NetworkTopologyStrategy seems to be more robust)?


Brief pointers to above will give huge confidence-boosts in our endeavours.


Thanks and Regards,
Ajay



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Why can't nodetool status include a hostname?

[SEAN_R_DURITY]

I ended up writing some of my own utilities and aliases to make output more 
useful for me (and reduce some typing, too). Resolving host names was a big one 
for me, too. Ip addresses are almost useless. Up time in seconds is useless.

The –r in nodetool is a nice addition, but I like the short host name instead.

hostname:/home/cassuser> cinfo
DSE Version: 4.7.0
Cassandra Ver  : 2.1.5.469
Gossip active  : true
Thrift active  : true
Native Transport active: true
Load   : 2.67 GB
Up since   : Sun Sep 13 00:36:50 EDT 2015
Heap Memory (MB)   : 3645.74 / 7987.25
Off Heap Memory (MB)   : 203.93
Heap Used %: 45.64
Thrift Conns   : 0
CQL Conns  : 12
Topology   : DC1 : RAC1

hostname:/home/cassuser> cstatus
Datacenter: DC1
===
Status=Up/Down
/ State=Normal/Leaving/Joining/Moving
--  Address Load   Tokens  OwnsHost ID  
 Rack
UN  cplinpys   8.47 GB256 ?   
f498c0f9-0041-404c-979d-d1269c6a2287  RAC1
UN  cplinpyr   2.67 GB256 ?   
397546c2-e229-482e-aa50-de367ab6add8  RAC1
UN  cplinpyt   2.17 GB256 ?   
f61da10c-c2c6-4a5a-8fdc-d2693f2239bc  RAC1

Sean Durity – Lead Cassandra Admin

From: Gene [mailto:gh5...@gmail.com]
Sent: Thursday, October 08, 2015 12:43 PM
To: user@cassandra.apache.org
Subject: Re: Why can't nodetool status include a hostname?

Yeah, -r or --resolve-ip is what you're looking for.

Cassandra's nodetool command is kind of wonky.  Inconsistent across functions 
(e.g. sometimes 'keyspace.columnfamily' other times 'keyspace columnfamily', 
pay attention to the character between the items), doesn't resolve IPs by 
default (while standard linux commands require you to pass something like -n to 
not resolve names), so on and so forth.

When in doubt run nodetool without specifying a command and it'll list all of 
the available options (another example of wonkiness, the 'help' argument is not 
listed in this output)

-Gene

On Thu, Oct 8, 2015 at 7:01 AM, Paulo Motta 
> wrote:
Have you tried using the -r or --resolve-ip option?

2015-10-07 19:59 GMT-07:00 Kevin Burton 
>:
I find it really frustrating that nodetool status doesn't include a hostname

Makes it harder to track down problems.

I realize it PRIMARILY uses the IP but perhaps cassandra.yml can include an 
optional 'hostname' parameter that can be set by the user.  OR have the box 
itself include the hostname in gossip when it starts up.

I realize that hostname wouldn't be authoritative and that the IP must still be 
shown but we could add another column for the hostname.

--
We’re hiring if you know of any awesome Java Devops or Linux Operations 
Engineers!

Founder/CEO Spinn3r.com
Location: San Francisco, CA
blog: http://burtonator.wordpress.com
… or check out my Google+ 
profile





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: DC's versions compatibility

[SEAN_R_DURITY]

You CAN run mixed versions in your cluster for a short time when you are 
upgrading. However, you will most likely not be able to run repairs or any 
other streaming operations (adding nodes, adding a DC, etc.) while in this 
state.

The advice below is correct. Upgrade your current ring to the version you need; 
then add the DC at the same version. This is the only way to be sure that the 
streaming required will work.


Sean Durity – Lead Cassandra Admin

From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Monday, September 28, 2015 9:34 AM
To: user@cassandra.apache.org
Subject: Re: DC's versions compatibility

No, they won't.  Always run the same version across your cluster.

On Mon, Sep 28, 2015 at 5:29 AM Carlos Alonso 
> wrote:
Hi guys.

I have a very old cassandra cluster 1.2.19 and I'm looking to add a new 
datacenter to it for analytics purposes in a newer version, let's say 2.1.8. 
Will those DC's communicate properly?

Regards

Carlos Alonso | Software Engineer | @calonso



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Cassandra certification

[SEAN_R_DURITY]

I took (and passed) the Admin exam at the Cassandra Summit. Tushar does hit the 
main points in his article. One of the conditions of the test is to not reveal 
details on the test, so we shouldn’t reveal anything very substantial. I will 
say that the exam was about the same difficulty level as Informix certification 
that I took several years ago. It is not simple. You have to know the material 
AND be careful about the test’s format.

Sean Durity – Lead Cassandra Admin
Big DATA Team

From: Tushar Agrawal [mailto:agrawal.tus...@gmail.com]
Sent: Thursday, October 01, 2015 12:01 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra certification

Check this out:
Get Trained, Get Certified, Get Better Paid

https://www.linkedin.com/pulse/get-trained-certified-better-paid-tushar-agrawal

Thanks,
Tushar
(Sent from iPhone)

On Oct 1, 2015, at 9:00 AM, Fernandez Gomara, Ruben (CCI-Atlanta) 
> wrote:
Hi,
Did anybody took the Cassandra (Administrator) certification on the Summit? Any 
feedback? How was the test?
Thanks
Ruben




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Issues on upgrading from 2.2.3 to 3.0

[SEAN_R_DURITY]

Is there a JIRA for the discussion of dropping PropertyFileSnitch? That is all 
that we use, and it is much easier to package and distribute than 
GossipingPropertyFileSnitch. I would vote against dropping the more useful 
PropertyFileSnitch.


Sean Durity – Lead Cassandra Admin

From: Paulo Motta [mailto:pauloricard...@gmail.com]
Sent: Thursday, December 03, 2015 7:04 AM
To: user@cassandra.apache.org
Subject: Re: Issues on upgrading from 2.2.3 to 3.0

You can migrate from the RackInferingSnitch to PropertiesFileSnitch by 
populating the cassandra-topology.properties with the same rack/dc assignments 
of the previous snitch (what you can't change is the assignment, but you can 
change snitches if you maintain the same assignments as before).
For example, if you were using using the RackInferingSnitch before, an 
equivalent configuration on the PropertiesFileSnitch is:
IP=DC:RACK
175.56.12.105=56:12 (3rd octet is DC, 2nd is RACK on RackInferingSnitch)
175.56.13.200=56:13
175.54.35.197=54:35
120.54.24.101=54:24

There is a chance the PropertiesFileSnitch is deprecated in the near future, so 
it's preferable to use the GossipingPropertyFileSnitch which is also simpler to 
configure.

2015-12-02 23:30 GMT-08:00 Carlos A 
>:
Bryan, thanks for replying. I had that figured out already few days ago. The 
issue was that the snitch method was also changed on the configuration hence 
the problem.

Now, if you change from rackInferingSnitch to PropertiesFileSnitch it will not 
run as the data has to be migrated. Is that correct? Or do you have the change 
the replication class on the keyspace?

Putting back to rackInferingSnitch worked just fine.

On Wed, Dec 2, 2015 at 6:30 PM, Bryan Cheng 
> wrote:
Has your configuration changed?

This is a new check- https://issues.apache.org/jira/browse/CASSANDRA-10242. It 
seems likely either your snitch changed, your properties changed, or something 
caused Cassandra to think one of the two happened...

What's your node layout?

On Fri, Nov 27, 2015 at 6:45 PM, Carlos A 
> wrote:
Hello all,

I had 2 of my systems upgraded to 3.0 from the same previous version.

The first cluster seem to be fine.

But the second, each node starts and then fails.

On the log I have the following on all of them:

INFO  [main] 2015-11-27 19:40:21,168 ColumnFamilyStore.java:381 - Initializing 
system_schema.keyspaces
INFO  [main] 2015-11-27 19:40:21,177 ColumnFamilyStore.java:381 - Initializing 
system_schema.tables
INFO  [main] 2015-11-27 19:40:21,185 ColumnFamilyStore.java:381 - Initializing 
system_schema.columns
INFO  [main] 2015-11-27 19:40:21,192 ColumnFamilyStore.java:381 - Initializing 
system_schema.triggers
INFO  [main] 2015-11-27 19:40:21,198 ColumnFamilyStore.java:381 - Initializing 
system_schema.dropped_columns
INFO  [main] 2015-11-27 19:40:21,203 ColumnFamilyStore.java:381 - Initializing 
system_schema.views
INFO  [main] 2015-11-27 19:40:21,208 ColumnFamilyStore.java:381 - Initializing 
system_schema.types
INFO  [main] 2015-11-27 19:40:21,215 ColumnFamilyStore.java:381 - Initializing 
system_schema.functions
INFO  [main] 2015-11-27 19:40:21,220 ColumnFamilyStore.java:381 - Initializing 
system_schema.aggregates
INFO  [main] 2015-11-27 19:40:21,225 ColumnFamilyStore.java:381 - Initializing 
system_schema.indexes
ERROR [main] 2015-11-27 19:40:21,831 CassandraDaemon.java:250 - Cannot start 
node if snitch's rack differs from previous rack. Please fix the snitch or 
decommission and rebootstrap this node.

It asks to "Please fix the snitch or decommission and rebootstrap this node"

If none of the nodes can go up, how can I decommission all of them?

Doesn't make sense.

Any suggestions?

Thanks,

C.






The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Issues on upgrading from 2.2.3 to 3.0

[SEAN_R_DURITY]

Thank you! I made my comments in the JIRA.

Sean Durity – Lead Cassandra Admin
From: Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com]
Sent: Monday, December 14, 2015 3:25 PM
To: user@cassandra.apache.org
Subject: Re: Issues on upgrading from 2.2.3 to 3.0

https://issues.apache.org/jira/browse/CASSANDRA-10745

GPFS seems like it SHOULD be easier to package and distribute in most use cases…


From: "sean_r_dur...@homedepot.com"
Reply-To: "user@cassandra.apache.org"
Date: Monday, December 14, 2015 at 11:56 AM
To: "user@cassandra.apache.org"
Subject: RE: Issues on upgrading from 2.2.3 to 3.0

Is there a JIRA for the discussion of dropping PropertyFileSnitch? That is all 
that we use, and it is much easier to package and distribute than 
GossipingPropertyFileSnitch. I would vote against dropping the more useful 
PropertyFileSnitch.


Sean Durity – Lead Cassandra Admin

From: Paulo Motta [mailto:pauloricard...@gmail.com]
Sent: Thursday, December 03, 2015 7:04 AM
To: user@cassandra.apache.org
Subject: Re: Issues on upgrading from 2.2.3 to 3.0

You can migrate from the RackInferingSnitch to PropertiesFileSnitch by 
populating the cassandra-topology.properties with the same rack/dc assignments 
of the previous snitch (what you can't change is the assignment, but you can 
change snitches if you maintain the same assignments as before).
For example, if you were using using the RackInferingSnitch before, an 
equivalent configuration on the PropertiesFileSnitch is:
IP=DC:RACK
175.56.12.105=56:12 (3rd octet is DC, 2nd is RACK on RackInferingSnitch)
175.56.13.200=56:13
175.54.35.197=54:35
120.54.24.101=54:24

There is a chance the PropertiesFileSnitch is deprecated in the near future, so 
it's preferable to use the GossipingPropertyFileSnitch which is also simpler to 
configure.

2015-12-02 23:30 GMT-08:00 Carlos A 
>:
Bryan, thanks for replying. I had that figured out already few days ago. The 
issue was that the snitch method was also changed on the configuration hence 
the problem.

Now, if you change from rackInferingSnitch to PropertiesFileSnitch it will not 
run as the data has to be migrated. Is that correct? Or do you have the change 
the replication class on the keyspace?

Putting back to rackInferingSnitch worked just fine.

On Wed, Dec 2, 2015 at 6:30 PM, Bryan Cheng 
> wrote:
Has your configuration changed?

This is a new check- https://issues.apache.org/jira/browse/CASSANDRA-10242. It 
seems likely either your snitch changed, your properties changed, or something 
caused Cassandra to think one of the two happened...

What's your node layout?

On Fri, Nov 27, 2015 at 6:45 PM, Carlos A 
> wrote:
Hello all,

I had 2 of my systems upgraded to 3.0 from the same previous version.

The first cluster seem to be fine.

But the second, each node starts and then fails.

On the log I have the following on all of them:

INFO  [main] 2015-11-27 19:40:21,168 ColumnFamilyStore.java:381 - Initializing 
system_schema.keyspaces
INFO  [main] 2015-11-27 19:40:21,177 ColumnFamilyStore.java:381 - Initializing 
system_schema.tables
INFO  [main] 2015-11-27 19:40:21,185 ColumnFamilyStore.java:381 - Initializing 
system_schema.columns
INFO  [main] 2015-11-27 19:40:21,192 ColumnFamilyStore.java:381 - Initializing 
system_schema.triggers
INFO  [main] 2015-11-27 19:40:21,198 ColumnFamilyStore.java:381 - Initializing 
system_schema.dropped_columns
INFO  [main] 2015-11-27 19:40:21,203 ColumnFamilyStore.java:381 - Initializing 
system_schema.views
INFO  [main] 2015-11-27 19:40:21,208 ColumnFamilyStore.java:381 - Initializing 
system_schema.types
INFO  [main] 2015-11-27 19:40:21,215 ColumnFamilyStore.java:381 - Initializing 
system_schema.functions
INFO  [main] 2015-11-27 19:40:21,220 ColumnFamilyStore.java:381 - Initializing 
system_schema.aggregates
INFO  [main] 2015-11-27 19:40:21,225 ColumnFamilyStore.java:381 - Initializing 
system_schema.indexes
ERROR [main] 2015-11-27 19:40:21,831 CassandraDaemon.java:250 - Cannot start 
node if snitch's rack differs from previous rack. Please fix the snitch or 
decommission and rebootstrap this node.

It asks to "Please fix the snitch or decommission and rebootstrap this node"

If none of the nodes can go up, how can I decommission all of them?

Doesn't make sense.

Any suggestions?

Thanks,

C.






The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any 

RE: Using TTL for data purge

[SEAN_R_DURITY]

If you know how long the records should last, TTL is a good way to go. Remember 
that neither TTL or deletes are right-away purge strategies. Each inserts a 
special record called a tombstone to indicate a deleted record. After 
compaction (that is after gc_grace_seconds for the table, default 10 days), the 
data will be removed and you will regain disk space.

If the data is relatively volatile and read speeds are important, you might 
look at leveled compaction, though it can keep your nodes a bit busier than 
size-tiered. (An issue with size-tiered, over time, is that the tombstoned data 
in the larger and older sstables may rarely, if ever, get compacted out.)


Sean Durity – Lead Cassandra Admin
From: jaalex.tech [mailto:jaalex.t...@gmail.com]
Sent: Tuesday, December 22, 2015 4:36 AM
To: user@cassandra.apache.org
Subject: Using TTL for data purge

Hi,

I'm looking for suggestions/caveats on using TTL as a subsitute for a manual 
data purge job.

We have few tables that hold user information - this could be guest or 
registered users, and there could be between 500K to 1M records created per day 
per table. Currently, these tables have a secondary indexed updated_date column 
which is populated on each update. However, we have been getting timeouts when 
running queries using updated_date when the number of records are high, so i 
don't think this would be a reliable option in the long term when we need to 
purge records that have not been used for the last X days.

In this scenario, is it advisable to include a high enough TTL (i.e the amount 
of time we want these to last, could be 3 to 6 months) when inserting/updating 
records?

There could be cases where the TTL may get reset after couple of days/weeks, 
when the user visits the site again.

The tables have fixed number of columns, except for one which has a clustering 
key, and may have max 10 entries per  partition key.

I need to know the overhead of having so many rows with TTL hanging around for 
a relatively longer duration (weeks/months), and the impacts it could have on 
performance/storage. If this is not a recommended approach, what would be an 
alternate design which could be used for a manual purge job, without using 
secondary indices.

We are using Cassandra 2.0.x.

Thanks,
Joseph




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Cassandra 3.1 - Aggregation query failure

[SEAN_R_DURITY]

It shouldn’t be called an aggregate. That is more like a user defined function. 
If you are correct, the term “aggregate” will lead people to do “bad things” – 
just like secondary indexes. I think the dev team needs a naming expert.


Sean Durity – Lead Cassandra Admin

From: Robert Stupp [mailto:sn...@snazy.de]
Sent: Wednesday, December 23, 2015 12:15 PM
To: user@cassandra.apache.org
Cc: dinesh.shanb...@isanasystems.com
Subject: Re: Cassandra 3.1 - Aggregation query failure

Well, the usual access goal for queries in C* is “one partition per query” - 
maybe a handful partitions in some cases.
That does not differ for aggregates since the read path is still the same.

Aggregates in C* are meant to move some computation (for example on the data in 
a time-frame materialized in a partition) to the coordinator and reduce the 
amount of data pumped through the wire.

For queries that span huge datasets, Spark is the easiest way to go.


On 23 Dec 2015, at 18:02, 
> 
> wrote:

An aggregate only within a partition? That is rather useless and shouldn’t be 
called an aggregate.

I am hoping the functionality can be used to support at least “normal” types of 
aggregates like count, sum, avg, etc.


Sean Durity – Lead Cassandra Admin

From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Monday, December 21, 2015 2:50 PM
To: user@cassandra.apache.org; 
dinesh.shanb...@isanasystems.com
Subject: Re: Cassandra 3.1 - Aggregation query failure

Even if you get this to work for now, I really recommend using a different 
tool, like Spark.  Personally I wouldn't use UDAs outside of a single partition.

On Mon, Dec 21, 2015 at 1:50 AM Dinesh Shanbhag 
> 
wrote:

Thanks for the pointers!  I edited jvm.options in
$CASSANDRA_HOME/conf/jvm.options to increase -Xms and -Xmx to 1536M.
The result is the same.

And in $CASSANDRA_HOME/logs/system.log, grep GC system.log produces this
(when jvm.options had not been changed):

INFO  [Service Thread] 2015-12-18 15:26:31,668 GCInspector.java:284 -
ConcurrentMarkSweep GC in 296ms.  CMS Old Gen: 18133664 -> 15589256;
Code Cache: 5650880 -> 8122304; Compressed Class Space: 2530064 ->
3345624; Metaspace: 21314000 -> 28040984; Par Eden Space: 7019256 ->
164070848;
INFO  [Service Thread] 2015-12-18 15:48:39,736 GCInspector.java:284 -
ConcurrentMarkSweep GC in 379ms.  CMS Old Gen: 649257416 -> 84190176;
Code Cache: 20772224 -> 20726848; Par Eden Space: 2191408 -> 52356736;
Par Survivor Space: 2378448 -> 2346840
INFO  [Service Thread] 2015-12-18 15:58:35,118 GCInspector.java:284 -
ConcurrentMarkSweep GC in 406ms.  CMS Old Gen: 648847808 -> 86954856;
Code Cache: 21182080 -> 21188032; Par Eden Space: 1815696 -> 71525744;
Par Survivor Space: 2388648 -> 2364696
INFO  [Service Thread] 2015-12-18 16:13:45,821 GCInspector.java:284 -
ConcurrentMarkSweep GC in 211ms.  CMS Old Gen: 648343768 -> 73135720;
Par Eden Space: 3224880 -> 7957464; Par Survivor Space: 2379912 -> 2414520
INFO  [Service Thread] 2015-12-18 16:32:46,419 GCInspector.java:284 -
ConcurrentMarkSweep GC in 387ms.  CMS Old Gen: 648476072 -> 6832;
Par Eden Space: 2006624 -> 64263360; Par Survivor Space: 2403792 -> 2387664
INFO  [Service Thread] 2015-12-18 16:42:38,648 GCInspector.java:284 -
ConcurrentMarkSweep GC in 365ms.  CMS Old Gen: 649126336 -> 137359384;
Code Cache: 22972224 -> 22979840; Metaspace: 41374464 -> 41375104; Par
Eden Space: 4286080 -> 154449480; Par Survivor Space: 1575440 -> 2310768
INFO  [Service Thread] 2015-12-18 16:51:57,538 GCInspector.java:284 -
ConcurrentMarkSweep GC in 322ms.  CMS Old Gen: 648338928 -> 79783856;
Par Eden Space: 2058968 -> 56931312; Par Survivor Space: 2342760 -> 2400336
INFO  [Service Thread] 2015-12-18 17:02:49,543 GCInspector.java:284 -
ConcurrentMarkSweep GC in 212ms.  CMS Old Gen: 648702008 -> 122954344;
Par Eden Space: 3269032 -> 61433328; Par Survivor Space: 2395824 -> 3448760
INFO  [Service Thread] 2015-12-18 17:11:54,090 GCInspector.java:284 -
ConcurrentMarkSweep GC in 306ms.  CMS Old Gen: 648748576 -> 70965096;
Par Eden Space: 2174840 -> 27074432; Par Survivor Space: 2365992 -> 2373984
INFO  [Service Thread] 2015-12-18 17:22:28,949 GCInspector.java:284 -
ConcurrentMarkSweep GC in 350ms.  CMS Old Gen: 648243024 -> 90897272;
Par Eden Space: 2150168 -> 43487192; Par Survivor Space: 2401872 -> 2410728


After modifying jvm.options to increase -Xms & -Xmx (to 1536M):

INFO  [Service Thread] 2015-12-21 11:39:24,918 GCInspector.java:284 -
ConcurrentMarkSweep GC in 342ms.  CMS Old Gen: 18579136 -> 16305144;
Code Cache: 8600128 -> 10898752; Compressed Class Space: 3431288 ->
3761496; Metaspace: 29551832 -> 33307352; Par Eden Space: 4822000 ->
94853272;
INFO  [Service Thread] 2015-12-21 11:39:30,710 GCInspector.java:284 

RE: Cassandra 3.1 - Aggregation query failure

[SEAN_R_DURITY]

An aggregate only within a partition? That is rather useless and shouldn’t be 
called an aggregate.

I am hoping the functionality can be used to support at least “normal” types of 
aggregates like count, sum, avg, etc.


Sean Durity – Lead Cassandra Admin

From: Jonathan Haddad [mailto:j...@jonhaddad.com]
Sent: Monday, December 21, 2015 2:50 PM
To: user@cassandra.apache.org; dinesh.shanb...@isanasystems.com
Subject: Re: Cassandra 3.1 - Aggregation query failure

Even if you get this to work for now, I really recommend using a different 
tool, like Spark.  Personally I wouldn't use UDAs outside of a single partition.

On Mon, Dec 21, 2015 at 1:50 AM Dinesh Shanbhag 
> 
wrote:

Thanks for the pointers!  I edited jvm.options in
$CASSANDRA_HOME/conf/jvm.options to increase -Xms and -Xmx to 1536M.
The result is the same.

And in $CASSANDRA_HOME/logs/system.log, grep GC system.log produces this
(when jvm.options had not been changed):

INFO  [Service Thread] 2015-12-18 15:26:31,668 GCInspector.java:284 -
ConcurrentMarkSweep GC in 296ms.  CMS Old Gen: 18133664 -> 15589256;
Code Cache: 5650880 -> 8122304; Compressed Class Space: 2530064 ->
3345624; Metaspace: 21314000 -> 28040984; Par Eden Space: 7019256 ->
164070848;
INFO  [Service Thread] 2015-12-18 15:48:39,736 GCInspector.java:284 -
ConcurrentMarkSweep GC in 379ms.  CMS Old Gen: 649257416 -> 84190176;
Code Cache: 20772224 -> 20726848; Par Eden Space: 2191408 -> 52356736;
Par Survivor Space: 2378448 -> 2346840
INFO  [Service Thread] 2015-12-18 15:58:35,118 GCInspector.java:284 -
ConcurrentMarkSweep GC in 406ms.  CMS Old Gen: 648847808 -> 86954856;
Code Cache: 21182080 -> 21188032; Par Eden Space: 1815696 -> 71525744;
Par Survivor Space: 2388648 -> 2364696
INFO  [Service Thread] 2015-12-18 16:13:45,821 GCInspector.java:284 -
ConcurrentMarkSweep GC in 211ms.  CMS Old Gen: 648343768 -> 73135720;
Par Eden Space: 3224880 -> 7957464; Par Survivor Space: 2379912 -> 2414520
INFO  [Service Thread] 2015-12-18 16:32:46,419 GCInspector.java:284 -
ConcurrentMarkSweep GC in 387ms.  CMS Old Gen: 648476072 -> 6832;
Par Eden Space: 2006624 -> 64263360; Par Survivor Space: 2403792 -> 2387664
INFO  [Service Thread] 2015-12-18 16:42:38,648 GCInspector.java:284 -
ConcurrentMarkSweep GC in 365ms.  CMS Old Gen: 649126336 -> 137359384;
Code Cache: 22972224 -> 22979840; Metaspace: 41374464 -> 41375104; Par
Eden Space: 4286080 -> 154449480; Par Survivor Space: 1575440 -> 2310768
INFO  [Service Thread] 2015-12-18 16:51:57,538 GCInspector.java:284 -
ConcurrentMarkSweep GC in 322ms.  CMS Old Gen: 648338928 -> 79783856;
Par Eden Space: 2058968 -> 56931312; Par Survivor Space: 2342760 -> 2400336
INFO  [Service Thread] 2015-12-18 17:02:49,543 GCInspector.java:284 -
ConcurrentMarkSweep GC in 212ms.  CMS Old Gen: 648702008 -> 122954344;
Par Eden Space: 3269032 -> 61433328; Par Survivor Space: 2395824 -> 3448760
INFO  [Service Thread] 2015-12-18 17:11:54,090 GCInspector.java:284 -
ConcurrentMarkSweep GC in 306ms.  CMS Old Gen: 648748576 -> 70965096;
Par Eden Space: 2174840 -> 27074432; Par Survivor Space: 2365992 -> 2373984
INFO  [Service Thread] 2015-12-18 17:22:28,949 GCInspector.java:284 -
ConcurrentMarkSweep GC in 350ms.  CMS Old Gen: 648243024 -> 90897272;
Par Eden Space: 2150168 -> 43487192; Par Survivor Space: 2401872 -> 2410728


After modifying jvm.options to increase -Xms & -Xmx (to 1536M):

INFO  [Service Thread] 2015-12-21 11:39:24,918 GCInspector.java:284 -
ConcurrentMarkSweep GC in 342ms.  CMS Old Gen: 18579136 -> 16305144;
Code Cache: 8600128 -> 10898752; Compressed Class Space: 3431288 ->
3761496; Metaspace: 29551832 -> 33307352; Par Eden Space: 4822000 ->
94853272;
INFO  [Service Thread] 2015-12-21 11:39:30,710 GCInspector.java:284 -
ParNew GC in 206ms.  CMS Old Gen: 22932208 -> 41454520; Par Eden Space:
167772160 -> 0; Par Survivor Space: 13144872 -> 20971520
INFO  [Service Thread] 2015-12-21 13:08:14,922 GCInspector.java:284 -
ConcurrentMarkSweep GC in 468ms.  CMS Old Gen: 21418016 -> 16146528;
Code Cache: 11693888 -> 11744704; Compressed Class Space: 4331224 ->
4344192; Metaspace: 37191144 -> 37249960; Par Eden Space: 146089224 ->
148476848;
INFO  [Service Thread] 2015-12-21 13:08:53,068 GCInspector.java:284 -
ParNew GC in 216ms.  CMS Old Gen: 16146528 -> 26858568; Par Eden Space:
167772160 -> 0;


Earlier the node had OpenJDK 8.  For today's tests I installed and used
Oracle Java 8.

Do the above messages provide any clue? Or any debug logging I can
enable to progress further?
Thanks,
Dinesh.

On 12/18/2015 9:56 PM, Tyler Hobbs wrote:
>
> On Fri, Dec 18, 2015 at 9:17 AM, DuyHai Doan 
> 
> >> wrote:
>
> Cassandra will perform a full table scan and fetch all the data in
> memory to apply the aggregate function.
>
>
> Just to clarify for others on the list: when executing aggregation
> 

RE: External authentication options for C* (no DSE)

[SEAN_R_DURITY]

You can write your own using the appropriate interface(s) (for authentication 
and authorization). However, the interfaces have changed over the years, so it 
is likely not a write it and forget it bit of code.

External security is one of the important features of DSE, though.

Sean Durity – Lead Cassandra Admin

From: Giampaolo Trapasso [mailto:giampaolo.trapa...@radicalbit.io]
Sent: Wednesday, December 23, 2015 10:46 AM
To: user@cassandra.apache.org
Subject: Re: External authentication options for C* (no DSE)

Search a bit deeper in DSE docs, I've found this: 
http://www.datastax.com/wp-content/themes/datastax-2014-08/files/FF-DataStax-AdvancedSecurity.pdf.
Pratically no external authentication is available for Apache Cassandra.
giampaolo

2015-12-23 15:13 GMT+01:00 Giampaolo Trapasso 
>:
Hi,
while for DSE versions of C* it's quite clear what external authentication 
options are available, I'm not sure about DSC or Apache versions. Can anyone 
point me to the right documentation on or provide a list of possibilities?
Thank you in advance.
giampaolo




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Change authorization from AllowAllAuthorizer to CassandraAuthorizer

[SEAN_R_DURITY]

Which Cassandra version? Most of my authentication-from-non-authentication 
experience is from Cassandra 1.1 – 2.0. After that, I just enable from the 
beginning.

Sean Durity – Lead Cassandra Admin
Big DATA Team
MTC 2250
For support, create a 
JIRA

From: Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com]
Sent: Tuesday, June 07, 2016 8:31 PM
To: user@cassandra.apache.org
Subject: Re: Change authorization from AllowAllAuthorizer to CassandraAuthorizer

Hello Sean,

Recently I tried to enable Authentication on a existing cluster, I have see the 
below behaviour. (Clients already have the credentials and 3 node C* cluster)

cluster 1 - Enabled Authentication on node1 by adding iptable rules (so that 
client will not communicate to this node) and I was able to connect to cql with 
default user and create the required users.

cluster 2- Enabled Authentication on node1 by adding iptable rules but the 
default user was not created and below are the logs.

WARN  [NonPeriodicTasks:1] 2016-06-07 20:59:17,898 
PasswordAuthenticator.java:230 - PasswordAuthenticator skipped default user 
setup: some nodes were not ready
WARN  [NonPeriodicTasks:1] 2016-06-07 20:59:28,007 Auth.java:241 - Skipped 
default superuser setup: some nodes were not ready

Any idea why the behaviour is not consistent across the two clusters?

P.S: In both the cases the system_auth keyspace was created when the first node 
was updated.

On Tue, Jun 7, 2016 at 11:19 AM, Felipe Esteves 
> wrote:
Thank you, Sean!


Felipe Esteves

Tecnologia

felipe.este...@b2wdigital.com

Tel.: (21) 3504-7162 ramal 57162

Skype: felipe2esteves

2016-06-07 14:20 GMT-03:00 
>:
I answered a similar question here:
https://groups.google.com/forum/#!topic/nosql-databases/lLBebUCjD8Y


Sean Durity – Lead Cassandra Admin

From: Felipe Esteves 
[mailto:felipe.este...@b2wdigital.com]
Sent: Tuesday, June 07, 2016 12:07 PM
To: user@cassandra.apache.org
Subject: Change authorization from AllowAllAuthorizer to CassandraAuthorizer

Hi guys,

I have a Cassandra 2.1.8 Community cluster running with AllowAllAuthorizer and 
have to change it, so I can implement permissions in different users.
As I've checked in the docs, seems like a simple change, from 
AllowAllAuthorizer to CassandraAuthorizer in cassandra.yaml.
However, I'm a litte concerned about the performance of the cluster while I'm 
restarting all the nodes. Is it possible to have any downtime (access errors, 
maybe), as all the data was created with AllowAllAuthorizer?
--

Felipe Esteves

Tecnologia

felipe.este...@b2wdigital.com

Tel.: (21) 3504-7162 ramal 57162




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.






The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, 

RE: Change authorization from AllowAllAuthorizer to CassandraAuthorizer

[SEAN_R_DURITY]

I answered a similar question here:
https://groups.google.com/forum/#!topic/nosql-databases/lLBebUCjD8Y


Sean Durity – Lead Cassandra Admin

From: Felipe Esteves [mailto:felipe.este...@b2wdigital.com]
Sent: Tuesday, June 07, 2016 12:07 PM
To: user@cassandra.apache.org
Subject: Change authorization from AllowAllAuthorizer to CassandraAuthorizer

Hi guys,

I have a Cassandra 2.1.8 Community cluster running with AllowAllAuthorizer and 
have to change it, so I can implement permissions in different users.
As I've checked in the docs, seems like a simple change, from 
AllowAllAuthorizer to CassandraAuthorizer in cassandra.yaml.
However, I'm a litte concerned about the performance of the cluster while I'm 
restarting all the nodes. Is it possible to have any downtime (access errors, 
maybe), as all the data was created with AllowAllAuthorizer?
--

Felipe Esteves

Tecnologia

felipe.este...@b2wdigital.com

Tel.: (21) 3504-7162 ramal 57162




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Per node limit for Disk Space

[SEAN_R_DURITY]

Eric is right on.

Let me share my experience. I have found that dense nodes over 4 TB are a pain 
to manage (for rebuilds, repair, compaction, etc.) with size-tiered compaction 
and basically a single table schema. However, 1 TB nodes that yield only about 
500 GB of usable space can create rings with just too many nodes (and too 
expensive for the usable storage). 2 TB seems to be a good sweet spot to avoid 
either extreme. However, Eric is correct, there are lots of factors to weigh in 
the decision.

Sean Durity – Lead Cassandra Admin

From: Eric Stevens [mailto:migh...@gmail.com]
Sent: Monday, May 30, 2016 10:56 AM
To: user@cassandra.apache.org
Subject: Re: Per node limit for Disk Space

Those are rough guidelines, actual effective node size is going to depend on 
your read/write workload and the compaction strategy you choose.  The biggest 
reason data density per node usually needs to be limited is due to data 
grooming overhead introduced by compaction.  Data at rest essentially becomes 
I/O debt.  If you're using Leveled compaction, the interest rate on that debt 
is higher.

If you're writing aggressively you'll find that you run out of I/O capacity for 
smaller data at rest.  If you use compaction strategies that allow for data to 
eventually stop compacting (Date Tiered, Time Windowed), you may be able to 
have higher data density per node assuming that some of your data is going into 
the no-longer-compacting stages.

Beyond that it'll be hard to say what the right size for you is.  Target the 
recommended numbers and if you find that you're not running out of I/O as you 
approach them you can probably go bigger.  Just remember to leave ~50% disk 
capacity free to leave room for compaction to happen.

On Fri, May 27, 2016 at 1:52 PM Anshu Vajpayee 
> wrote:
Hi All,
I have question regarding max disk space limit  on a node.

As per Data stax, We can have 1TB max disk space for rotational disks and up to 
5 TB for SSDs on a node.

Could you please suggest per your experience what would be limit for space on a 
single node with out causing so much stress on a  node?




​Thanks,​





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: "Not enough replicas available for query" after reboot

[SEAN_R_DURITY]

What client are you using?

It is possible that the client saw nodes down and has kept them marked that way 
(without retrying). Depending on the client, you may have options to set in 
RetryPolicy, FailoverPolicy, etc. A bounce of the client will probably fix the 
problem for now.


Sean Durity

From: Flavien Charlon [mailto:flavien.char...@gmail.com]
Sent: Thursday, February 04, 2016 4:06 PM
To: user@cassandra.apache.org
Subject: Re: "Not enough replicas available for query" after reboot

Yes, all three nodes see all three nodes as UN.

Also, connecting from a local Cassandra machine using cqlsh, I can run the same 
query just fine (with QUORUM consistency level).

On 4 February 2016 at 21:02, Robert Coli 
> wrote:
On Thu, Feb 4, 2016 at 12:53 PM, Flavien Charlon 
> wrote:
My cluster was running fine. I rebooted all three nodes (one by one), and now 
all nodes are back up and running. "nodetool status" shows UP for all three 
nodes on all three nodes:

--  AddressLoad   Tokens  OwnsHost ID   
Rack
UN  xx.xx.xx.xx331.84 GB  1   ?   
d3d3a79b-9ca5-43f9-88c4-c3c7f08ca538  RAC1
UN  xx.xx.xx.xx317.2 GB   1   ?   
de7917ed-0de9-434d-be88-bc91eb4f8713  RAC1
UN  xx.xx.xx.xx  291.61 GB  1   ?   
b489c970-68db-44a7-90c6-be734b41475f  RAC1

However, now the client application fails to run queries on the cluster with:

Cassandra.UnavailableException: Not enough replicas available for query at 
consistency Quorum (2 required but only 1 alive)

Do *all* nodes see each other as UP/UN?

=Rob





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Restart Cassandra automatically

[SEAN_R_DURITY]

Call me naïve, but we do use an in-house built program for keeping nodes 
started (based on a flag-check). The program is something that was written for 
all kinds of daemon processes here, not Cassandra specifically. The basic idea 
is that is runs a status check. If that fails, and the flag is set, start 
Cassandra. In my opinion, it has helped more than hurt us – especially with the 
very fragile 1.1 releases that were prone to heap problems.

Sean Durity

From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: Friday, February 05, 2016 1:28 PM
To: user@cassandra.apache.org
Subject: Re: Restart Cassandra automatically

On Thu, Feb 4, 2016 at 8:26 PM, Debraj Manna 
> wrote:

What is the best way to keep cassandra running? My requirement is if for some 
reason cassandra stops then it should get started automatically.
I recommend against this mode of operation. When automatically restarting, you 
have no idea how long Cassandra has been stopped and for what reason. In some 
cases, you really do not want it to start up and attempt to participate in 
whatever cluster it was formerly participating in.

I understand this creates a support overhead, especially with very large 
clusters, but it's difficult for me to accept the premise that net operational 
safety will be improved by naively restarting nodes.

=Rob




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Restart Cassandra automatically

[SEAN_R_DURITY]

What anti-pattern are you mocking me for exactly?


Sean Durity

From: daemeon reiydelle [mailto:daeme...@gmail.com]
Sent: Tuesday, February 23, 2016 11:21 AM
To: user@cassandra.apache.org
Subject: RE: Restart Cassandra automatically


Cassandra nodes do not go down "for no reason". They are not stateless. I would 
like to thank you for this marvelous example of a wonderful antipattern. 
Absolutely fantastic.

Thank you! I am not being a satirical smartass. I sometimes am challenged by 
clients in my presentations about sre best practices around c*, hadoop, and elk 
on the grounds that "noone would ever do this in production". Now I have 
objective proof!

Daemeon

sent from my mobile
Daemeon C.M. Reiydelle
USA 415.501.0198
London +44.0.20.8144.9872
On Feb 23, 2016 7:53 AM, 
> wrote:
Yes, I can see the potential problem in theory. However, we never do your #2. 
Generally, we don’t have unused spare hardware. We just fix the host that is 
down and run repairs. (Side note: while I have seen nodes fight it out over who 
owns a particular token in earlier versions, it seems that 1.2+ doesn’t allow 
that to happen as easily. The second node will just not come up.)

For most of our use cases, I would agree with your Coli Conjecture.


Sean Durity

From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: Tuesday, February 09, 2016 4:41 PM
To: user@cassandra.apache.org
Subject: Re: Restart Cassandra automatically

On Tue, Feb 9, 2016 at 6:20 AM, 
> wrote:
Call me naïve, but we do use an in-house built program for keeping nodes 
started (based on a flag-check). The program is something that was written for 
all kinds of daemon processes here, not Cassandra specifically. The basic idea 
is that is runs a status check. If that fails, and the flag is set, start 
Cassandra. In my opinion, it has helped more than hurt us – especially with the 
very fragile 1.1 releases that were prone to heap problems.

Ok, you're naïve.. ;P

But seriously, think of this scenario :

1) Node A, responsible for range A-M, goes down due to hardware failure of a 
disk in a RAID
2) Node B is put into service and is made responsible for A-M
3) Months pass
4) Node A comes back up, announces that it is responsible for A-M, and the 
cluster agrees

Consistency is now permanently broken for any involved rows. Why doesn't it 
(usually) matter?

It's not so much that you are naïve but that you are providing still more 
support for the Coli Conjecture : "If you are using a distributed database you 
probably do not care about consistency, even if you think you do." You have 
repeatedly chosen Availability over Consistency and it has never had a negative 
impact on your actual application.

=Rob




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its 

RE: High Bloom filter false ratio

[SEAN_R_DURITY]

I see the sstablemetadata tool as far back as 1.2.19 (in tools/bin).


Sean Durity
From: Anishek Agarwal [mailto:anis...@gmail.com]
Sent: Tuesday, February 23, 2016 3:37 AM
To: user@cassandra.apache.org
Subject: Re: High Bloom filter false ratio

Looks like that sstablemetadata is available in 2.2 , we are on 2.0.x do you 
know anything that will work on 2.0.x

On Tue, Feb 23, 2016 at 1:48 PM, Anishek Agarwal 
> wrote:
Thanks Jeff, Awesome will look at the tools and JMX endpoint.

our settings are below originated from the jira you posted above as the base. 
we are running on 48 core machines with 2 SSD disks of 800 GB each .


MAX_HEAP_SIZE="6G"

HEAP_NEWSIZE="4G"

JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC"

JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"

JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"

JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=6"

JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=4"

JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=70"

JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"

JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"

JVM_OPTS="$JVM_OPTS -XX:MaxPermSize=256m"

JVM_OPTS="$JVM_OPTS -XX:+AggressiveOpts"

JVM_OPTS="$JVM_OPTS -XX:+UseCompressedOops"

JVM_OPTS="$JVM_OPTS -XX:+CMSScavengeBeforeRemark"

JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=48"

JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=48"

JVM_OPTS="$JVM_OPTS -XX:-ExplicitGCInvokesConcurrent"

JVM_OPTS="$JVM_OPTS -XX:+UnlockDiagnosticVMOptions"

JVM_OPTS="$JVM_OPTS -XX:+UseGCTaskAffinity"

JVM_OPTS="$JVM_OPTS -XX:+BindGCTaskThreadsToCPUs"

# earlier value 131072

JVM_OPTS="$JVM_OPTS -XX:ParGCCardsPerStrideChunk=32678"

JVM_OPTS="$JVM_OPTS -XX:CMSScheduleRemarkEdenSizeThreshold=104857600"

JVM_OPTS="$JVM_OPTS -XX:CMSRescanMultiple=32678"

JVM_OPTS="$JVM_OPTS -XX:CMSConcMarkMultiple=32678"


On Tue, Feb 23, 2016 at 1:06 PM, Jeff Jirsa 
> wrote:
There exists a JMX endpoint called forceUserDefinedCompaction that takes a 
comma separated list of sstables to compact together.

There also exists a tool called sstablemetadata (may be in a ‘cassandra-tools’ 
package separate from whatever package you used to install cassandra, or in the 
tools/ directory of your binary package). Using sstablemetadata, you can look 
at the maxTimestamp for each table, and the ‘Estimated droppable tombstones’. 
Using those two fields, you could, very easily, write a script that gives you a 
list of sstables that you could feed to forceUserDefinedCompaction to join 
together to eliminate leftover waste.

Your long ParNew times may be fixable by increasing the new gen size of your 
heap – the general guidance in cassandra-env.sh is out of date, you may want to 
reference CASSANDRA-8150 for “newer” advice ( 
http://issues.apache.org/jira/browse/CASSANDRA-8150 )

- Jeff

From: Anishek Agarwal
Reply-To: "user@cassandra.apache.org"
Date: Monday, February 22, 2016 at 8:33 PM

To: "user@cassandra.apache.org"
Subject: Re: High Bloom filter false ratio

Hey Jeff,

Thanks for the clarification, I did not explain my self clearly, the 
max_stable_age_days is set to 30 days and the ttl on every insert is set to 30 
days also by default. gc_grace_seconds is 0, so i would think the sstable as a 
whole would be deleted.

Because of the problems mentioned by at 1) above it looks like, there might be 
cases where the table just lies around since no compaction is happening on it 
and even though everything is expired it would still not be deleted?

for 3) the average read is pretty good, though the throughput doesn't seem to 
be that great, when no repair is running we get GCIns > 200ms every couple of 
hours once, otherwise its every 10-20 mins

INFO [ScheduledTasks:1] 2016-02-23 05:15:03,070 GCInspector.java (line 116) GC 
for ParNew: 205 ms for 1 collections, 1712439128 used; max is 7784628224

 INFO [ScheduledTasks:1] 2016-02-23 08:30:47,709 GCInspector.java (line 116) GC 
for ParNew: 242 ms for 1 collections, 1819126928 used; max is 7784628224

 INFO [ScheduledTasks:1] 2016-02-23 09:09:55,085 GCInspector.java (line 116) GC 
for ParNew: 374 ms for 1 collections, 1829660304 used; max is 7784628224

 INFO [ScheduledTasks:1] 2016-02-23 09:11:21,245 GCInspector.java (line 116) GC 
for ParNew: 419 ms for 1 collections, 2309875224 used; max is 7784628224

 INFO [ScheduledTasks:1] 2016-02-23 09:35:50,717 GCInspector.java (line 116) GC 
for ParNew: 231 ms for 1 collections, 2515325328 used; max is 7784628224

 INFO [ScheduledTasks:1] 2016-02-23 09:38:47,194 GCInspector.java (line 116) GC 
for ParNew: 252 ms for 1 collections, 1724241952 used; max is 7784628224


our reading patterns are dependent on BF to work efficiently as we do a lot of 
reads for keys that may not exists because its time series and we segregate 
data based on hourly boundary from epoch.


hey Christoper,

yes every row in the stable that should have 

RE: Restart Cassandra automatically

[SEAN_R_DURITY]

Yes, I can see the potential problem in theory. However, we never do your #2. 
Generally, we don’t have unused spare hardware. We just fix the host that is 
down and run repairs. (Side note: while I have seen nodes fight it out over who 
owns a particular token in earlier versions, it seems that 1.2+ doesn’t allow 
that to happen as easily. The second node will just not come up.)

For most of our use cases, I would agree with your Coli Conjecture.


Sean Durity

From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: Tuesday, February 09, 2016 4:41 PM
To: user@cassandra.apache.org
Subject: Re: Restart Cassandra automatically

On Tue, Feb 9, 2016 at 6:20 AM, 
> wrote:
Call me naïve, but we do use an in-house built program for keeping nodes 
started (based on a flag-check). The program is something that was written for 
all kinds of daemon processes here, not Cassandra specifically. The basic idea 
is that is runs a status check. If that fails, and the flag is set, start 
Cassandra. In my opinion, it has helped more than hurt us – especially with the 
very fragile 1.1 releases that were prone to heap problems.

Ok, you're naïve.. ;P

But seriously, think of this scenario :

1) Node A, responsible for range A-M, goes down due to hardware failure of a 
disk in a RAID
2) Node B is put into service and is made responsible for A-M
3) Months pass
4) Node A comes back up, announces that it is responsible for A-M, and the 
cluster agrees

Consistency is now permanently broken for any involved rows. Why doesn't it 
(usually) matter?

It's not so much that you are naïve but that you are providing still more 
support for the Coli Conjecture : "If you are using a distributed database you 
probably do not care about consistency, even if you think you do." You have 
repeatedly chosen Availability over Consistency and it has never had a negative 
impact on your actual application.

=Rob




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Nodes go down periodically

[SEAN_R_DURITY]

You didn’t mention version, but I saw this kind of thing very often in the 1.1 
line. Often this is connected to network flakiness. Are these VMs? In the 
cloud? Connected over a WAN? You mention that ping seems fine. Take a look at 
the phi_convict_threshold in c assandra.yaml. You may need to increase it to 
reduce the UP/DOWN flapping behavior.


Sean Durity

From: Joel Samuelsson [mailto:samuelsson.j...@gmail.com]
Sent: Tuesday, February 23, 2016 9:41 AM
To: user@cassandra.apache.org
Subject: Re: Nodes go down periodically

Hi,

Thanks for your reply.

I have debug logging on and see no GC pauses that are that long. GC pauses are 
all well below 1s and 99 times out of 100 below 100ms.
Do I need to enable GC log options to see the pauses?
I see plenty of these lines:
DEBUG [ScheduledTasks:1] 2016-02-22 10:43:02,891 GCInspector.java (line 118) GC 
for ParNew: 24 ms for 1 collections
as well as a few CMS GC log lines.

Best regards,
Joel

2016-02-23 15:14 GMT+01:00 Hannu Kröger 
>:
Hi,

Those are probably GC pauses. Memory tuning is probably needed. Check the 
parameters that you already have customised if they make sense.

http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html

Hannu


On 23 Feb 2016, at 16:08, Joel Samuelsson 
> wrote:

Our nodes go down periodically, around 1-2 times each day. Downtime is from <1 
second to 30 or so seconds.

INFO [GossipTasks:1] 2016-02-22 10:05:14,896 Gossiper.java (line 992) 
InetAddress /109.74.13.67 is now DOWN
 INFO [RequestResponseStage:8844] 2016-02-22 10:05:38,331 Gossiper.java (line 
978) InetAddress /109.74.13.67 is now UP

I find nothing odd in the logs around the same time. I logged a ping with 
timestamp and checked during the same time and saw nothing weird (ping is less 
than 2ms at all times).

Does anyone have any suggestions as to why this might happen?

Best regards,
Joel





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Using TTL for data purge

[SEAN_R_DURITY]

Thank, I appreciate the correction to my understanding.


Sean Durity

From: Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com]
Sent: Friday, January 22, 2016 1:04 PM
To: user@cassandra.apache.org
Subject: Re: Using TTL for data purge

"As I understand TTL, if there is a compaction of a cell (or row) with a TTL 
that has been reached, a tombstone will be written.”

The expiring cell is treated as a tombstone once it reaches it’s end of life, 
it does not write an additional tombstone to disk.


From: "sean_r_dur...@homedepot.com"
Reply-To: "user@cassandra.apache.org"
Date: Friday, January 22, 2016 at 7:27 AM
To: "user@cassandra.apache.org"
Subject: RE: Using TTL for data purge

An upsert is a second insert. Cassandra’s sstables are immutable. There are no 
real “overwrites” (of the data on disk). It is another record/row. Upon read, 
it acts like an overwrite, because Cassandra will read both inserts and take 
the last one in as the correct data. This strategy will work for changing the 
TTL (and anything else that changes in the data).

Compaction creates a new sstable from existing ones. It will (if the inserts 
are in the compacted sstables) write only the latest data, so the older insert 
is effectively deleted/dropped from the new sstable now on disk.

As I understand TTL, if there is a compaction of a cell (or row) with a TTL 
that has been reached, a tombstone will be written.

Sean Durity – Lead Cassandra Admin
Big DATA Team
For support, create a 
JIRA

From: Joseph TechMails [mailto:jaalex.t...@gmail.com]
Sent: Wednesday, December 30, 2015 3:59 AM
To: user@cassandra.apache.org
Subject: Re: Using TTL for data purge

Thanks, Sean. Our usecase is to delete records after few months of inactivity, 
and that period is fixed, but the TTL could get reset if the record is accessed 
within that timeframe - similar to extending a session. All reads are done 
based on the key, and there would be multiple upserts (all columns are 
re-INSERTed, including TTL) while it's active, so it's not exactly 
write-once/read-many. Are there any overheads for processes like compaction due 
to this overwriting of TTL? . I guess reads won't be affected since it's always 
done with the key, and won't have to filter out tombstones.

Regarding the data size, i could see a small decrease in the disk usage (du) of 
the "data" directory immediately after the rows with TTL expired, and still 
further reduction after running compaction on the CF (though this wasn't 
replicable always). Since the tombstones should ideally stay for 10 days, i 
assume this observation is not related to data expiry. Please confirm

Thanks,
Joseph


On Tue, Dec 29, 2015 at 11:20 PM, 
> wrote:
If you know how long the records should last, TTL is a good way to go. Remember 
that neither TTL or deletes are right-away purge strategies. Each inserts a 
special record called a tombstone to indicate a deleted record. After 
compaction (that is after gc_grace_seconds for the table, default 10 days), the 
data will be removed and you will regain disk space.

If the data is relatively volatile and read speeds are important, you might 
look at leveled compaction, though it can keep your nodes a bit busier than 
size-tiered. (An issue with size-tiered, over time, is that the tombstoned data 
in the larger and older sstables may rarely, if ever, get compacted out.)


Sean Durity – Lead Cassandra Admin
From: jaalex.tech [mailto:jaalex.t...@gmail.com]
Sent: Tuesday, December 22, 2015 4:36 AM
To: user@cassandra.apache.org
Subject: Using TTL for data purge

Hi,

I'm looking for suggestions/caveats on using TTL as a subsitute for a manual 
data purge job.

We have few tables that hold user information - this could be guest or 
registered users, and there could be between 500K to 1M records created per day 
per table. Currently, these tables have a secondary indexed updated_date column 
which is populated on each update. However, we have been getting timeouts when 
running queries using updated_date when the number of records are high, so i 
don't think this would be a reliable option in the long term when we need to 
purge records that have not been used for the last X days.

In this scenario, is it advisable to include a high enough TTL (i.e the amount 
of time we want these to last, could be 3 to 6 months) when inserting/updating 
records?

There could be cases where the TTL may get reset after couple of days/weeks, 
when the user visits the site again.

The tables have fixed number of columns, except for one which has a clustering 
key, and may have max 10 entries per  partition key.

I need to know the 

RE: NullPointerException when trying to compact under 3.2

[SEAN_R_DURITY]

Can you wipe all the data directories, saved_cache, and commitlog and let the 
node bootstrap again?


Sean Durity

From: Nimi Wariboko Jr [mailto:n...@channelmeter.com]
Sent: Monday, January 25, 2016 6:59 PM
To: cassandra-u...@apache.org
Subject: NullPointerException when trying to compact under 3.2

Hi,

I recently upgraded from 2.1.12 to 3.2, and one issue I'm having is I can no 
longer read certain rows from a table. A simple SELECT * FROM `table` times 
out, only when the bad partition keys are reached. Trying to query the affected 
partition keys directly also causes a timeout.

I think the SSTable might be corrupted because nodetool compact (and repair) 
fails (although scrub succeeds). With Debug logging, a compact results in:

cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3778-big-Data.db:level=0,
 
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3780-big-Data.db:level=0,
 
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3775-big-Data.db:level=0,
 
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3776-big-Data.db:level=0,
 
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3798-big-Data.db:level=0,
 ]
WARN  [CompactionExecutor:5] 2016-01-25 15:13:02,734 SSTableReader.java:261 - 
Reading cardinality from Statistics.db failed for 
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3778-big-Data.db
ERROR [CompactionExecutor:5] 2016-01-25 15:13:02,777 CassandraDaemon.java:195 - 
Exception in thread Thread[CompactionExecutor:5,1,main]
java.lang.NullPointerException: null

(the NPE has no stack trace).

On disk, the SSTable "ma-3778-big-Data.db" does not exist. Even if I scrub or 
restart, Cassandra seems to always try to compact this non existent table (the 
SSTables on Disk are numbered 3851+).

I'm assuming the SSTables are fubared, and I'd like to restore a snapshot, but 
the fact this "ghost" sstable causes the compaction to fail, I'm unsure if 
restoring a new set of sstables would actually solve the issue.

Nimi



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: opscenter doesn't work with cassandra 3.0

[SEAN_R_DURITY]

This is a very strange move considering how well DataStax has supported open 
source Cassandra. I hope there is a reasonable and well-publicized explanation 
for this apparent change in direction.


Sean Durity

From: George Sigletos [mailto:sigle...@textkernel.nl]
Sent: Tuesday, January 26, 2016 4:09 AM
To: user@cassandra.apache.org
Subject: Re: opscenter doesn't work with cassandra 3.0

Unfortunately Datastax decided to discontinue Opscenter for open source 
Cassandra, starting from version 2.2.

Pitty

On Wed, Jan 6, 2016 at 6:00 PM, Michael Shuler 
> wrote:
On 01/06/2016 10:55 AM, Michael Shuler wrote:
> On 01/06/2016 01:47 AM, Wills Feng wrote:
>> Looks like opscenter doesn't support cassandra 3.0?
>
> This is correct. OpsCenter does not support Cassandra >= 3.0.

It took me a minute to find the correct document:

http://docs.datastax.com/en/upgrade/doc/upgrade/opscenter/opscCompatibility.html

According to this version table, OpsCenter does not officially support
Cassandra > 2.1.

--
Michael




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Using TTL for data purge

[SEAN_R_DURITY]

An upsert is a second insert. Cassandra’s sstables are immutable. There are no 
real “overwrites” (of the data on disk). It is another record/row. Upon read, 
it acts like an overwrite, because Cassandra will read both inserts and take 
the last one in as the correct data. This strategy will work for changing the 
TTL (and anything else that changes in the data).

Compaction creates a new sstable from existing ones. It will (if the inserts 
are in the compacted sstables) write only the latest data, so the older insert 
is effectively deleted/dropped from the new sstable now on disk.

As I understand TTL, if there is a compaction of a cell (or row) with a TTL 
that has been reached, a tombstone will be written.

Sean Durity – Lead Cassandra Admin
Big DATA Team
For support, create a 
JIRA

From: Joseph TechMails [mailto:jaalex.t...@gmail.com]
Sent: Wednesday, December 30, 2015 3:59 AM
To: user@cassandra.apache.org
Subject: Re: Using TTL for data purge

Thanks, Sean. Our usecase is to delete records after few months of inactivity, 
and that period is fixed, but the TTL could get reset if the record is accessed 
within that timeframe - similar to extending a session. All reads are done 
based on the key, and there would be multiple upserts (all columns are 
re-INSERTed, including TTL) while it's active, so it's not exactly 
write-once/read-many. Are there any overheads for processes like compaction due 
to this overwriting of TTL? . I guess reads won't be affected since it's always 
done with the key, and won't have to filter out tombstones.

Regarding the data size, i could see a small decrease in the disk usage (du) of 
the "data" directory immediately after the rows with TTL expired, and still 
further reduction after running compaction on the CF (though this wasn't 
replicable always). Since the tombstones should ideally stay for 10 days, i 
assume this observation is not related to data expiry. Please confirm

Thanks,
Joseph


On Tue, Dec 29, 2015 at 11:20 PM, 
> wrote:
If you know how long the records should last, TTL is a good way to go. Remember 
that neither TTL or deletes are right-away purge strategies. Each inserts a 
special record called a tombstone to indicate a deleted record. After 
compaction (that is after gc_grace_seconds for the table, default 10 days), the 
data will be removed and you will regain disk space.

If the data is relatively volatile and read speeds are important, you might 
look at leveled compaction, though it can keep your nodes a bit busier than 
size-tiered. (An issue with size-tiered, over time, is that the tombstoned data 
in the larger and older sstables may rarely, if ever, get compacted out.)


Sean Durity – Lead Cassandra Admin
From: jaalex.tech [mailto:jaalex.t...@gmail.com]
Sent: Tuesday, December 22, 2015 4:36 AM
To: user@cassandra.apache.org
Subject: Using TTL for data purge

Hi,

I'm looking for suggestions/caveats on using TTL as a subsitute for a manual 
data purge job.

We have few tables that hold user information - this could be guest or 
registered users, and there could be between 500K to 1M records created per day 
per table. Currently, these tables have a secondary indexed updated_date column 
which is populated on each update. However, we have been getting timeouts when 
running queries using updated_date when the number of records are high, so i 
don't think this would be a reliable option in the long term when we need to 
purge records that have not been used for the last X days.

In this scenario, is it advisable to include a high enough TTL (i.e the amount 
of time we want these to last, could be 3 to 6 months) when inserting/updating 
records?

There could be cases where the TTL may get reset after couple of days/weeks, 
when the user visits the site again.

The tables have fixed number of columns, except for one which has a clustering 
key, and may have max 10 entries per  partition key.

I need to know the overhead of having so many rows with TTL hanging around for 
a relatively longer duration (weeks/months), and the impacts it could have on 
performance/storage. If this is not a recommended approach, what would be an 
alternate design which could be used for a manual purge job, without using 
secondary indices.

We are using Cassandra 2.0.x.

Thanks,
Joseph




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this 

RE: Experiencing strange disconnect issue

[SEAN_R_DURITY]

Are you using any of the Tuning Policies 
(https://docs.datastax.com/en/developer/java-driver/2.0/common/drivers/reference/tuningPolicies_c.html)?
 It could be that you are hitting some peak load and the driver is not retrying 
hosts once they are marked “down.”


Sean Durity – Lead Cassandra Admin
Big DATA Team
For support, create a 
JIRA

From: Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com]
Sent: Tuesday, March 15, 2016 5:24 AM
To: user@cassandra.apache.org
Subject: Experiencing strange disconnect issue

Hi,

We are currently trying to convert an existing java web application to use 
cassandra, and while most of it works great :) we have a "small" issue.

After some time, we all connectivity seems to be lost and we get the following 
errors:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried 
for query failed (tried: /10.61.70.107:9042 
(com.datastax.driver.core.exceptions.TransportException: 
[/10.61.70.107] Connection has been closed), 
/10.61.70.108:9042 
(com.datastax.driver.core.exceptions.TransportException: 
[/10.61.70.108] Connection has been closed))

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried 
for query failed (tried: /10.61.70.107:9042 
(com.datastax.driver.core.exceptions.DriverException: Timeout while trying to 
acquire available connection (you may want to increase the driver number of 
per-host connections)), /10.61.70.108:9042 
(com.datastax.driver.core.exceptions.TransportException: 
[/10.61.70.108] Connection has been closed), 
/10.61.70.110:9042 
(com.datastax.driver.core.exceptions.TransportException: 
[/10.61.70.110] Connection has been closed))

The errors persists, and the application needs to be restarted to recover.

At application startup we create a cluster and a session which we reuse through 
out the application as pr. the documentation. We don't specify any other 
options when connecting than the IP's of the three servers. We are running 
cassandra 3.0.3 tar ball in EC2 in a cluster of three machines. The connections 
are made using v3.0.0 java driver.

I have uploaded the configuration and logs from our cassandra cluster here: 
https://gist.github.com/anonymous/452e736b401317b5b38d
The issue happend at 00:44:46.

I would greatly appreciate any ideas as to what we are doing wrong to 
experience this? :)

Thank you in advance!

Yours sincerely,
  Bo Madsen



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: cross DC data sync starts without rebuilding nodes on new DC

[SEAN_R_DURITY]

What do you mean by “automatically starts syncing?” Are you seeing streaming of 
existing data or just the addition of new, incoming data? Do you have repairs 
running as part of your automated maintenance, perhaps?

I would expect that new, incoming data would be replicated to the new DC after 
the “alter table.” But I would not expect any streaming of existing data unless 
repairs are running somewhere.

Also, does your nodetool status reflect the new DC?

Sean Durity

From: Vijay Patil [mailto:vijay2110.t...@gmail.com]
Sent: Tuesday, April 05, 2016 2:27 AM
To: user@cassandra.apache.org
Subject: cross DC data sync starts without rebuilding nodes on new DC

Hi,

I have configured new DC as per instructions at below link.
https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html

After completing step 7.a (which is altering keyspace with desired replication 
factor for new DC) data automatically starts syncing from existing DC to new 
DC, which is not expected because auto_bootstrap is false on all nodes 
(existing as well as new DC).

Any idea why it's happening?
If this is the desired behaviour then what's the purpose of rebuilding each 
node on new DC (step 7.b)?

Cassandra version is 2.0.17 on all nodes in both DC's and I am using 
GossipingPropertyFileSnitch.

Regards,
Vijay



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: 1, 2, 3...

[SEAN_R_DURITY]

Cassandra is not good for table scan type queries (which count(*) typically 
is). While there are some attempts to do that (as noted below), this is a path 
I avoid.


Sean Durity

From: Max C [mailto:mc_cassan...@core43.com]
Sent: Saturday, April 09, 2016 6:19 PM
To: user@cassandra.apache.org
Subject: Re: 1, 2, 3...

Looks like this guy (Brian Hess) wrote a script to split the token range and 
run count(*) on each subrange:

https://github.com/brianmhess/cassandra-count

- Max

On Apr 8, 2016, at 10:56 pm, Jeff Jirsa 
> wrote:

SELECT COUNT(*) probably works (with internal paging) on many datasets with 
enough time and assuming you don’t have any partitions that will kill you.

No, it doesn’t count extra replicas / duplicates.

The old way to do this (before paging / fetch size) was to use manual paging 
based on tokens/clustering keys:

https://docs.datastax.com/en/cql/3.1/cql/cql_using/paging_c.html – SELECT’s 
WHERE clause can use token(), which is what you’d want to use to page through 
the whole token space.

You could, in theory, issue thousands of queries in parallel, all for different 
token ranges, and then sum the results. That’s what something like spark would 
be doing. If you want to determine rows per node, limit the token range to that 
owned by the node (easier with 1 token than vnodes, with vnodes repeat 
num_tokens times).




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: StatusLogger output

[SEAN_R_DURITY]

I am not sure the status logger output helps determine the problem. However, 
the dropped mutations and the status logger output is what I see when there is 
too high of a load on one or more Cassandra nodes. It could be long GC pauses, 
something reading too much data (a large row or a multi-partition query), or 
just too many requests for the number of nodes you have. Are you using 
OpsCenter to monitor the rings? Do you have read or write spikes at the time? 
Any GC messages in the log. Any nodes going down at the time?


Sean Durity

From: Vasileios Vlachos [mailto:vasileiosvlac...@gmail.com]
Sent: Thursday, March 24, 2016 8:13 AM
To: user@cassandra.apache.org
Subject: Re: StatusLogger output

Just to clarify, I can see line 29 which seems to explain the format (first 
number ops, second is data), however I don't know they actually mean.

Thanks,
Vasilis

On Thu, Mar 24, 2016 at 11:45 AM, Vasileios Vlachos 
> wrote:
Hello,

Environment:
- Cassandra 2.0.17, 8 nodes, 4 per DC
- Ubuntu 12.04, 6-Cores, 16GB of RAM (we use VMWare)

Every node seems to be dropping messages (anywhere from 10 to 300) twice a day. 
I don't know it this has always been the case, but has definitely been going 
for the past month or so. Whenever that happens we get StatusLogger.java output 
in the log, which is the state of the node at the time it dropped messages. 
This output contains information similar/identical to nodetool tpstats, but 
further from that, information regarding system CF follows as can be seen here: 
http://ur1.ca/ooan6

How can we use this information to find out what the problem was? I am 
specifically referring to the information regarding the system CF. I had a look 
in the system tables but I cannot draw anything from that. The output in the 
log seems to contain two values (comma separated). What are these numbers?

I wasn't able to find anything on the web/DataStax docs. Any help would be 
greatly appreciated!

Thanks,
Vasilis




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: What is the best way to model my time series?

[SEAN_R_DURITY]

I think this one is better…
https://www.google.com/url?sa=t=j==s=web=1=rja=8=0ahUKEwjF6eD9hdzLAhWCCj4KHfNwDHoQFggdMAA=https%3A%2F%2Fmail-archives.apache.org%2Fmod_mbox%2Fcassandra-user%2F201603.mbox%2F%253CCANeMN%3D-KEiXXgyWLnSYKnhQMWfWmmy3b68PkLsw55CGSm_UmmQ%40mail.gmail.com%253E=AFQjCNGGod8eDERVUbwHe6SVBoOq1y8N7Q=zzAKcoVlUb2M6YuuNy8AHw

Sean Durity

From: K. Lawson [mailto:klawso...@gmail.com]
Sent: Friday, March 25, 2016 10:17 AM
To: user@cassandra.apache.org
Subject: Re: What is the best way to model my time series?


Sean, the link you have supplied does not seem to work.

On Fri, Mar 25, 2016 at 9:43 AM, 
> wrote:
You might take a look at this previous conversation on queue-type applications 
and Cassandra. Generally this is an anti-pattern for a distributed system like 
Cassandra.

RE: What is the best way to model my time series?

[SEAN_R_DURITY]

You might take a look at this previous conversation on queue-type applications 
and Cassandra. Generally this is an anti-pattern for a distributed system like 
Cassandra.

RE: Gossip heartbeat and packet capture

[SEAN_R_DURITY]

Is this from the 1.1 line, perhaps? In my experience it could be very flappy 
for no particular reason we could discover. 1.1 is a pretty dusty version. 
Upgrading into the 2.1 or later would be a good idea. If you have to upgrade in 
place without down time, you will need to go through many upgrades to get to 
the latest versions. (The late versions of 1.2 are pretty stable, though.)

Also, you can look at the phi_convict_threshold Cassandra.yaml parameter to 
help with network flakiness. Default is 8, I believe in most versions. 
Increasing to 10 or 12 might help.

Finally, a 2 node cluster doesn't give you many of the benefits of a 
distributed system. Consider adding some nodes, if you can.

Sean Durity - Lead Cassandra Admin
Big DATA Team
For support, create a 
JIRA

From: Michael Fong [mailto:michael.f...@ruckuswireless.com]
Sent: Wednesday, March 23, 2016 4:12 AM
To: user@cassandra.apache.org
Subject: Gossip heartbeat and packet capture

Hi, all,


We are trying to reason the possible scenarios when a C*(v1.x) cluster 
connection keeps flapping in production. (Two node cluster, each node keeps 
marking the other node DOWN but came back UP within seconds; multiple times a 
day) We have checked the load on the cluster i- very light and low GC 
activities also. We have also checked the network interface / devices were 
working just fine on the nodes during the incidence. We are changing our 
investigation direction to the network topology/settings, so we are thinking to 
capture gossip heartbeat packet to verify if the packet is received as expected 
on the other end.

Has anyone tried to capture the packet of gossip internode communication? What 
would be the filter / criteria to grep heartbeat-related packet only?

Thanks in advance!


Michael



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Modeling transactional messages

[SEAN_R_DURITY]

As you have it, this is not a good model for Cassandra. Your partition key has 
only 2 specific values. You would end up with only 2 partitions (perhaps owned 
by just 2 nodes) that would quickly get huge (and slow). Also, secondary 
indexes are generally a bad idea. You would either want to create new table to 
support additional queries or look at the materialized views in the 3.x 
versions.

You are setting up something like a queue, which is typically an anti-pattern 
for Cassandra.

However, I will at least toss out an idea for the rest of the community to 
improve (or utterly reject):

You could have an unsent mail table and a sent mail table.
For unsent mail, just use the objectID as the partition key. The drivers can 
page through results, though if it gets very large, you might see problems. 
Delete the row from unsent mail once it is sent. Try leveled compaction with a 
short gc_grace. There would be a lot of churn on this table, so it may still be 
less than ideal.

Then you could do the sent email table with objectID and all the email details. 
Add separate lookup tables for:
- (emailaddr), object ID (if this is going to be large/wide, perhaps add a time 
bucket to the partition key, like mm)
- (domain, time bucket), objectID

Set TTL on these rows (either default or with the insert) to get the purge to 
be automatic.


Sean Durity

From: I PVP [mailto:i...@hotmail.com]
Sent: Thursday, March 03, 2016 7:51 PM
To: user@cassandra.apache.org
Subject: Modeling transactional messages

Hi everyone,

Can anyone please let me know if I am heading to an antiparttern or 
somethingelse bad?

How would you model the following ... ?

I am migrating from MYSQL to Cassandra, I have a scenario in which need to 
store the content of "to be sent" transactional email messages that the 
customer will receive on events like : an order was created, an order was 
updated, an order was canceled,an order was  shipped,an account was created, an 
account was confirmed, an account was locked and so on.

On MYSQL there is table for email message "type", like: a table to store 
messages of "order-created”, a table to store messages of "order-updated" and 
so on.

The messages are sent by a non-parallelized java worker, scheduled to run every 
X seconds, that push the messages to a service like Sendgrid/Mandrill/Mailjet.

For better performance, easy to purge and overall code maintenance I am looking 
to have all message "types" on a single table/column family as following:

CREATE TABLE communication.transactional_email (
objectid timeuuid,
subject text,
content text,
fromname text,
fromaddr text,
toname text,
toaddr text,
wassent boolean,
createdate timestamp,
sentdate timestamp,
type text,// example: order_created, order_canceled
domain text, // exaple: hotmail.com. in case need to stop sending to a specific 
domain
PRIMARY KEY (wassent, objectid)
);

create index on toaddr
create index on sentdate
create index on domain
create index on type


The requirements are :

1) select * from transactional_email where was_sent = false and objectid < 
minTimeuuid(current timestamp) limit 

(to get the messages that need to be sent)

2) update transactional_email set was_sent = true where objectid = 

(to update the message  right after it was sent)

3) select * from transactional_email where toaddr = 

(to get all messages that were sent to a specific emailaddr)

4) select * from transactional_email where domain = 

(to get all messages that were sent to a specific domain)

5) delete from transactional_email where was_sent = true and objectid < 
minTimeuuid(a timestamp)

(to do purge, delete all messages send before the last X days)

6) delete from transactional_email where toaddr = 

(to be able to delete all messages when a user account is closed)


Thanks

IPVP



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Inconsistent Reads after Restoring Snapshot

[SEAN_R_DURITY]

What about the commitlogs? Are you saving those off anywhere in between the 
snapshot and the crash?


Sean Durity

From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Monday, April 25, 2016 10:26 PM
To: User
Subject: Inconsistent Reads after Restoring Snapshot

Hi,

We have 2.0.14. We use RF=3 and read/write at Quorum. Moreover, we dont use 
incremental backups. As per the documentation at 
https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html
 , if i need to restore a Snapshot on SINGLE node in a cluster, I would run 
repair at the end. But while the repair is going on, reads may get inconsistent.


Consider following scenario:
10 AM Daily Snapshot taken of node A and moved to backup location
11 AM A record is inserted such that node A and B insert the record but there 
is a mutation drop on node C.
1 PM Node A crashes and data is restored from latest 10 AM snapshot. Now, only 
Node B has the record.

Now, my question is:

Till the repair is completed on node A,a read at Quorum may return inconsistent 
result based on the nodes from which data is read.If data is read from node A 
and node C, nothing is returned and if data is read from node A and node B, 
record is returned. This is a vital point which is not highlighted anywhere.


Please confirm my understanding.If my understanding is right, how to make sure 
that my reads are not inconsistent while a node is being repair after restoring 
a snapshot.

I think, autobootstrapping the node without joining the ring till the repair is 
completed, is an alternative option. But snapshots save lot of streaming as 
compared to bootstrap.

Will incremental backups guarantee that

Thanks
Anuj


Sent from Yahoo Mail on 
Android



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Inconsistent Reads after Restoring Snapshot

[SEAN_R_DURITY]

https://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configLogArchive_t.html

Sean Durity

From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Wednesday, April 27, 2016 10:44 PM
To: user@cassandra.apache.org
Subject: RE: Inconsistent Reads after Restoring Snapshot

No.We are not saving them.I have never read that in DataStax documentation.

Thanks
Anuj
Sent from Yahoo Mail on 
Android

On Thu, 28 Apr, 2016 at 12:45 AM, 
sean_r_dur...@homedepot.com
> wrote:
What about the commitlogs? Are you saving those off anywhere in between the 
snapshot and the crash?


Sean Durity

From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Monday, April 25, 2016 10:26 PM
To: User
Subject: Inconsistent Reads after Restoring Snapshot

Hi,

We have 2.0.14. We use RF=3 and read/write at Quorum. Moreover, we dont use 
incremental backups. As per the documentation at 
https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html
 , if i need to restore a Snapshot on SINGLE node in a cluster, I would run 
repair at the end. But while the repair is going on, reads may get inconsistent.


Consider following scenario:
10 AM Daily Snapshot taken of node A and moved to backup location
11 AM A record is inserted such that node A and B insert the record but there 
is a mutation drop on node C.
1 PM Node A crashes and data is restored from latest 10 AM snapshot. Now, only 
Node B has the record.

Now, my question is:

Till the repair is completed on node A,a read at Quorum may return inconsistent 
result based on the nodes from which data is read.If data is read from node A 
and node C, nothing is returned and if data is read from node A and node B, 
record is returned. This is a vital point which is not highlighted anywhere.


Please confirm my understanding.If my understanding is right, how to make sure 
that my reads are not inconsistent while a node is being repair after restoring 
a snapshot.

I think, autobootstrapping the node without joining the ring till the repair is 
completed, is an alternative option. But snapshots save lot of streaming as 
compared to bootstrap.

Will incremental backups guarantee that

Thanks
Anuj


Sent from Yahoo Mail on 
Android



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Replication lag between data center

[SEAN_R_DURITY]

Just wanted to chime in that this is a very well-written and explained answer. 
Nice job, Jeff!

Sean Durity
From: Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com]
Sent: Wednesday, May 18, 2016 11:41 PM
To: user@cassandra.apache.org
Subject: Re: Replication lag between data center

Cassandra isn’t a traditional DB – it doesn’t “replicate” in the same way that 
a relational DB replicas.

Cassandra clients send mutations (via native protocol or thrift). Those 
mutations include a minimum consistency level for the server to return a 
successful write.

If a write says “Consistency: ALL” - then as soon as the write returns, the 
mutation exists on all nodes (no replication delay – it’s done).
If a write is anything other than ALL, it’s possible that any individual node 
may not have the write when the client is told the write succeeds. At that 
point, the coordinator will make a best effort to deliver the write to all 
nodes in real time, but may fail or time out. As far as I know, there are no 
metrics on this delivery – I believe the writes prior to the coordinator 
returning may have some basic data in TRACE, but wouldn’t expect writes after 
the coordinator returned to have tracing data available.

If any individual times out completely, the coordinator writes a hint. When the 
coordinator sees the node come back online, it will try to replay the writes by 
replaying the hints – this may happen minutes or hours later.

If it’s unable to replay hints, or if writes are missed for some other reason, 
the data may never “replicate” to the other nodes/Dcs on its own – you may need 
to manually “replicate” it using the `nodetool repair` tool.

Taken together, there’s no simple “replication lag” here – if you write with 
ALL, the lag is “none”. If you write with CL:QUORUM and read with CL:QUORUM, 
your effective lag is “probably none”, because missing replicas will 
read-repair the data on read. If you read or write with low consistency, your 
lag may be milliseconds, hours, weeks, or forever, depending on how long your 
link is down and how often you repair.



From: cass savy
Reply-To: "user@cassandra.apache.org"
Date: Wednesday, May 18, 2016 at 8:03 PM
To: "user@cassandra.apache.org"
Subject: Replication lag between data center

How can we determine/measure the replication lag or latency between on premise 
data centers or cross region/Availability zones?



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Set up authentication on a live production cluster

[SEAN_R_DURITY]

Do the clients already send the credentials? That is the first thing to address.

Setting up a cluster for authentication (and authorization) requires a restart 
with the properties turned on in cassandra.yaml. However, the actual keyspace 
(system_auth) and tables are not created until the last node is restarted with 
the parameters changed. So, as you are changing each node, what you get is 
individual nodes that are requiring a password, but have no system_auth 
keyspace to authenticate against. Thus, clients cannot connect to these nodes.

With open source Cassandra you cannot implement authentication without at least 
a brief degradation of service (as nodes can’t authenticate) and an outage 
(while the keyspace and tables are created, users are created, and permissions 
are granted). The outage can be relatively brief, depending on cluster size, 
CL, speed to restart, etc.

With DataStax Enterprise, there is a TransitionalAuthenticator (and Authorizer) 
that lets you implement security without a full outage. You basically switch to 
the Transitional classes so that system_auth gets created. You create all your 
security objects. Then you switch to PasswordAuthenticator and 
CassandraAuthorizer. It takes two rolling bounces to get it done, but no outage.

I have done both of the above. The DataStax stuff is very helpful, when 
downtime is a concern. Perhaps you could write your own implementation of the 
various interfaces to do something like TransitionalAuthenticator, but we have 
seen that the security interfaces change, so you will probably break/rewrite in 
later versions. (For one-time use, maybe it is worth a shot?)

For anyone setting up new clusters, just start with security turned on so that 
you don’t end up in the It’s-Production-Can’t-Stop quandary above.


Sean Durity

From: Vigneshwaran [mailto:vigneshwaran2...@gmail.com]
Sent: Wednesday, April 13, 2016 3:36 AM
To: user@cassandra.apache.org
Subject: Set up authentication on a live production cluster

Hi,

I have setup a 16 node cluster (8 per DC; C* 2.2.4) up and running in our 
production setup. We use Datastax Java driver 2.1.8.

I would like to set up Authentication and Authorization in the cluster without 
breaking the live clients.

From the references I found by googling, I can setup credentials for a new 
cluster. But it is not clear to me what steps I should take for setting up 
credentials in an already running cluster without breaking existing clients.

Can someone clarify me or link me to a reference I may have missed? I'd really 
appreciate it.

Thank you,
Vigneshwaran



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: setting ttl using timestamp field of record during insertion in cassandra

[SEAN_R_DURITY]

Perhaps you can set the default TTL when you create the table instead of on 
every insert:
http://docs.datastax.com/en/cql/3.1/cql/cql_reference/tabProp.html
(see default_time_to_live property)

Then no need for UDF.


Sean Durity
lord of the (C*) rings (Lead Cassandra Admin)
Big DATA Team
MTC 2250
#cassandra - for the latest news and updates

From: Sujeet Kumar [mailto:genioussuj...@gmail.com]
Sent: Monday, August 08, 2016 11:18 AM
To: user@cassandra.apache.org
Subject: setting ttl using timestamp field of record during insertion in 
cassandra

Hi

I want to set TTL 30 days from the time field of table record(id,name,time) 
during insertion.For this I am creating a User Defined Function(UDF) bigint 
fun(rTime,cTime) as

CREATE FUNCTION fun(rtime timestamp,ctime timestamp) CALLED ON NULL INPUT 
RETURNS bigint LANGUAGE java as 'return 2592000-((ctime.toTime() 
-rtime.toTime())/1000);';

here,function fun is calculating the time in seconds this data should live.
2592000 is the time in seconds for 30 days.

I am trying to use above function for setting TTL as

INSERT INTO record(id,name,time) VALUES (123,'data123','2016-08-08 06:06:00') 
USING TTL fun('2016-08-08 06:06:00',totimestamp(now()));

getting error as
Syntax Exception: ErrorMessage code=2000 

Is there any other way to set ttl based on record time field. What is problem 
with above approach.

Thanks.

Sujeet

--

--
Sujeet Kumar
ME(CSA)
IISc Banglore
sujeet.ku...@csa.iisc.ernet.in
sujeet.cs...@mnnit.ac.in



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Set up authentication on a live production cluster

[SEAN_R_DURITY]

Thanks for the updates on later versions. My experience on authentication was 
mostly 1.1 – 2.0. I am glad that it is improving a bit. However, it does seem 
that it is still wise to start rings with authentication on to avoid this 
activation procedure.


Sean Durity

From: li...@beobal.com [mailto:li...@beobal.com] On Behalf Of Sam Tunnicliffe
Sent: Tuesday, August 02, 2016 1:00 PM
To: user@cassandra.apache.org
Subject: Re: Set up authentication on a live production cluster

However, the actual keyspace (system_auth) and tables are not created until the 
last node is restarted with the parameters changed

Actually, this is not strictly true. On 2.2+ the tables in system_auth are 
created up front, regardless of the auth config. Practically you can't go about 
setting up your roles until all nodes in the cluster are on 2.2 or higher, but 
if they are, then you can.

With open source Cassandra you cannot implement authentication without at least 
a brief degradation of service (as nodes can’t authenticate) and an outage 
(while the keyspace and tables are created, users are created, and permissions 
are granted).

This is also not 100% accurate. Using a modern driver, your clients can be 
configured with credentials before the cluster requires them. Drivers will not 
send those credentials unless the server they're connecting to asks for them. 
So provided you modify your clients to begin sending credentials before turning 
authentication on you can enable it without downtime.

The ugly part is that you need to enable auth on at least one node in order to 
set up the roles in the system. This is obviously racy as clients may start 
connecting to that node as soon as you've enabled authentication but before 
you've added all necessary roles. However, if you can stop clients connecting 
to that node (maybe via iptables), then you won't hit that problem. Once all 
the necessary roles and credentials are configured, you can re-enable client 
connections to it and enable authentication on the rest of the cluster in a 
rolling fashion (i.e. stop node, modify yaml, restart node) and you shouldn't 
encounter any downtime.

This is now covered in the latest docs, which are targeted at 3.x but this in 
case they apply equally to 2.2.
http://cassandra.apache.org/doc/latest/operating/security.html#authentication


On Tue, Aug 2, 2016 at 5:21 PM, Jai 
> wrote:
I have done it in production without downtime on apache cassandra by 
manipulating the user creation using iptables on first node.

Sent from my iPhone

On Aug 2, 2016, at 9:11 PM, DuyHai Doan 
> wrote:
Thank you Sean for the excellent and details explanation, a lot of people out 
there start their Cassandra in production without security and wake up some 
days, too late

On Wed, Apr 13, 2016 at 10:54 PM, 
> wrote:
Do the clients already send the credentials? That is the first thing to address.

Setting up a cluster for authentication (and authorization) requires a restart 
with the properties turned on in cassandra.yaml. However, the actual keyspace 
(system_auth) and tables are not created until the last node is restarted with 
the parameters changed. So, as you are changing each node, what you get is 
individual nodes that are requiring a password, but have no system_auth 
keyspace to authenticate against. Thus, clients cannot connect to these nodes.

With open source Cassandra you cannot implement authentication without at least 
a brief degradation of service (as nodes can’t authenticate) and an outage 
(while the keyspace and tables are created, users are created, and permissions 
are granted). The outage can be relatively brief, depending on cluster size, 
CL, speed to restart, etc.

With DataStax Enterprise, there is a TransitionalAuthenticator (and Authorizer) 
that lets you implement security without a full outage. You basically switch to 
the Transitional classes so that system_auth gets created. You create all your 
security objects. Then you switch to PasswordAuthenticator and 
CassandraAuthorizer. It takes two rolling bounces to get it done, but no outage.

I have done both of the above. The DataStax stuff is very helpful, when 
downtime is a concern. Perhaps you could write your own implementation of the 
various interfaces to do something like TransitionalAuthenticator, but we have 
seen that the security interfaces change, so you will probably break/rewrite in 
later versions. (For one-time use, maybe it is worth a shot?)

For anyone setting up new clusters, just start with security turned on so that 
you don’t end up in the It’s-Production-Can’t-Stop quandary above.


Sean Durity

From: Vigneshwaran 
[mailto:vigneshwaran2...@gmail.com]
Sent: Wednesday, April 13, 2016 3:36 AM
To: user@cassandra.apache.org
Subject: 

RE: Current data density limits with Open Source Cassandra

[SEAN_R_DURITY]

I request 1-2 TB of disk per node, depending on how large the data is estimated 
to be (for larger data, 2 TB). I have some dense nodes (4+ TB of disk 
available). They are harder to manage for repairs, bootstrapping, compaction, 
etc. because it takes so long to stream the data, etc. For the actual 
application, I have not seen a great impact based on the size of disk available.


Sean Durity

From: daemeon reiydelle [mailto:daeme...@gmail.com]
Sent: Wednesday, February 08, 2017 10:56 PM
To: user@cassandra.apache.org
Subject: Re: Current data density limits with Open Source Cassandra

your MMV. Think of that storage limit as fairly reasonable for active data 
likely to tombstone. Add more for older/historic data. Then think about time to 
recover a node.


...

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872

On Wed, Feb 8, 2017 at 2:14 PM, Ben Slater 
> wrote:
The major issue we’ve seen with very high density (we generally say <2TB node 
is best) is manageability - if you need to replace a node or add node then 
restreaming data takes a *long* time and there we fairly high chance of a 
glitch in the universe meaning you have to start again before it’s done.

Also, if you’re uses STCS you can end up with gigantic compactions which also 
take a long time and can cause issues.

Heap limitations are mainly related to partition size rather than node density 
in my experience.

Cheers
Ben

On Thu, 9 Feb 2017 at 08:20 Hannu Kröger 
> wrote:
Hello,

Back in the day it was recommended that max disk density per node for Cassandra 
1.2 was at around 3-5TB of uncompressed data.

IIRC it was mostly because of heap memory limitations? Now that off-heap 
support is there for certain data and 3.x has different data storage format, is 
that 3-5TB still a valid limit?

Does anyone have experience on running Cassandra with 3-5TB compressed data ?

Cheers,
Hannu
--

Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Troubleshooting random node latency spikes

[SEAN_R_DURITY]

Is this Java 8 with the G1 garbage collector or CMS? With Java 7 and CMS, 
garbage collection can cause delays like you are seeing. I haven’t seen that 
problem with G1, but garbage collection is where I would start looking.


Sean Durity
From: Ted Pearson [mailto:t...@tedpearson.com]
Sent: Thursday, January 05, 2017 2:34 PM
To: user@cassandra.apache.org
Subject: Troubleshooting random node latency spikes


Greetings!

I'm working on setting up a new cassandra cluster with a write-heavy workload 
(50% writes), and I've run into a strange spiky latency problem. My application 
metrics showed random latency spikes. I tracked the latency back to spikes on 
individual cassandra nodes. ClientRequest.Latency.Read/Write.p99 is 
occasionally jumping on one node at a time to several seconds, instead of its 
normal value of around 1000 microseconds. I also noticed that 
ReadRepair.RepairedBackground.m1_rate goes from zero to a non-zero (around 
1-2/sec) during the spike on that node. I'm lost as to why these spikes are 
happening, hope someone can give me ideas.

I attempted to test if the ReadRepair metric is causally linked to the latency 
spikes, but even when I changed dclocal_read_repair_chance to 0 on my tables, 
even though the metrics showed no ReadRepair.Attempted, the 
ReadRepair.RepairedBackground metric still went up during latency spikes. Am I 
misunderstanding what this metric tracks? I don't understand why it went up if 
I turned off read repair.

I'm currently running 2.2.6 in a dual-datacenter setup. It's patched to allow 
metrics to be recency-biased instead of tracking latency over the entire 
running of the java process. I'm using STCS. There is a large amount of data 
per node, about 500GB currently. I expect each row to be less than 10KB. It's 
currently running on way overpowered hardware - 512GB/raid 0 on nvme/44 cores 
on 2 sockets. All of my queries (reads and writes) are LOCAL_ONE and I'm using 
r=3.



Thanks,

Ted



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Announcement: Atlanta Meetup, January 10th

[SEAN_R_DURITY]

Down and Durity - Cassandra Admin Discussion

Now, you are running several Cassandra clusters (or leaning heavily that way). 
How do you deploy them, monitor them, and do various other administrative 
tasks? Come and join in a discussion and let's learn from each other.

Sean Durity, our facilitator for the evening, is a Lead Cassandra Administrator 
(aka "lord of the (C*) rings") from The Home Depot, where he administers 
several production clusters. They range from 6-112 nodes, and he has worked on 
Cassandra versions 1.0.8 - 2.1 (and the accompanying DataStax Enterprise 
versions). He will share tips and tools from his 3+ years of hands-on Cassandra 
experience and hope to learn even more from the rest of the group. Expect a 
lively discussion with many take-aways.

More details or RSVP:
https://www.meetup.com/atlcassandra/events/236498977/


Sean Durity




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Query

[SEAN_R_DURITY]

A few of the many companies that rely on Cassandra are mentioned here:
http://cassandra.apache.org
Apple, Netflix, Weather Channel, etc.
(Not nearly as good as the Planet Cassandra list that DataStax used to 
maintain. Boo for the Apache/DataStax squabble!)

DataStax has a list of many case studies, too, with their enterprise version of 
Cassandra:
http://www.datastax.com/resources/casestudies


Sean Durity

From: Sikander Rafiq [mailto:hafiz_ra...@hotmail.com]
Sent: Friday, December 30, 2016 8:00 AM
To: user@cassandra.apache.org
Subject: Re: Query


Thanks for your comments/suggestions.



Yes I understand my project needs and requirements. Surely it requires to 
handle huge data for what i'm exploring what suits for it.



Though Cassandra is distributed, scalable and highly available, but it is NoSql 
means Sql part is missing and needs to be handled.



Can anyone please tell me some big name who is using Cassandra for handling its 
huge data sets like Twitter etc.





Sent from Outlook


From: Edward Capriolo >
Sent: Friday, December 30, 2016 5:53 AM
To: user@cassandra.apache.org
Subject: Re: Query

You should start with understanding your needs. Once you understand your need 
you can pick the software that fits your need. Staring with a software stack is 
backwards.

On Thu, Dec 29, 2016 at 11:34 PM, Ben Slater 
> wrote:
I wasn't familiar with Gizzard either so I thought I'd take a look. The first 
things on their github readme is:
NB: This project is currently not recommended as a base for new consumers.
(And no commits since 2013)

So, Cassandra definitely looks like a better choice as your datastore for a new 
project.

Cheers
Ben

On Fri, 30 Dec 2016 at 12:41 Manoj Khangaonkar 
> wrote:
I am not that familiar with gizzard but with gizzard + mysql , you have 
multiple moving parts in the system that need to managed separately. You'll 
need the mysql expert for mysql and the gizzard expert to manage the 
distributed part. It can be argued that long term this will have higher 
adminstration cost
Cassandra's value add is its simple peer to peer architecture that is easy to 
manage - a single database solution that is distributed, scalable, highly 
available etc. In other words, once you gain expertise cassandra, you get 
everything in one package.
regards




On Thu, Dec 29, 2016 at 4:05 AM, Sikander Rafiq 
> wrote:

Hi,

I'm exploring Cassandra for handling large data sets for mobile app, but i'm 
not clear where it stands.



If we use MySQL as  underlying database and Gizzard for building custom 
distributed databases (with arbitrary storage technology) and Memcached for 
highly queried data, then where lies Cassandra?



As i have read that Twitter uses both Cassandra and Gizzard. Please explain me 
where Cassandra will act.



Thanks in advance.



Regards,

Sikander




Sent from Outlook


--
http://khangaonkar.blogspot.com/




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this 

RE: Growing Hints

[SEAN_R_DURITY]

Version number may help.

Sean Durity

From: Anshu Vajpayee [mailto:anshu.vajpa...@gmail.com]
Sent: Tuesday, January 03, 2017 10:09 AM
To: user@cassandra.apache.org
Subject: Re: Growing Hints


Anyone aware about  issue ?

Hints are still growing although gossip and repair was successfull. Gossip is 
flowing without any issue as all nodes status is UN.

Hints are growing and there is timed out message in log hinted handoff 
delivery.  Hints are not truncating after defined time period.

Please let me know if you have any question. Thanks






On Dec 29, 2016 10:06 AM, "Anshu Vajpayee" 
> wrote:
Hello All
We have one unusual issue on our cluster. We are seeing growing hints table on  
node although all the nodes are up  and coming online with notetool status.

I know  Cassandra  appends the hints in case if there is  write timeout for 
other nodes. In our case  all nodes are up and functional , Gossip is also 
flowing well.  Also write time out value is quite high in our cluster .
Can anyone suggest what could be other possible reason for these growing hints ?








The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: Query on Cassandra clusters

[SEAN_R_DURITY]

A couple thoughts (for after you up/downgrade to one version for all nodes):

-  16 GB of total RAM on a node is a minimum I would use; 32 would be 
much better

-  With a lower amount of memory, I think would keep memtables on-heap 
in order to keep a tighter rein on how much they use. If you are consistently 
using 75% or more of heap space, you need more (either more nodes or more 
memory per node).

-  I would try giving Cassandra 50% of the RAM on the host. And remove 
any client or non-Cassandra processes. Nodes should be dedicated to Cassandra 
(for Production)

-  For disk, my rule for size-tiered is that you need 50% overhead IF 
it is primarily a single table application (90%+ of data in one table). 
Otherwise, I am ok with 35-40% overhead. Just know you can hit issues down the 
road as the sstables get larger.


Sean Durity
From: Sumit Anvekar [mailto:sumit.anve...@gmail.com]
Sent: Wednesday, December 21, 2016 3:47 PM
To: user@cassandra.apache.org
Subject: Re: Query on Cassandra clusters

Thank you Alain for the detailed explanation.
To answer you question on Java version, JVM settings and Memory usage. We are 
using using 1.8.0_45. precisely
>java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
JVM settings are identical on all nodes (cassandra-env.sh is identical).
Further when I say high on memory usage, Cassandra is using heap (-Xmx3767M) 
and off heap of about 6GB out of the total system memory of 14.7 GB. Along with 
this there are other processes running on this system which is bring the 
overall memory usage to >95%. This bring me to another point whether heap 
memory + off heap (sum of values of Space used (total)) from nodetool cfstats 
is the total memory used by Cassandra on a node?
Also, on the disk front, what is a good amount of empty space to be left out 
unused in the partition(~50%
 should be?) considering we use SizeTieredCompaction strategy?

On Wed, Dec 21, 2016 at 6:30 PM, Alain RODRIGUEZ 
> wrote:
Hi Sumit,

1. I have a Cassandra cluster with 11 nodes, 5 of which have Cassandra version 
3.0.3 and then newer 5 nodes have 3.6.0 version.

I strongly recommend to:


  *   Stick with one version of Apache Cassandra per cluster.
  *   Always be as close as possible from the last minor release of the 
Cassandra version in use.

So you really should not be using 3.0.6 AND 3.6.0 but rather 3.0.10 OR 3.7 
(currently). Note that Cassandra 3.X (with X > 0) uses a tic toc release cycle 
where odd are bug fixes only and even numbers introduce new features as well.

Running multiple version for a long period can induces errors, Cassandra is 
built to handle multiple versions only to give the time to operators to run a 
rolling restart. No streaming (adding / removing / repairing nodes) should 
happen during this period. Also, I have seen in the past some cases where 
changing the schema was also an issue with multiple versions leading to schema 
disagreements.

Due to this scenario, a couple boxes are running very high on memory (95% 
usage) whereas some of the older version nodes have just 60-70% memory usage.

Hard to say if this is related to the mutiple versions of Cassandra but it 
could. Are you sure nodes are using the same JVM / GC options 
(cassandra-env.sh) and Java version?

Also, what is exactly "high on memory 95%"? Are we talking about heap or Native 
memory. Isn't the memory used as page cache (that would still be available for 
the system)?

2. To counter #1, I am planning to upgrade system configuration of the nodes 
where there is higher memory usage. But the question is, will it be a problem 
if we have a Cassandra cluster, where in a couple of nodes have double the 
system configuration than other nodes in the cluster.

It is not a problem per se to have distinct configurations on distinct nodes. 
Cassandra does it very well, and it is frequently used to test some 
configuration change on a canary node, to prevent it from impacting the whole 
service.

Yet, all the nodes should be doing the same work (unless you have some 
heterogenous hardware and are using distinct number of vnodes on each node). 
Keeping things homogenous allows the operator to easily compare how nodes are 
doing and it makes reasoning about Cassandra, as well as troubleshooting issues 
a way easier.

So I would:

- Fully upgrade / downgrade asap to a chosen version (3.X is known as being not 
yet stable, but going back to 3.0.X might be more painful)
- Make sure nodes are well balanced and using the same number of ranges 
'nodetool status '
- Make sure the node are using the same Java version and JVM settings.

Hope that helps,

C*heers,
---
Alain Rodriguez - @arodream - 
al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting