Re: [EXTERNAL] Re: About Cassandra stable version having Java 17 support

[Bowen Song via user]

Short answer:

There's no definite answer to that question.


Longer answer:

I doubt such date has already been decided. It's largely driven by the 
time required to fix known issues and any potential new issues 
discovered during the BETA and RC process. If you want to track the 
progress, feel free to look at the project's Jira boards, there's a 5.0 
GA board dedicated for that.


Furthermore, it's likely there will only be an experimental support for 
Java 17 in Cassandra 5.0, which means it shouldn't be used on production 
environments.


So, would you like to keep waiting indefinitely for the Java 17 official 
support, or run Cassandra 4.1 on Java 11 today and upgrade when newer 
version becomes available?



On 18/03/2024 13:10, Divyanshi Kaushik via user wrote:

Thanks for your reply.

As Cassandra has moved to Java 17 in it's *5.0-BETA1* (Latest release 
on 2023-12-05). Can you please let us know when the team is planning 
to GA Cassandra 5.0 version which has Java 17 support?


Regards,
Divyanshi

*From:* Bowen Song via user 
*Sent:* Monday, March 18, 2024 5:14 PM
*To:* user@cassandra.apache.org 
*Cc:* Bowen Song 
*Subject:* [EXTERNAL] Re: About Cassandra stable version having Java 
17 support


*CAUTION:* This email originated from outside the organization. Do not 
click links or open attachments unless you recognize the sender and 
know the content is safe.


Why Java 17? It makes no sense to choose an officially non-supported 
library version for a piece of software. That decision making process 
is the problem, not the software's library version compatibility.



On 18/03/2024 09:44, Divyanshi Kaushik via user wrote:

Hi All,

As per my project requirement, Java 17 needs to be used. Can you
please let us know when you are planning to release the next
stable version of Cassandra having Java 17 support?

Regards,
Divyanshi
This email and any files transmitted with it are confidential,
proprietary and intended solely for the individual or entity to
whom they are addressed. If you have received this email in error
please delete it immediately.

This email and any files transmitted with it are confidential, 
proprietary and intended solely for the individual or entity to whom 
they are addressed. If you have received this email in error please 
delete it immediately. 

Re: [EXTERNAL] Re: About Cassandra stable version having Java 17 support

[Divyanshi Kaushik via user]

Thanks for your reply.

As Cassandra has moved to Java 17 in it's 5.0-BETA1 (Latest release on 
2023-12-05). Can you please let us know when the team is planning to GA 
Cassandra 5.0 version which has Java 17 support?

Regards,
Divyanshi

From: Bowen Song via user 
Sent: Monday, March 18, 2024 5:14 PM
To: user@cassandra.apache.org 
Cc: Bowen Song 
Subject: [EXTERNAL] Re: About Cassandra stable version having Java 17 support


CAUTION: This email originated from outside the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

Why Java 17? It makes no sense to choose an officially non-supported library 
version for a piece of software. That decision making process is the problem, 
not the software's library version compatibility.


On 18/03/2024 09:44, Divyanshi Kaushik via user wrote:
Hi All,

As per my project requirement, Java 17 needs to be used. Can you please let us 
know when you are planning to release the next stable version of Cassandra 
having Java 17 support?

Regards,
Divyanshi
This email and any files transmitted with it are confidential, proprietary and 
intended solely for the individual or entity to whom they are addressed. If you 
have received this email in error please delete it immediately.
This email and any files transmitted with it are confidential, proprietary and 
intended solely for the individual or entity to whom they are addressed. If you 
have received this email in error please delete it immediately.

RE: [EXTERNAL] Re: Running and Managing Large Cassandra Clusters

[Gediminas Blazys]

Hey,

@Tom I just wanted to clarify when I mention workload in the last email I meant 
it as the amount of requests the cluster has to serve.

Gediminas

From: Tom van der Woerdt 
Sent: Wednesday, October 28, 2020 14:35
To: user 
Subject: [EXTERNAL] Re: Running and Managing Large Cassandra Clusters

Heya,

We're running version 3.11.7, can't use 3.11.8 as it won't even start 
(CASSANDRA-16091). Our policy is to use LCS for everything unless there's a 
good argument for a different compaction strategy (I don't think we have *any* 
STCS at all other than system keyspaces). Since our nodes are mostly on-prem 
they are generally oversized on cpu count, but when idle the cluster with 360 
nodes ends up using less than two cores *peak* for background tasks like (full, 
weekly) repairs and tombstone compactions. That said they do get 32 logical 
threads because that's what the hardware ships with (-:

Haven't had major problems with Gossip over the years. I think we've had to run 
nodetool assassinate exactly once, a few years ago. Probably the only gossip 
related annoyance is that when you decommission all seed nodes Cassandra will 
happily run a single core at 100% trying to connect until you update the list 
of seeds, but that's really minor.

There's also one cluster that has 50TB nodes, 60 of them, storing reasonably 
large cells (using LCS, previously TWCS, both fine). Replacing a node takes a 
few days, but other than that it's not particularly problematic.

In my experience it's the small clusters that wake you up ;-)

Tom van der Woerdt
Senior Site Reliability Engineer
Booking.com BV
Vijzelstraat Amsterdam Netherlands 1017HL
[Booking.com]
Making it easier for everyone to experience the world since 1996
43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 million 
reported listings
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)


On Wed, Oct 28, 2020 at 12:32 PM Joshua McKenzie 
mailto:jmcken...@apache.org>> wrote:

A few questions for you Tom if you have 30 seconds and care to disclose:

  1.  What version of C*?
  2.  What compaction strategy?
  3.  What's core count allocated per C* node?
  4.  Gossip give you any headaches / you have to be delicate there or does it 
behave itself?
Context: pmc/committer and I manage the OSS C* team at DataStax. We're doing a 
lot of thinking about how to generally improve the operator experience across 
the board for folks in the post 4.0 time frame, so data like the above (where 
things are going well at scale and why) is super useful to help feed into that 
effort.

Thanks!



On Wed, Oct 28, 2020 at 7:14 AM, Tom van der Woerdt 
mailto:tom.vanderwoe...@booking.com.invalid>>
 wrote:
Does 360 count? :-)

num_tokens is 16, works fine (had 256 on a 300 node cluster as well, not too 
many problems either). Roughly 2.5TB per node, running on-prem on reasonably 
stable hardware so replacements end up happening once a week at most, and 
there's no particular change needed in the automation. Scaling up or down takes 
a while, but it doesn't appear to be slower than any other cluster. 
Configuration wise it's no different than a 5-node cluster either. Pretty 
uneventful tbh.

Tom van der Woerdt
Senior Site Reliability Engineer
Booking.com
 BV
Vijzelstraat Amsterdam Netherlands 1017HL
[Booking.com]
Making it easier for everyone to experience the world since 1996
43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 million 
reported listings
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)


On Wed, Oct 28, 2020 at 8:58 AM Gediminas Blazys 
mailto:gediminas.bla...@microsoft.com.invalid>>
 wrote:
Hello,

I wanted to seek out your opinion and experience.

Has anyone of you had a chance to run a Cassandra cluster of more than 350 
nodes?
What are the major configuration considerations that you had to focus on? What 
number of 

RE: [EXTERNAL] Re: Running and Managing Large Cassandra Clusters

[Gediminas Blazys]

Hey,

Thanks chipping in Tomas. Could you describe what sort of workload is the big 
cluster receiving in terms of local C* reads, writes and client requests as 
well?

You mention repairs, how do you run them?

Gediminas

From: Tom van der Woerdt 
Sent: Wednesday, October 28, 2020 14:35
To: user 
Subject: [EXTERNAL] Re: Running and Managing Large Cassandra Clusters

Heya,

We're running version 3.11.7, can't use 3.11.8 as it won't even start 
(CASSANDRA-16091). Our policy is to use LCS for everything unless there's a 
good argument for a different compaction strategy (I don't think we have *any* 
STCS at all other than system keyspaces). Since our nodes are mostly on-prem 
they are generally oversized on cpu count, but when idle the cluster with 360 
nodes ends up using less than two cores *peak* for background tasks like (full, 
weekly) repairs and tombstone compactions. That said they do get 32 logical 
threads because that's what the hardware ships with (-:

Haven't had major problems with Gossip over the years. I think we've had to run 
nodetool assassinate exactly once, a few years ago. Probably the only gossip 
related annoyance is that when you decommission all seed nodes Cassandra will 
happily run a single core at 100% trying to connect until you update the list 
of seeds, but that's really minor.

There's also one cluster that has 50TB nodes, 60 of them, storing reasonably 
large cells (using LCS, previously TWCS, both fine). Replacing a node takes a 
few days, but other than that it's not particularly problematic.

In my experience it's the small clusters that wake you up ;-)

Tom van der Woerdt
Senior Site Reliability Engineer
Booking.com BV
Vijzelstraat Amsterdam Netherlands 1017HL
[Booking.com]
Making it easier for everyone to experience the world since 1996
43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 million 
reported listings
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)


On Wed, Oct 28, 2020 at 12:32 PM Joshua McKenzie 
mailto:jmcken...@apache.org>> wrote:

A few questions for you Tom if you have 30 seconds and care to disclose:

  1.  What version of C*?
  2.  What compaction strategy?
  3.  What's core count allocated per C* node?
  4.  Gossip give you any headaches / you have to be delicate there or does it 
behave itself?
Context: pmc/committer and I manage the OSS C* team at DataStax. We're doing a 
lot of thinking about how to generally improve the operator experience across 
the board for folks in the post 4.0 time frame, so data like the above (where 
things are going well at scale and why) is super useful to help feed into that 
effort.

Thanks!



On Wed, Oct 28, 2020 at 7:14 AM, Tom van der Woerdt 
mailto:tom.vanderwoe...@booking.com.invalid>>
 wrote:
Does 360 count? :-)

num_tokens is 16, works fine (had 256 on a 300 node cluster as well, not too 
many problems either). Roughly 2.5TB per node, running on-prem on reasonably 
stable hardware so replacements end up happening once a week at most, and 
there's no particular change needed in the automation. Scaling up or down takes 
a while, but it doesn't appear to be slower than any other cluster. 
Configuration wise it's no different than a 5-node cluster either. Pretty 
uneventful tbh.

Tom van der Woerdt
Senior Site Reliability Engineer
Booking.com
 BV
Vijzelstraat Amsterdam Netherlands 1017HL
[Booking.com]
Making it easier for everyone to experience the world since 1996
43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 million 
reported listings
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)


On Wed, Oct 28, 2020 at 8:58 AM Gediminas Blazys 
mailto:gediminas.bla...@microsoft.com.invalid>>
 wrote:
Hello,

I wanted to seek out your opinion and experience.

Has anyone of you had a chance to run a Cassandra cluster of more than 350 
nodes?
What are the major 

Re: [EXTERNAL] Re: Adding new DC results in clients failing to connect

[João Reis]

 Hi,

We have received another report of this issue and this time we were able to
identify the bug and fix it. Today's release of the driver (version 3.16.1)
contains this fix. The JIRA issue is CSHARP-943 [1]

Thanks,
João Reis

[1] https://datastax-oss.atlassian.net/browse/CSHARP-943

Gediminas Blazys  escreveu no dia
segunda, 18/05/2020 à(s) 07:16:

> Hey,
>
>
>
> Apologies for the late reply João.
>
>
>
> We really, really appreciate your interest and likewise we could not
> reproduce this issue anywhere else but in production where it occurred,
> which is slightly undesirable. As we could not afford to keep the DC in
> this state we have removed it from our cluster. I’m afraid we cannot
> provide you with the info you’ve requested.
>
>
>
> Gediminas
>
>
>
> *From:* João Reis 
> *Sent:* Tuesday, May 12, 2020 19:58
> *To:* user@cassandra.apache.org
> *Subject:* Re: [EXTERNAL] Re: Adding new DC results in clients failing to
> connect
>
>
>
> Unfortunately I'm not able to reproduce this.
>
>
>
> Would it be possible for you to run a couple of queries and give us the
> results? The queries are "SELECT * FROM system.peers" and "SELECT * FROM
> system_schema.keyspaces". You should run both of these queries on any node
> that the driver uses to set up the control connection when that error
> occurs. To determine the node you can look for this driver log message:
> "Connection established to [NODE_ADDRESS] using protocol version [VERSION]."
>
>
>
> It should be easier to reproduce the issue with the results of those
> queries.
>
>
>
> Thanks,
>
> João Reis
>
>
>
> Gediminas Blazys  escreveu no dia
> sexta, 8/05/2020 à(s) 08:27:
>
> Hello,
>
>
>
> Thanks for looking into this. As far as the time for token map calculation
> goes, we are considering reducing the number of vnodes for future DCs.
> However, in the mean time we were able to deploy another DC8  (testing the
> hypothesis that this may be isolated to DC7 only) and the deployment
> worked. DC8 is part of the cluster now, currently being rebuilt and we did
> not notice login issues with this expansion. So the topology now is this:
>
>
>
> DC1 - 18 nodes - 256 vnodes - working
>
> DC2 - 18 nodes - 256 vnodes - working
>
> DC3 - 18 nodes - 256 vnodes - working
>
> DC4 - 18 nodes - 256 vnodes - working
>
> DC5 - 18 nodes - 256 vnodes - working
>
> DC6 - 60 nodes - 256 vnodes - working
>
> DC7 - 60 nodes - 256 vnodes - once added to replication, clients can't
> connect to any DC
>
> DC8 - 60 nodes - 256 vnodes - rebuilding at the moment, including this DC
> into replication did not cause login issues.
>
>
>
> The major difference between DC7 and other DCs is that in DC7 we only have
> two racks while in other locations we use three, the replication factor
> however for all keyspaces remains the same – 3 for all user defined
> keyspaces. Maybe this is something that could cause issues with duplicates? 
> It's
> a theoretical but cassandra having to place two replicas  on the same  rack
> maybe placed both the primary and a backup replica on the same node. Hence
> a duplicate...
>
>
>
> Gediminas
>
>
>
> *From:* João Reis 
> *Sent:* Thursday, May 7, 2020 19:22
> *To:* user@cassandra.apache.org
> *Subject:* Re: [EXTERNAL] Re: Adding new DC results in clients failing to
> connect
>
>
>
> Hi,
>
>
>
> I don't believe that the peers entry is responsible for that exception.
> Looking at the driver code, I can't even think of a scenario where that
> exception would be thrown... I will run some tests in the next couple of
> days to try and figure something out.
>
>
>
> One thing that is certain from those log messages is that the tokenmap
> computation is very slow (20 seconds). With 100+ nodes and 256 vnodes per
> node, we should expect the token map computation to be a bit slower but 20
> seconds is definitely too much. I've opened CSHARP-901 to track this. [1]
>
>
>
> João Reis
>
>
>
> [1] https://datastax-oss.atlassian.net/browse/CSHARP-901
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatastax-oss.atlassian.net%2Fbrowse%2FCSHARP-901=02%7C01%7CGediminas.Blazys%40microsoft.com%7C699b48a02b404847fd1908d7f695b020%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637248995123287800=cX1uFLsvyJPt%2FdL6x84d1CdYCN8m17A%2FpTFi1VmrG1c%3D=0>
>
>
>
> Gediminas Blazys  escreveu no dia
> segunda, 4/05/2020 à(s) 11:13:
>
> Hello again,
>
>
>
> Looking into system.peers we found that some nodes contain entries about
> themselves with null values. Not sure if this could be an issue, maybe

RE: [EXTERNAL] Re: Adding new DC results in clients failing to connect

[Gediminas Blazys]

Hey,

Apologies for the late reply João.

We really, really appreciate your interest and likewise we could not reproduce 
this issue anywhere else but in production where it occurred, which is slightly 
undesirable. As we could not afford to keep the DC in this state we have 
removed it from our cluster. I’m afraid we cannot provide you with the info 
you’ve requested.

Gediminas

From: João Reis 
Sent: Tuesday, May 12, 2020 19:58
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Re: Adding new DC results in clients failing to connect

Unfortunately I'm not able to reproduce this.

Would it be possible for you to run a couple of queries and give us the 
results? The queries are "SELECT * FROM system.peers" and "SELECT * FROM 
system_schema.keyspaces". You should run both of these queries on any node that 
the driver uses to set up the control connection when that error occurs. To 
determine the node you can look for this driver log message: "Connection 
established to [NODE_ADDRESS] using protocol version [VERSION]."

It should be easier to reproduce the issue with the results of those queries.

Thanks,
João Reis

Gediminas Blazys 
mailto:gediminas.bla...@microsoft.com.invalid>>
 escreveu no dia sexta, 8/05/2020 à(s) 08:27:
Hello,

Thanks for looking into this. As far as the time for token map calculation 
goes, we are considering reducing the number of vnodes for future DCs. However, 
in the mean time we were able to deploy another DC8  (testing the hypothesis 
that this may be isolated to DC7 only) and the deployment worked. DC8 is part 
of the cluster now, currently being rebuilt and we did not notice login issues 
with this expansion. So the topology now is this:

DC1 - 18 nodes - 256 vnodes - working
DC2 - 18 nodes - 256 vnodes - working
DC3 - 18 nodes - 256 vnodes - working
DC4 - 18 nodes - 256 vnodes - working
DC5 - 18 nodes - 256 vnodes - working
DC6 - 60 nodes - 256 vnodes - working
DC7 - 60 nodes - 256 vnodes - once added to replication, clients can't connect 
to any DC
DC8 - 60 nodes - 256 vnodes - rebuilding at the moment, including this DC into 
replication did not cause login issues.

The major difference between DC7 and other DCs is that in DC7 we only have two 
racks while in other locations we use three, the replication factor however for 
all keyspaces remains the same – 3 for all user defined keyspaces. Maybe this 
is something that could cause issues with duplicates? It's a theoretical but 
cassandra having to place two replicas  on the same  rack maybe placed both the 
primary and a backup replica on the same node. Hence a duplicate...

Gediminas

From: João Reis mailto:joao.r.r...@outlook.com>>
Sent: Thursday, May 7, 2020 19:22
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: Re: [EXTERNAL] Re: Adding new DC results in clients failing to connect

Hi,

I don't believe that the peers entry is responsible for that exception. Looking 
at the driver code, I can't even think of a scenario where that exception would 
be thrown... I will run some tests in the next couple of days to try and figure 
something out.

One thing that is certain from those log messages is that the tokenmap 
computation is very slow (20 seconds). With 100+ nodes and 256 vnodes per node, 
we should expect the token map computation to be a bit slower but 20 seconds is 
definitely too much. I've opened CSHARP-901 to track this. [1]

João Reis

[1] 
https://datastax-oss.atlassian.net/browse/CSHARP-901<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatastax-oss.atlassian.net%2Fbrowse%2FCSHARP-901=02%7C01%7CGediminas.Blazys%40microsoft.com%7C699b48a02b404847fd1908d7f695b020%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637248995123287800=cX1uFLsvyJPt%2FdL6x84d1CdYCN8m17A%2FpTFi1VmrG1c%3D=0>

Gediminas Blazys 
mailto:gediminas.bla...@microsoft.com.invalid>>
 escreveu no dia segunda, 4/05/2020 à(s) 11:13:
Hello again,

Looking into system.peers we found that some nodes contain entries about 
themselves with null values. Not sure if this could be an issue, maybe someone 
saw something similar? This state is there before including the funky DC into 
replication.
peer
 data_center
 host_id
 preferred_ip
 rack
 release_version
 rpc_address
 schema_version
 tokens

null
 null
 192.168.104.111
  null
null
null
null
null

Have a wonderful day 

Gediminas

From: Gediminas Blazys 
mailto:gediminas.bla...@microsoft.com.INVALID>>
Sent: Monday, May 4, 2020 10:09
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: [EXTERNAL] Re: Adding new DC results in clients failing to connect

Hello,

Thanks for the reply.

Following your advice we took a look at system.local for seed nodes and 
compared that data with nodetool ring. Both sources contain the same tokens for 
these specific hosts. Will continue looking into system.peers.

We have

Re: [EXTERNAL] Re: Adding new DC results in clients failing to connect

[João Reis]

Unfortunately I'm not able to reproduce this.

Would it be possible for you to run a couple of queries and give us the
results? The queries are "SELECT * FROM system.peers" and "SELECT * FROM
system_schema.keyspaces". You should run both of these queries on any node
that the driver uses to set up the control connection when that error
occurs. To determine the node you can look for this driver log message:
"Connection established to [NODE_ADDRESS] using protocol version [VERSION]."

It should be easier to reproduce the issue with the results of those
queries.

Thanks,
João Reis

Gediminas Blazys  escreveu no dia
sexta, 8/05/2020 à(s) 08:27:

> Hello,
>
>
>
> Thanks for looking into this. As far as the time for token map calculation
> goes, we are considering reducing the number of vnodes for future DCs.
> However, in the mean time we were able to deploy another DC8  (testing the
> hypothesis that this may be isolated to DC7 only) and the deployment
> worked. DC8 is part of the cluster now, currently being rebuilt and we did
> not notice login issues with this expansion. So the topology now is this:
>
>
>
> DC1 - 18 nodes - 256 vnodes - working
>
> DC2 - 18 nodes - 256 vnodes - working
>
> DC3 - 18 nodes - 256 vnodes - working
>
> DC4 - 18 nodes - 256 vnodes - working
>
> DC5 - 18 nodes - 256 vnodes - working
>
> DC6 - 60 nodes - 256 vnodes - working
>
> DC7 - 60 nodes - 256 vnodes - once added to replication, clients can't
> connect to any DC
>
> DC8 - 60 nodes - 256 vnodes - rebuilding at the moment, including this DC
> into replication did not cause login issues.
>
>
>
> The major difference between DC7 and other DCs is that in DC7 we only have
> two racks while in other locations we use three, the replication factor
> however for all keyspaces remains the same – 3 for all user defined
> keyspaces. Maybe this is something that could cause issues with duplicates? 
> It's
> a theoretical but cassandra having to place two replicas  on the same  rack
> maybe placed both the primary and a backup replica on the same node. Hence
> a duplicate...
>
>
>
> Gediminas
>
>
>
> *From:* João Reis 
> *Sent:* Thursday, May 7, 2020 19:22
> *To:* user@cassandra.apache.org
> *Subject:* Re: [EXTERNAL] Re: Adding new DC results in clients failing to
> connect
>
>
>
> Hi,
>
>
>
> I don't believe that the peers entry is responsible for that exception.
> Looking at the driver code, I can't even think of a scenario where that
> exception would be thrown... I will run some tests in the next couple of
> days to try and figure something out.
>
>
>
> One thing that is certain from those log messages is that the tokenmap
> computation is very slow (20 seconds). With 100+ nodes and 256 vnodes per
> node, we should expect the token map computation to be a bit slower but 20
> seconds is definitely too much. I've opened CSHARP-901 to track this. [1]
>
>
>
> João Reis
>
>
>
> [1] https://datastax-oss.atlassian.net/browse/CSHARP-901
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatastax-oss.atlassian.net%2Fbrowse%2FCSHARP-901=02%7C01%7CGediminas.Blazys%40microsoft.com%7Cb82cb4f2ca784a9fd4a608d7f2a2d300%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637244653454584013=%2B9ojISBaiyNt%2Fvlyat2wOCgDbFJIyXjmjuYMhPCB4YU%3D=0>
>
>
>
> Gediminas Blazys  escreveu no dia
> segunda, 4/05/2020 à(s) 11:13:
>
> Hello again,
>
>
>
> Looking into system.peers we found that some nodes contain entries about
> themselves with null values. Not sure if this could be an issue, maybe
> someone saw something similar? This state is there before including the
> funky DC into replication.
>
> peer
>
>  data_center
>
>  host_id
>
>  preferred_ip
>
>  rack
>
>  release_version
>
>  rpc_address
>
>  schema_version
>
>  tokens
>
> 
>
> null
>
>  null
>
>  192.168.104.111
>
>   null
>
> null
>
> null
>
> null
>
> null
>
>
>
> Have a wonderful day 
>
>
>
> Gediminas
>
>
>
> *From:* Gediminas Blazys 
> *Sent:* Monday, May 4, 2020 10:09
> *To:* user@cassandra.apache.org
> *Subject:* RE: [EXTERNAL] Re: Adding new DC results in clients failing to
> connect
>
>
>
> Hello,
>
>
>
> Thanks for the reply.
>
>
>
> Following your advice we took a look at system.local for seed nodes and
> compared that data with nodetool ring. Both sources contain the same tokens
> for these specific hosts. Will continue looking into system.peers.
>
>
>
> We have enabled more verbosity on the 

RE: [EXTERNAL] Re: Adding new DC results in clients failing to connect

[Gediminas Blazys]

Hello,

Thanks for looking into this. As far as the time for token map calculation 
goes, we are considering reducing the number of vnodes for future DCs. However, 
in the mean time we were able to deploy another DC8  (testing the hypothesis 
that this may be isolated to DC7 only) and the deployment worked. DC8 is part 
of the cluster now, currently being rebuilt and we did not notice login issues 
with this expansion. So the topology now is this:

DC1 - 18 nodes - 256 vnodes - working
DC2 - 18 nodes - 256 vnodes - working
DC3 - 18 nodes - 256 vnodes - working
DC4 - 18 nodes - 256 vnodes - working
DC5 - 18 nodes - 256 vnodes - working
DC6 - 60 nodes - 256 vnodes - working
DC7 - 60 nodes - 256 vnodes - once added to replication, clients can't connect 
to any DC
DC8 - 60 nodes - 256 vnodes - rebuilding at the moment, including this DC into 
replication did not cause login issues.

The major difference between DC7 and other DCs is that in DC7 we only have two 
racks while in other locations we use three, the replication factor however for 
all keyspaces remains the same – 3 for all user defined keyspaces. Maybe this 
is something that could cause issues with duplicates? It's a theoretical but 
cassandra having to place two replicas  on the same  rack maybe placed both the 
primary and a backup replica on the same node. Hence a duplicate...

Gediminas

From: João Reis 
Sent: Thursday, May 7, 2020 19:22
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Re: Adding new DC results in clients failing to connect

Hi,

I don't believe that the peers entry is responsible for that exception. Looking 
at the driver code, I can't even think of a scenario where that exception would 
be thrown... I will run some tests in the next couple of days to try and figure 
something out.

One thing that is certain from those log messages is that the tokenmap 
computation is very slow (20 seconds). With 100+ nodes and 256 vnodes per node, 
we should expect the token map computation to be a bit slower but 20 seconds is 
definitely too much. I've opened CSHARP-901 to track this. [1]

João Reis

[1] 
https://datastax-oss.atlassian.net/browse/CSHARP-901<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatastax-oss.atlassian.net%2Fbrowse%2FCSHARP-901=02%7C01%7CGediminas.Blazys%40microsoft.com%7Cb82cb4f2ca784a9fd4a608d7f2a2d300%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637244653454584013=%2B9ojISBaiyNt%2Fvlyat2wOCgDbFJIyXjmjuYMhPCB4YU%3D=0>

Gediminas Blazys 
mailto:gediminas.bla...@microsoft.com.invalid>>
 escreveu no dia segunda, 4/05/2020 à(s) 11:13:
Hello again,

Looking into system.peers we found that some nodes contain entries about 
themselves with null values. Not sure if this could be an issue, maybe someone 
saw something similar? This state is there before including the funky DC into 
replication.
peer
 data_center
 host_id
 preferred_ip
 rack
 release_version
 rpc_address
 schema_version
 tokens

null
 null
 192.168.104.111
  null
null
null
null
null

Have a wonderful day 

Gediminas

From: Gediminas Blazys 
mailto:gediminas.bla...@microsoft.com.INVALID>>
Sent: Monday, May 4, 2020 10:09
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: RE: [EXTERNAL] Re: Adding new DC results in clients failing to connect

Hello,

Thanks for the reply.

Following your advice we took a look at system.local for seed nodes and 
compared that data with nodetool ring. Both sources contain the same tokens for 
these specific hosts. Will continue looking into system.peers.

We have enabled more verbosity on the C# driver and this is the message that we 
get now:


ControlConnection: 05/03/2020 14:28:42.346 +03:00 : Updating keyspaces metadata

ControlConnection: 05/03/2020 14:28:42.377 +03:00 : Rebuilding token map

ControlConnection: 05/03/2020 14:29:03.837 +03:00 : Finished building TokenMap 
for 7 keyspaces and 210 hosts. It took 19403 milliseconds.

ControlConnection: 05/03/2020 14:29:03.901 +03:00 ALARMA: ENDPOINT: 
<>:9042 EXCEPTION: System.ArgumentException: The source argument 
contains duplicate keys.

   at 
System.Collections.Concurrent.ConcurrentDictionary`2.InitializeFromCollection(IEnumerable`1
 collection)

   at System.Collections.Concurrent.ConcurrentDictionary`2..ctor(IEnumerable`1 
collection, IEqualityComparer`1 comparer)

   at System.Collections.Concurrent.ConcurrentDictionary`2..ctor(IEnumerable`1 
collection)

   at Cassandra.TokenMap..ctor(TokenFactory factory, IReadOnlyDictionary`2 
tokenToHostsByKeyspace, List`1 ring, IReadOnlyDictionary`2 primaryReplicas, 
IReadOnlyDictionary`2 keyspaceTokensCache, IReadOnlyDictionary`2 datacenters, 
Int32 numberOfHostsWithTokens)

   at Cassandra.TokenMap.Build(String partitioner, ICollection`1 hosts, 
ICollection`1 keyspaces)

   at Cassandra.Metadata.d__59.MoveNext()

--- End of stack trace from previous locat

Re: [EXTERNAL] Re: Adding new DC results in clients failing to connect

[João Reis]

Hi,

I don't believe that the peers entry is responsible for that exception.
Looking at the driver code, I can't even think of a scenario where that
exception would be thrown... I will run some tests in the next couple of
days to try and figure something out.

One thing that is certain from those log messages is that the tokenmap
computation is very slow (20 seconds). With 100+ nodes and 256 vnodes per
node, we should expect the token map computation to be a bit slower but 20
seconds is definitely too much. I've opened CSHARP-901 to track this. [1]

João Reis

[1] https://datastax-oss.atlassian.net/browse/CSHARP-901

Gediminas Blazys  escreveu no dia
segunda, 4/05/2020 à(s) 11:13:

> Hello again,
>
>
>
> Looking into system.peers we found that some nodes contain entries about
> themselves with null values. Not sure if this could be an issue, maybe
> someone saw something similar? This state is there before including the
> funky DC into replication.
>
> peer
>
>  data_center
>
>  host_id
>
>  preferred_ip
>
>  rack
>
>  release_version
>
>  rpc_address
>
>  schema_version
>
>  tokens
>
> 
>
> null
>
>  null
>
>  192.168.104.111
>
>   null
>
> null
>
> null
>
> null
>
> null
>
>
>
> Have a wonderful day 
>
>
>
> Gediminas
>
>
>
> *From:* Gediminas Blazys 
> *Sent:* Monday, May 4, 2020 10:09
> *To:* user@cassandra.apache.org
> *Subject:* RE: [EXTERNAL] Re: Adding new DC results in clients failing to
> connect
>
>
>
> Hello,
>
>
>
> Thanks for the reply.
>
>
>
> Following your advice we took a look at system.local for seed nodes and
> compared that data with nodetool ring. Both sources contain the same tokens
> for these specific hosts. Will continue looking into system.peers.
>
>
>
> We have enabled more verbosity on the C# driver and this is the message
> that we get now:
>
>
>
> ControlConnection: 05/03/2020 14:28:42.346 +03:00 : Updating keyspaces
> metadata
>
> ControlConnection: 05/03/2020 14:28:42.377 +03:00 : Rebuilding token map
>
> ControlConnection: 05/03/2020 14:29:03.837 +03:00 : Finished building
> TokenMap for 7 keyspaces and 210 hosts. It took 19403 milliseconds.
>
> ControlConnection: 05/03/2020 14:29:03.901 +03:00 ALARMA: ENDPOINT:
> <>:9042 EXCEPTION: System.ArgumentException: The source argument
> contains duplicate keys.
>
>at
> System.Collections.Concurrent.ConcurrentDictionary`2.InitializeFromCollection(IEnumerable`1
> collection)
>
>at
> System.Collections.Concurrent.ConcurrentDictionary`2..ctor(IEnumerable`1
> collection, IEqualityComparer`1 comparer)
>
>at
> System.Collections.Concurrent.ConcurrentDictionary`2..ctor(IEnumerable`1
> collection)
>
>at Cassandra.TokenMap..ctor(TokenFactory factory, IReadOnlyDictionary`2
> tokenToHostsByKeyspace, List`1 ring, IReadOnlyDictionary`2 primaryReplicas,
> IReadOnlyDictionary`2 keyspaceTokensCache, IReadOnlyDictionary`2
> datacenters, Int32 numberOfHostsWithTokens)
>
>at Cassandra.TokenMap.Build(String partitioner, ICollection`1 hosts,
> ICollection`1 keyspaces)
>
>at Cassandra.Metadata.d__59.MoveNext()
>
> --- End of stack trace from previous location where exception was thrown
> ---
>
>at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
> task)
>
>at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
> task)
>
>at
> System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
>
>at Cassandra.Connections.ControlConnection.d__44.MoveNext()
>
>
>
> The error occurs on Cassandra.TokenMap. We are analyzing objects that the
> driver initializes during the token map creation but we are yet to find
> that dictionary with duplicated keys.
>
> Just to note, once this new DC is added to replication python driver is
> unable to establish a connection either. cqlsh though, seems to be ok. It
> is hard to say for sure, but for now at least, this issue seems to be
> pointing to Cassandra.
>
>
>
> Gediminas
>
>
>
> *From:* Jorge Bay Gondra 
> *Sent:* Thursday, April 30, 2020 11:45
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: Adding new DC results in clients failing to
> connect
>
>
>
> Hi,
>
> You can enable logging at driver to see what's happening under the hood:
> https://docs.datastax.com/en/developer/csharp-driver/3.14/faq/#how-can-i-enable-logging-in-the-driver
> <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.datastax.com%2Fen%2Fdeveloper%

RE: [EXTERNAL] Re: Adding new DC results in clients failing to connect

[Gediminas Blazys]

Hello again,

Looking into system.peers we found that some nodes contain entries about 
themselves with null values. Not sure if this could be an issue, maybe someone 
saw something similar? This state is there before including the funky DC into 
replication.
peer
 data_center
 host_id
 preferred_ip
 rack
 release_version
 rpc_address
 schema_version
 tokens

null
 null
 192.168.104.111
  null
null
null
null
null

Have a wonderful day 

Gediminas

From: Gediminas Blazys 
Sent: Monday, May 4, 2020 10:09
To: user@cassandra.apache.org
Subject: RE: [EXTERNAL] Re: Adding new DC results in clients failing to connect

Hello,

Thanks for the reply.

Following your advice we took a look at system.local for seed nodes and 
compared that data with nodetool ring. Both sources contain the same tokens for 
these specific hosts. Will continue looking into system.peers.

We have enabled more verbosity on the C# driver and this is the message that we 
get now:


ControlConnection: 05/03/2020 14:28:42.346 +03:00 : Updating keyspaces metadata

ControlConnection: 05/03/2020 14:28:42.377 +03:00 : Rebuilding token map

ControlConnection: 05/03/2020 14:29:03.837 +03:00 : Finished building TokenMap 
for 7 keyspaces and 210 hosts. It took 19403 milliseconds.

ControlConnection: 05/03/2020 14:29:03.901 +03:00 ALARMA: ENDPOINT: 
<>:9042 EXCEPTION: System.ArgumentException: The source argument 
contains duplicate keys.

   at 
System.Collections.Concurrent.ConcurrentDictionary`2.InitializeFromCollection(IEnumerable`1
 collection)

   at System.Collections.Concurrent.ConcurrentDictionary`2..ctor(IEnumerable`1 
collection, IEqualityComparer`1 comparer)

   at System.Collections.Concurrent.ConcurrentDictionary`2..ctor(IEnumerable`1 
collection)

   at Cassandra.TokenMap..ctor(TokenFactory factory, IReadOnlyDictionary`2 
tokenToHostsByKeyspace, List`1 ring, IReadOnlyDictionary`2 primaryReplicas, 
IReadOnlyDictionary`2 keyspaceTokensCache, IReadOnlyDictionary`2 datacenters, 
Int32 numberOfHostsWithTokens)

   at Cassandra.TokenMap.Build(String partitioner, ICollection`1 hosts, 
ICollection`1 keyspaces)

   at Cassandra.Metadata.d__59.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at 
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
 task)

   at 
System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()

   at Cassandra.Connections.ControlConnection.d__44.MoveNext()

The error occurs on Cassandra.TokenMap. We are analyzing objects that the 
driver initializes during the token map creation but we are yet to find that 
dictionary with duplicated keys.
Just to note, once this new DC is added to replication python driver is unable 
to establish a connection either. cqlsh though, seems to be ok. It is hard to 
say for sure, but for now at least, this issue seems to be pointing to 
Cassandra.

Gediminas

From: Jorge Bay Gondra 
mailto:jorgebaygon...@gmail.com>>
Sent: Thursday, April 30, 2020 11:45
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Re: Adding new DC results in clients failing to connect

Hi,
You can enable logging at driver to see what's happening under the hood: 
https://docs.datastax.com/en/developer/csharp-driver/3.14/faq/#how-can-i-enable-logging-in-the-driver<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.datastax.com%2Fen%2Fdeveloper%2Fcsharp-driver%2F3.14%2Ffaq%2F%23how-can-i-enable-logging-in-the-driver=02%7C01%7CGediminas.Blazys%40microsoft.com%7C6a5b382a16e54752bb8e08d7effa07bc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637241729477296305=a3XX8EzNAZk7ak3EE3Q7U4kxTtNii2svHqNpoKZgADI%3D=0>
With logging information, it should be easy to track the issue down.

Can you query system.local and system.peers on a seed node / contact point to 
see if all the node list / token info is expected. You can compare it to 
nodetool ring info.

Not directly related: 256 vnodes is probably more than you want.

Thanks,
Jorge

On Thu, Apr 30, 2020 at 9:48 AM Gediminas Blazys 
mailto:gediminas.bla...@microsoft.com.invalid>>
 wrote:
Hello,

We have run into a very interesting issue and maybe some of you have 
encountered it or just have an idea where to look.

We are working towards adding new dcs into our cluster, here's the current 
topology:
DC1 - 18 nodes
DC2 - 18 nodes
DC3 - 18 nodes
DC4 - 18 nodes
DC5 - 18 nodes

Recently we introduced a new DC6 (60 nodes) into our cluster. The joining and 
rebuilding of DC6 went smoothly, clients are using it without issue. This is 
how it looked after joining DC6:
DC1 - 18 nodes
DC2 - 18 nodes
DC3 - 18 nodes
DC4 - 18 nodes
DC5 - 18 nodes
DC6 - 60 nodes

Next we wanted to add another DC7 (also 60 nodes) making it a total of 210 
nodes in the cluster, an

RE: [EXTERNAL] Re: Adding new DC results in clients failing to connect

[Gediminas Blazys]

Hello,

Thanks for the reply.

Following your advice we took a look at system.local for seed nodes and 
compared that data with nodetool ring. Both sources contain the same tokens for 
these specific hosts. Will continue looking into system.peers.

We have enabled more verbosity on the C# driver and this is the message that we 
get now:


ControlConnection: 05/03/2020 14:28:42.346 +03:00 : Updating keyspaces metadata

ControlConnection: 05/03/2020 14:28:42.377 +03:00 : Rebuilding token map

ControlConnection: 05/03/2020 14:29:03.837 +03:00 : Finished building TokenMap 
for 7 keyspaces and 210 hosts. It took 19403 milliseconds.

ControlConnection: 05/03/2020 14:29:03.901 +03:00 ALARMA: ENDPOINT: 
<>:9042 EXCEPTION: System.ArgumentException: The source argument 
contains duplicate keys.

   at 
System.Collections.Concurrent.ConcurrentDictionary`2.InitializeFromCollection(IEnumerable`1
 collection)

   at System.Collections.Concurrent.ConcurrentDictionary`2..ctor(IEnumerable`1 
collection, IEqualityComparer`1 comparer)

   at System.Collections.Concurrent.ConcurrentDictionary`2..ctor(IEnumerable`1 
collection)

   at Cassandra.TokenMap..ctor(TokenFactory factory, IReadOnlyDictionary`2 
tokenToHostsByKeyspace, List`1 ring, IReadOnlyDictionary`2 primaryReplicas, 
IReadOnlyDictionary`2 keyspaceTokensCache, IReadOnlyDictionary`2 datacenters, 
Int32 numberOfHostsWithTokens)

   at Cassandra.TokenMap.Build(String partitioner, ICollection`1 hosts, 
ICollection`1 keyspaces)

   at Cassandra.Metadata.d__59.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

   at 
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
 task)

   at 
System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()

   at Cassandra.Connections.ControlConnection.d__44.MoveNext()

The error occurs on Cassandra.TokenMap. We are analyzing objects that the 
driver initializes during the token map creation but we are yet to find that 
dictionary with duplicated keys.
Just to note, once this new DC is added to replication python driver is unable 
to establish a connection either. cqlsh though, seems to be ok. It is hard to 
say for sure, but for now at least, this issue seems to be pointing to 
Cassandra.

Gediminas

From: Jorge Bay Gondra 
Sent: Thursday, April 30, 2020 11:45
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Adding new DC results in clients failing to connect

Hi,
You can enable logging at driver to see what's happening under the hood: 
https://docs.datastax.com/en/developer/csharp-driver/3.14/faq/#how-can-i-enable-logging-in-the-driver
With logging information, it should be easy to track the issue down.

Can you query system.local and system.peers on a seed node / contact point to 
see if all the node list / token info is expected. You can compare it to 
nodetool ring info.

Not directly related: 256 vnodes is probably more than you want.

Thanks,
Jorge

On Thu, Apr 30, 2020 at 9:48 AM Gediminas Blazys 
mailto:gediminas.bla...@microsoft.com.invalid>>
 wrote:
Hello,

We have run into a very interesting issue and maybe some of you have 
encountered it or just have an idea where to look.

We are working towards adding new dcs into our cluster, here's the current 
topology:
DC1 - 18 nodes
DC2 - 18 nodes
DC3 - 18 nodes
DC4 - 18 nodes
DC5 - 18 nodes

Recently we introduced a new DC6 (60 nodes) into our cluster. The joining and 
rebuilding of DC6 went smoothly, clients are using it without issue. This is 
how it looked after joining DC6:
DC1 - 18 nodes
DC2 - 18 nodes
DC3 - 18 nodes
DC4 - 18 nodes
DC5 - 18 nodes
DC6 - 60 nodes

Next we wanted to add another DC7 (also 60 nodes) making it a total of 210 
nodes in the cluster, and while joining new nodes went smoothly, once we 
changed the replication of user defined keyspaces to include DC7, no clients 
were able to connect to Cassandra (regardless of which DC is being addressed). 
They would throw an exception that I have provided at the end of the email.

Cassandra version 3.11.4.
C# driver version 3.12.0. Also tested with 3.14.0. We use dc round robin policy 
and update ring metadata for connecting clients.
Amount of vnodes per node: 256

The stack trace starts with an exception 'The source argument contains 
duplicate keys.'. Maybe you know what kind of data is in this dictionary? What 
data can be duplicated here?

Clients are unable to connect until the moment we remove DC7 from replication. 
Once replication is adjusted to exclude DC7, 

RE: [EXTERNAL] Re: Performance of Data Types used for Primary keys

[Durity, Sean R]

I agree. Cassandra already hashes the partition key to a numeric token.

Sean Durity

From: Jon Haddad 
Sent: Friday, March 6, 2020 9:29 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Performance of Data Types used for Primary keys

It's not going to matter at all.

On Fri, Mar 6, 2020, 2:15 AM Hanauer, Arnulf, Vodacom South Africa (External) 
mailto:arnulf.hana...@vcontractor.co.za>> 
wrote:
Hi Cassandra folks,

Is there any difference in performance of general operations if using a TEXT 
based Primary key versus a BIGINT Primary key.

Our use-case requires low latency reads but currently the Primary key is TEXT 
based but the data could work on BIGINT. We are trying to optimise where 
possible.
Any experiences that could point to a winner?


Kind regards
Arnulf Hanauer









"This e-mail is sent on the Terms and Conditions that can be accessed by 
Clicking on this link https://webmail.vodacom.co.za/tc/default.html 
[vodacom.co.za]
 "



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Hossein Ghiyasi Mehr]

There isn't any Rollback in real time systems.
It's better to test upgrade sstables and binary on one node. Then

   - if it was OK, upgrade binary on all nodes then run upgrade sstables
   one server at a time.

OR

   - If it was OK, upgrade servers (binary+sstables) one by one.

*---*
*VafaTech  : A Total Solution for Data Gathering &
Analysis*
*---*


On Wed, Mar 4, 2020 at 7:38 AM manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> Should upgradesstables not be run after every node is upgraded? If we need
> to rollback then  we will not be able to downgrade sstables to older
> version.
>
> Regards
> Manish
>
> On Tue, Mar 3, 2020 at 11:26 PM Hossein Ghiyasi Mehr <
> ghiyasim...@gmail.com> wrote:
>
>> It's more safe to upgrade one node before upgrading another node to avoid
>> down time.
>> After upgrading binary and package, run upgradesstables on candidate node
>> then do it on all cluster nodes one by one.
>> *---*
>> *VafaTech  : A Total Solution for Data Gathering
>> & Analysis*
>> *---*
>>
>>
>> On Thu, Feb 13, 2020 at 9:27 PM Sergio  wrote:
>>
>>>
>>>- Verify that nodetool upgradesstables has completed successfully on
>>>all nodes from any previous upgrade
>>>- Turn off repairs and any other streaming operations (add/remove
>>>nodes)
>>>- Nodetool drain on the node that needs to be stopped (seeds first,
>>>preferably)
>>>- Stop an un-upgraded node (seeds first, preferably)
>>>- Install new binaries and configs on the down node
>>>- Restart that node and make sure it comes up clean (it will
>>>function normally in the cluster – even with mixed versions)
>>>- nodetool statusbinary to verify if it is up and running
>>>- Repeat for all nodes
>>>- Once the binary upgrade has been performed in all the nodes: Run
>>>upgradesstables on each node (as many at a time as your load will allow).
>>>Minor upgrades usually don’t require this step (only if the sstable 
>>> format
>>>has changed), but it is good to check.
>>>- NOTE: in most cases applications can keep running and will not
>>>notice much impact – unless the cluster is overloaded and a single node
>>>down causes impact.
>>>
>>>
>>>
>>>I added 2 points to the list to clarify.
>>>
>>>Should we add this in a FAQ in the cassandra doc or in the awesome
>>>cassandra https://cassandra.link/awesome/
>>>
>>>Thanks,
>>>
>>>Sergio
>>>
>>>
>>> Il giorno mer 12 feb 2020 alle ore 10:58 Durity, Sean R <
>>> sean_r_dur...@homedepot.com> ha scritto:
>>>
 Check the readme.txt for any upgrade notes, but the basic procedure is
 to:

- Verify that nodetool upgradesstables has completed successfully
on all nodes from any previous upgrade
- Turn off repairs and any other streaming operations (add/remove
nodes)
- Stop an un-upgraded node (seeds first, preferably)
- Install new binaries and configs on the down node
- Restart that node and make sure it comes up clean (it will
function normally in the cluster – even with mixed versions)
- Repeat for all nodes
- Run upgradesstables on each node (as many at a time as your load
will allow). Minor upgrades usually don’t require this step (only if the
sstable format has changed), but it is good to check.
- NOTE: in most cases applications can keep running and will not
notice much impact – unless the cluster is overloaded and a single node
down causes impact.







 Sean Durity – Staff Systems Engineer, Cassandra



 *From:* Sergio 
 *Sent:* Wednesday, February 12, 2020 11:36 AM
 *To:* user@cassandra.apache.org
 *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades



 Hi guys!

 How do you usually upgrade your cluster for minor version upgrades?

 I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
 nodes.

 Is there any restriction?

 Best,

 Sergio

 --

 The information in this Internet Email is confidential and may be
 legally privileged. It is intended solely for the addressee. Access to this
 Email by anyone else is unauthorized. If you are not the intended
 recipient, any disclosure, copying, distribution or any action taken or
 omitted to be taken in reliance on it, is prohibited and may be unlawful.
 When addressed to our clients any opinions or advice contained in this
 Email are subject to the terms and conditions expressed in any applicable
 governing The Home Depot terms of business or client engagement letter. The
 Home 

RE: [EXTERNAL] Cassandra 3.11.X upgrades

[Durity, Sean R]

I agree – a back out becomes practically very challenging after the second node 
is upgraded, because the new data is written in the new disk format. To satisfy 
the “you must have a backout” rules, I just say that after node 1, I could stop 
that node, wipe the data, downgrade the binaries, and replace that node back to 
the original version (and yes, there could still be consistency problems with 
that). There is no going back after node 2. And I have never needed to try and 
go back, either. Test well in NP, and be ready to tackle any PR problems to 
keep going forward.

Sean Durity

From: Erick Ramirez 
Sent: Tuesday, March 3, 2020 11:35 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Cassandra 3.11.X upgrades

Should upgradesstables not be run after every node is upgraded? If we need to 
rollback then  we will not be able to downgrade sstables to older version

You can choose to (a) upgrade the SSTables one node at a time as you complete 
the binary upgrade, or (b) upgrade the binaries on all nodes then perform the 
SSTables upgrade in one hit. The choice is up to you but beware of the 
following caveats:
- there's a performance hit when C* reads "n - 1" versions of SSTables so the 
sooner you do it the better
- upgrading SSTables one node at a time is preferable due to the considerable 
performance hit or schedule it during low traffic periods

The idea of a rollback isn't what you're accustomed to. There is no concept of 
"downgrade" in Cassandra. If you decide to rollback or backout of an upgrade 
implementation, it means that you have to restore your cluster from backups so 
be aware of that too. This is because once you've performed an upgrade, things 
like schema and system tables are generally no longer backward-compatible. 
Also, new incoming mutations are written in the new format which again is not 
backward-compatible. To cut to the chase -- the decision to upgrade the 
SSTables has zero bearing on rollback. Cheers!




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Erick Ramirez]

>
> Should upgradesstables not be run after every node is upgraded? If we need
> to rollback then  we will not be able to downgrade sstables to older version
>

You can choose to (a) upgrade the SSTables one node at a time as you
complete the binary upgrade, or (b) upgrade the binaries on all nodes then
perform the SSTables upgrade in one hit. The choice is up to you but beware
of the following caveats:
- there's a performance hit when C* reads "n - 1" versions of SSTables so
the sooner you do it the better
- upgrading SSTables one node at a time is preferable due to the
considerable performance hit or schedule it during low traffic periods

The idea of a rollback isn't what you're accustomed to. There is no concept
of "downgrade" in Cassandra. If you decide to rollback or backout of an
upgrade implementation, it means that you have to restore your cluster from
backups so be aware of that too. This is because once you've performed an
upgrade, things like schema and system tables are generally no longer
backward-compatible. Also, new incoming mutations are written in the new
format which again is not backward-compatible. To cut to the chase -- the
decision to upgrade the SSTables has zero bearing on rollback. Cheers!

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Anthony Grasso]

Manish is correct.

Upgrade the Cassandra version of a single node only. If that node is
behaving as expected (i.e. is in an Up/Normal state and no errors in the
logs), then upgrade the Cassandra version for each node one at a time. Be
sure to check that each node is running as expected. Once the Cassandra
version is upgraded on all nodes in the cluster and all nodes are in a
healthy state, then you upgrade the SSTables.

As Manish has pointed out, you want to be able to rollback the Cassandra
version easily if something goes wrong with the software upgrade.

If you have the disk space, I recommend taking a snapshot of the SSTables
on all nodes first before upgrading the Cassandra software version. The
snapshots are a safe guard incase there is a major problem after upgrading
SSTables.

Regards,


On Wed, 4 Mar 2020 at 15:08, manish khandelwal 
wrote:

> Should upgradesstables not be run after every node is upgraded? If we need
> to rollback then  we will not be able to downgrade sstables to older
> version.
>
> Regards
> Manish
>
> On Tue, Mar 3, 2020 at 11:26 PM Hossein Ghiyasi Mehr <
> ghiyasim...@gmail.com> wrote:
>
>> It's more safe to upgrade one node before upgrading another node to avoid
>> down time.
>> After upgrading binary and package, run upgradesstables on candidate node
>> then do it on all cluster nodes one by one.
>> *---*
>> *VafaTech  : A Total Solution for Data Gathering
>> & Analysis*
>> *---*
>>
>>
>> On Thu, Feb 13, 2020 at 9:27 PM Sergio  wrote:
>>
>>>
>>>- Verify that nodetool upgradesstables has completed successfully on
>>>all nodes from any previous upgrade
>>>- Turn off repairs and any other streaming operations (add/remove
>>>nodes)
>>>- Nodetool drain on the node that needs to be stopped (seeds first,
>>>preferably)
>>>- Stop an un-upgraded node (seeds first, preferably)
>>>- Install new binaries and configs on the down node
>>>- Restart that node and make sure it comes up clean (it will
>>>function normally in the cluster – even with mixed versions)
>>>- nodetool statusbinary to verify if it is up and running
>>>- Repeat for all nodes
>>>- Once the binary upgrade has been performed in all the nodes: Run
>>>upgradesstables on each node (as many at a time as your load will allow).
>>>Minor upgrades usually don’t require this step (only if the sstable 
>>> format
>>>has changed), but it is good to check.
>>>- NOTE: in most cases applications can keep running and will not
>>>notice much impact – unless the cluster is overloaded and a single node
>>>down causes impact.
>>>
>>>
>>>
>>>I added 2 points to the list to clarify.
>>>
>>>Should we add this in a FAQ in the cassandra doc or in the awesome
>>>cassandra https://cassandra.link/awesome/
>>>
>>>Thanks,
>>>
>>>Sergio
>>>
>>>
>>> Il giorno mer 12 feb 2020 alle ore 10:58 Durity, Sean R <
>>> sean_r_dur...@homedepot.com> ha scritto:
>>>
 Check the readme.txt for any upgrade notes, but the basic procedure is
 to:

- Verify that nodetool upgradesstables has completed successfully
on all nodes from any previous upgrade
- Turn off repairs and any other streaming operations (add/remove
nodes)
- Stop an un-upgraded node (seeds first, preferably)
- Install new binaries and configs on the down node
- Restart that node and make sure it comes up clean (it will
function normally in the cluster – even with mixed versions)
- Repeat for all nodes
- Run upgradesstables on each node (as many at a time as your load
will allow). Minor upgrades usually don’t require this step (only if the
sstable format has changed), but it is good to check.
- NOTE: in most cases applications can keep running and will not
notice much impact – unless the cluster is overloaded and a single node
down causes impact.







 Sean Durity – Staff Systems Engineer, Cassandra



 *From:* Sergio 
 *Sent:* Wednesday, February 12, 2020 11:36 AM
 *To:* user@cassandra.apache.org
 *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades



 Hi guys!

 How do you usually upgrade your cluster for minor version upgrades?

 I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
 nodes.

 Is there any restriction?

 Best,

 Sergio

 --

 The information in this Internet Email is confidential and may be
 legally privileged. It is intended solely for the addressee. Access to this
 Email by anyone else is unauthorized. If you are not the intended
 recipient, any disclosure, copying, distribution or any action taken or
 omitted to be taken in reliance on 

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[manish khandelwal]

Should upgradesstables not be run after every node is upgraded? If we need
to rollback then  we will not be able to downgrade sstables to older
version.

Regards
Manish

On Tue, Mar 3, 2020 at 11:26 PM Hossein Ghiyasi Mehr 
wrote:

> It's more safe to upgrade one node before upgrading another node to avoid
> down time.
> After upgrading binary and package, run upgradesstables on candidate node
> then do it on all cluster nodes one by one.
> *---*
> *VafaTech  : A Total Solution for Data Gathering
> & Analysis*
> *---*
>
>
> On Thu, Feb 13, 2020 at 9:27 PM Sergio  wrote:
>
>>
>>- Verify that nodetool upgradesstables has completed successfully on
>>all nodes from any previous upgrade
>>- Turn off repairs and any other streaming operations (add/remove
>>nodes)
>>- Nodetool drain on the node that needs to be stopped (seeds first,
>>preferably)
>>- Stop an un-upgraded node (seeds first, preferably)
>>- Install new binaries and configs on the down node
>>- Restart that node and make sure it comes up clean (it will function
>>normally in the cluster – even with mixed versions)
>>- nodetool statusbinary to verify if it is up and running
>>- Repeat for all nodes
>>- Once the binary upgrade has been performed in all the nodes: Run
>>upgradesstables on each node (as many at a time as your load will allow).
>>Minor upgrades usually don’t require this step (only if the sstable format
>>has changed), but it is good to check.
>>- NOTE: in most cases applications can keep running and will not
>>notice much impact – unless the cluster is overloaded and a single node
>>down causes impact.
>>
>>
>>
>>I added 2 points to the list to clarify.
>>
>>Should we add this in a FAQ in the cassandra doc or in the awesome
>>cassandra https://cassandra.link/awesome/
>>
>>Thanks,
>>
>>Sergio
>>
>>
>> Il giorno mer 12 feb 2020 alle ore 10:58 Durity, Sean R <
>> sean_r_dur...@homedepot.com> ha scritto:
>>
>>> Check the readme.txt for any upgrade notes, but the basic procedure is
>>> to:
>>>
>>>- Verify that nodetool upgradesstables has completed successfully on
>>>all nodes from any previous upgrade
>>>- Turn off repairs and any other streaming operations (add/remove
>>>nodes)
>>>- Stop an un-upgraded node (seeds first, preferably)
>>>- Install new binaries and configs on the down node
>>>- Restart that node and make sure it comes up clean (it will
>>>function normally in the cluster – even with mixed versions)
>>>- Repeat for all nodes
>>>- Run upgradesstables on each node (as many at a time as your load
>>>will allow). Minor upgrades usually don’t require this step (only if the
>>>sstable format has changed), but it is good to check.
>>>- NOTE: in most cases applications can keep running and will not
>>>notice much impact – unless the cluster is overloaded and a single node
>>>down causes impact.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Sean Durity – Staff Systems Engineer, Cassandra
>>>
>>>
>>>
>>> *From:* Sergio 
>>> *Sent:* Wednesday, February 12, 2020 11:36 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades
>>>
>>>
>>>
>>> Hi guys!
>>>
>>> How do you usually upgrade your cluster for minor version upgrades?
>>>
>>> I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
>>> nodes.
>>>
>>> Is there any restriction?
>>>
>>> Best,
>>>
>>> Sergio
>>>
>>> --
>>>
>>> The information in this Internet Email is confidential and may be
>>> legally privileged. It is intended solely for the addressee. Access to this
>>> Email by anyone else is unauthorized. If you are not the intended
>>> recipient, any disclosure, copying, distribution or any action taken or
>>> omitted to be taken in reliance on it, is prohibited and may be unlawful.
>>> When addressed to our clients any opinions or advice contained in this
>>> Email are subject to the terms and conditions expressed in any applicable
>>> governing The Home Depot terms of business or client engagement letter. The
>>> Home Depot disclaims all responsibility and liability for the accuracy and
>>> content of this attachment and for any damages or losses arising from any
>>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>>> items of a destructive nature, which may be contained in this attachment
>>> and shall not be liable for direct, indirect, consequential or special
>>> damages in connection with this e-mail message or its attachment.
>>>
>>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Hossein Ghiyasi Mehr]

It's more safe to upgrade one node before upgrading another node to avoid
down time.
After upgrading binary and package, run upgradesstables on candidate node
then do it on all cluster nodes one by one.
*---*
*VafaTech  : A Total Solution for Data Gathering &
Analysis*
*---*


On Thu, Feb 13, 2020 at 9:27 PM Sergio  wrote:

>
>- Verify that nodetool upgradesstables has completed successfully on
>all nodes from any previous upgrade
>- Turn off repairs and any other streaming operations (add/remove
>nodes)
>- Nodetool drain on the node that needs to be stopped (seeds first,
>preferably)
>- Stop an un-upgraded node (seeds first, preferably)
>- Install new binaries and configs on the down node
>- Restart that node and make sure it comes up clean (it will function
>normally in the cluster – even with mixed versions)
>- nodetool statusbinary to verify if it is up and running
>- Repeat for all nodes
>- Once the binary upgrade has been performed in all the nodes: Run
>upgradesstables on each node (as many at a time as your load will allow).
>Minor upgrades usually don’t require this step (only if the sstable format
>has changed), but it is good to check.
>- NOTE: in most cases applications can keep running and will not
>notice much impact – unless the cluster is overloaded and a single node
>down causes impact.
>
>
>
>I added 2 points to the list to clarify.
>
>Should we add this in a FAQ in the cassandra doc or in the awesome
>cassandra https://cassandra.link/awesome/
>
>Thanks,
>
>Sergio
>
>
> Il giorno mer 12 feb 2020 alle ore 10:58 Durity, Sean R <
> sean_r_dur...@homedepot.com> ha scritto:
>
>> Check the readme.txt for any upgrade notes, but the basic procedure is to:
>>
>>- Verify that nodetool upgradesstables has completed successfully on
>>all nodes from any previous upgrade
>>- Turn off repairs and any other streaming operations (add/remove
>>nodes)
>>- Stop an un-upgraded node (seeds first, preferably)
>>- Install new binaries and configs on the down node
>>- Restart that node and make sure it comes up clean (it will function
>>normally in the cluster – even with mixed versions)
>>- Repeat for all nodes
>>- Run upgradesstables on each node (as many at a time as your load
>>will allow). Minor upgrades usually don’t require this step (only if the
>>sstable format has changed), but it is good to check.
>>- NOTE: in most cases applications can keep running and will not
>>notice much impact – unless the cluster is overloaded and a single node
>>down causes impact.
>>
>>
>>
>>
>>
>>
>>
>> Sean Durity – Staff Systems Engineer, Cassandra
>>
>>
>>
>> *From:* Sergio 
>> *Sent:* Wednesday, February 12, 2020 11:36 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades
>>
>>
>>
>> Hi guys!
>>
>> How do you usually upgrade your cluster for minor version upgrades?
>>
>> I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
>> nodes.
>>
>> Is there any restriction?
>>
>> Best,
>>
>> Sergio
>>
>> --
>>
>> The information in this Internet Email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this Email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful. When addressed
>> to our clients any opinions or advice contained in this Email are subject
>> to the terms and conditions expressed in any applicable governing The Home
>> Depot terms of business or client engagement letter. The Home Depot
>> disclaims all responsibility and liability for the accuracy and content of
>> this attachment and for any damages or losses arising from any
>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>> items of a destructive nature, which may be contained in this attachment
>> and shall not be liable for direct, indirect, consequential or special
>> damages in connection with this e-mail message or its attachment.
>>
>

RE: [EXTERNAL] Re: IN OPERATOR VS BATCH QUERY

[Durity, Sean R]

Batches are for atomicity, not performance.

I would do single deletes with a prepared statement. An IN clause causes extra 
work for the coordinator because multiple partitions are being impacted. So, 
the coordinator has to coordinate all nodes involved in those writes (up to the 
whole cluster). Availability and performance are compromised for multiple 
partition operations. I do not allow them.

Also – TTL at insert (or update) is a much better solution than large purge 
strategies. As someone who spent a month wrangling hundreds of billions of 
deletes, I am an ardent preacher of TTL during design time.

Sean Durity

From: Attila Wind 
Sent: Friday, February 21, 2020 2:52 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: IN OPERATOR VS BATCH QUERY

Hi Sergio,

AFAIK you use batches when you want to get "all or nothing" approach from 
Cassandra. So turning multiple statements into one atomic operation.

One very typical use case for this is when you have denormalized data in 
multiple tables (optimized for different queries) but you need to modify all of 
them the same way as they were just one entity.

This means that if any ofyour delete statements would fail for whatever reason 
then all of your delete statements would be rolled back.

I think you dont want that overhead here for sure...

We are not there yet with our development but we will need similar "cleanup" 
functionality soon.
I was also thinking about the IN operator for similar cases but I am curious if 
anyone here has better idea...
Why does the IN operator blowing up the coordinator? I do not entirely get it...

Thanks
Attila

Sergio mailto:lapostadiser...@gmail.com>> ezt írta 
(időpont: 2020. febr. 21., P 3:44):
The current approach is delete from key_value where id = whatever and it is 
performed asynchronously from the client.
I was thinking to reduce at least the network round-trips between client  and 
coordinator with that Batch approach. :)

In any case, I would test it it will improve or not. So when do you use batch 
then?

Best,

Sergio

On Thu, Feb 20, 2020, 6:18 PM Erick Ramirez 
mailto:erick.rami...@datastax.com>> wrote:
Batches aren't really meant for optimisation in the same way as RDBMS. If 
anything, it will just put pressure on the coordinator having to fire off 
multiple requests to lots of replicas. The IN operator falls into the same 
category and I personally wouldn't use it with more than 2 or 3 partitions 
because then the coordinator will suffer from the same problem.

If it were me, I'd just issue single-partition deletes and throttle it to a 
"reasonable" throughput that your cluster can handle. The word "reasonable" is 
in quotes because only you can determine that magic number for your cluster 
through testing. Cheers!



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: [EXTERNAL] Re: Null values in sasi indexed column

[Durity, Sean R]

I would consider building a lookup table instead. Something like:
Create table new_lookup (
   new-lookup-partition text,
   existing-key text
   PRIMARY KEY (new-lookup-partition)
)

For me, these are easier to understand and reason through for Cassandra 
performance and availability. I would use this approach for up to 3 or 4 
different lookup patterns. If it got to be more than that, I would be using DSE 
Search/SOLR.

Just be warned, I have seen teams asking for these kind of options just because 
they are guessing the access patterns they want. If they cannot identify their 
access patterns, I encourage them to use other technologies. Otherwise the pain 
will be great.


Sean Durity

From: Erick Ramirez 
Sent: Wednesday, February 19, 2020 6:58 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Null values in sasi indexed column

Rahul, in my opinion SASI is an experimental feature and isn't ready for 
primetime yet. It has some advantages over secondary indexes but if it were me, 
I'd stick with native secondary indexes. But test, test and test so you can 
make an informed decision on what works for your use case. Cheers!


Erick Ramirez  |  Developer Relations

erick.rami...@datastax.com | datastax.com 
[datastax.com]
[https://lh4.googleusercontent.com/GivrE4j_1bWvQnZP67Zpa5eomhEeKON-X6kFljLxDatL7QPL_aineBJzM_rXzrqNQkENnZt7KyXLROlLTHuMM3OFNlZ8IrW-adjXKRiD7ojG6OjjFoLio3HbKwVwXt7_Qna02H8I][linkedin.com]
 
[https://lh6.googleusercontent.com/0juOULc-Qhs6qzVY5mN0tzIMZ4w17Jv2fiV5xboewGBH0SFiEwV0uPTO_W5OwGr0jCOXmoJLBq1aNLsr1oChLMgJNvNt1e4bHxO2KJUK-iagQ4jw9SiuTMmpktVSfygdLS_vQe6v]
 
[facebook.com]
 
[https://lh5.googleusercontent.com/IdGeRVBWRf50wPOny50XQ3O0rtkebOh8e2D9DCanVuy-f3a-wpI3PpQJnGtVFL5aHPOwm4hsginvqhQfTXnP_XT_8fuQWS6Mt0KKRFkRANDhS22T3UiXpGfBkMHJxy48ZQJFaXsZ]
 
[twitter.com]
 
[https://lh4.googleusercontent.com/PbPMGIQsTltjGio5a_e7dp35l6ysZMG_ib69EUHmIvbXHXzRkrNKNMfMR8uwSS1AAoQaG6xn96PH-L1wLQE8FBLSjN_g10Q8y0n1Tp5SYtKO3L1JDN_T73HgSSQJayqn7YMTFXn-]
 
[feeds.feedburner.com]
 
[https://lh5.googleusercontent.com/Rnk5QTWTovfX-z1uRr0FQjt17VnMURyI8rDCi4rTJUY5lnX-QevuQWTFa39GS9fJCMqP0SXSkCLtKf064p0-59f80PmA2hZRqGRFFOlZlbJzXv2EevvdbeKYFq4s9g5zzP54KKQB]
 
[github.com]

[https://datastax.com/sites/default/files/content/graphics/email/datastax-email-signature-2019.jpg][datastax.com]



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Sergio]

   - Verify that nodetool upgradesstables has completed successfully on all
   nodes from any previous upgrade
   - Turn off repairs and any other streaming operations (add/remove nodes)
   - Nodetool drain on the node that needs to be stopped (seeds first,
   preferably)
   - Stop an un-upgraded node (seeds first, preferably)
   - Install new binaries and configs on the down node
   - Restart that node and make sure it comes up clean (it will function
   normally in the cluster – even with mixed versions)
   - nodetool statusbinary to verify if it is up and running
   - Repeat for all nodes
   - Once the binary upgrade has been performed in all the nodes: Run
   upgradesstables on each node (as many at a time as your load will allow).
   Minor upgrades usually don’t require this step (only if the sstable format
   has changed), but it is good to check.
   - NOTE: in most cases applications can keep running and will not notice
   much impact – unless the cluster is overloaded and a single node down
   causes impact.



   I added 2 points to the list to clarify.

   Should we add this in a FAQ in the cassandra doc or in the awesome
   cassandra https://cassandra.link/awesome/

   Thanks,

   Sergio


Il giorno mer 12 feb 2020 alle ore 10:58 Durity, Sean R <
sean_r_dur...@homedepot.com> ha scritto:

> Check the readme.txt for any upgrade notes, but the basic procedure is to:
>
>- Verify that nodetool upgradesstables has completed successfully on
>all nodes from any previous upgrade
>- Turn off repairs and any other streaming operations (add/remove
>nodes)
>- Stop an un-upgraded node (seeds first, preferably)
>- Install new binaries and configs on the down node
>- Restart that node and make sure it comes up clean (it will function
>normally in the cluster – even with mixed versions)
>- Repeat for all nodes
>- Run upgradesstables on each node (as many at a time as your load
>will allow). Minor upgrades usually don’t require this step (only if the
>sstable format has changed), but it is good to check.
>- NOTE: in most cases applications can keep running and will not
>notice much impact – unless the cluster is overloaded and a single node
>down causes impact.
>
>
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Sergio 
> *Sent:* Wednesday, February 12, 2020 11:36 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades
>
>
>
> Hi guys!
>
> How do you usually upgrade your cluster for minor version upgrades?
>
> I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
> nodes.
>
> Is there any restriction?
>
> Best,
>
> Sergio
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

Re: [EXTERNAL] Re: Cassandra Encyrption between DC

[Jai Bheemsen Rao Dhanwada]

thank you

On Thu, Feb 13, 2020 at 6:30 AM Durity, Sean R 
wrote:

> I will just add-on that I usually reserve security changes as the primary
> exception where app downtime may be necessary with Cassandra. (DSE has some
> Transitional tools that are useful, though.) Sometimes a short outage is
> preferred over a longer, more-complicated attempt to keep the app up. And,
> in many cases, there is no way to guarantee availability when making
> security-related changes (new cipher suites, adding encryption, turning on
> authentication, etc.). It is better to try and have those implemented from
> the beginning, where possible.
>
>
>
>
>
> Sean Durity
>
>
>
> *From:* Erick Ramirez 
> *Sent:* Wednesday, February 12, 2020 9:02 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: Cassandra Encyrption between DC
>
>
>
> I've just seen your questions on ASF Slack and didn't immediately make the
> connection that this post in the mailing list is one and the same. I
> understand what you're doing now -- you have an existing DC with no
> encryption and you want to add a new DC with encryption enabled but don't
> want the downtime associated with enabling encryption on the existing DC.
>
>
>
> As driftx, exlt, myself & co pointed out, there isn't a "transitional
> path" of implementing it without downtime in the current (released)
> versions of C*. Cheers!
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] Cassandra 3.11.X upgrades

[Durity, Sean R]

+1 on nodetool drain. I added that to our upgrade automation and it really 
helps with post-upgrade start-up time.

Sean Durity

From: Erick Ramirez 
Sent: Wednesday, February 12, 2020 10:29 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Cassandra 3.11.X upgrades

Yes to the steps. The only thing I would add is to run a nodetool drain before 
shutting C* down so all mutations are flushed to SSTables and there won't be 
any commit logs to replay on startup.

Also, the usual "backup your cluster and configuration files" boilerplate 
applies. 



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: [EXTERNAL] Re: Cassandra Encyrption between DC

[Durity, Sean R]

I will just add-on that I usually reserve security changes as the primary 
exception where app downtime may be necessary with Cassandra. (DSE has some 
Transitional tools that are useful, though.) Sometimes a short outage is 
preferred over a longer, more-complicated attempt to keep the app up. And, in 
many cases, there is no way to guarantee availability when making 
security-related changes (new cipher suites, adding encryption, turning on 
authentication, etc.). It is better to try and have those implemented from the 
beginning, where possible.


Sean Durity

From: Erick Ramirez 
Sent: Wednesday, February 12, 2020 9:02 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Cassandra Encyrption between DC

I've just seen your questions on ASF Slack and didn't immediately make the 
connection that this post in the mailing list is one and the same. I understand 
what you're doing now -- you have an existing DC with no encryption and you 
want to add a new DC with encryption enabled but don't want the downtime 
associated with enabling encryption on the existing DC.

As driftx, exlt, myself & co pointed out, there isn't a "transitional path" of 
implementing it without downtime in the current (released) versions of C*. 
Cheers!



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Erick Ramirez]

Yes to the steps. The only thing I would add is to run a nodetool drain
before shutting C* down so all mutations are flushed to SSTables and there
won't be any commit logs to replay on startup.

Also, the usual "backup your cluster and configuration files" boilerplate
applies. 

>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Sergio]

Should I follow the steps above right?
Thanks Erick!

On Wed, Feb 12, 2020, 6:58 PM Erick Ramirez 
wrote:

> In case you have an hybrid situation with 3.11.3 , 3.11.4 and 3.11.5 that
>> it is working and it is in production what do you recommend?
>
>
> You shouldn't end up in this mixed-version situation at all. I would
> highly recommend you upgrade all the nodes to 3.11.5 or whatever the latest
> version is installed on the nodes. Mixed-versions isn't a tested or
> supported scenario and the cluster's behaviour can be unpredictable. The
> behaviour might not be catastrophic but you don't want to be the one who
> discovers some exotic bug that arises out of that configuration. Cheers!
>
>>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Erick Ramirez]

>
> In case you have an hybrid situation with 3.11.3 , 3.11.4 and 3.11.5 that
> it is working and it is in production what do you recommend?


You shouldn't end up in this mixed-version situation at all. I would highly
recommend you upgrade all the nodes to 3.11.5 or whatever the latest
version is installed on the nodes. Mixed-versions isn't a tested or
supported scenario and the cluster's behaviour can be unpredictable. The
behaviour might not be catastrophic but you don't want to be the one who
discovers some exotic bug that arises out of that configuration. Cheers!

>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Sergio]

Thanks everyone!

In case you have an hybrid situation with 3.11.3 , 3.11.4 and 3.11.5 that
it is working and it is in production what do you recommend?



On Wed, Feb 12, 2020, 5:55 PM Erick Ramirez 
wrote:

> So unless the sstable format has not been changed I can avoid to do that.
>
>
> Just to reinforce what Jon and Sean already said, the above assumption is
> dangerous. It is always best to follow the recommended upgrade procedure
> and mixed-versions is never a good idea unless you've received instructions
> from a qualified source to address a specific issue. But as Jon said, we
> wouldn't be on this mailing list otherwise. 
>
> Erick Ramirez  |  Developer Relations
>
> erick.rami...@datastax.com | datastax.com 
> 
>  
>  
>
> 
>
>>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Erick Ramirez]

>
> So unless the sstable format has not been changed I can avoid to do that.


Just to reinforce what Jon and Sean already said, the above assumption is
dangerous. It is always best to follow the recommended upgrade procedure
and mixed-versions is never a good idea unless you've received instructions
from a qualified source to address a specific issue. But as Jon said, we
wouldn't be on this mailing list otherwise. 

Erick Ramirez  |  Developer Relations

erick.rami...@datastax.com | datastax.com 

 
 



>

RE: [EXTERNAL] Cassandra 3.11.X upgrades

[Durity, Sean R]

Ah - I should have looked it up! Thank you for fixing my mistake.

Sean Durity

-Original Message-
From: Michael Shuler 
Sent: Wednesday, February 12, 2020 3:17 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Cassandra 3.11.X upgrades

On 2/12/20 12:58 PM, Durity, Sean R wrote:
> Check the readme.txt for any upgrade notes

Just a quick correction:

NEWS.txt (upgrade (and other important) notes)
CHANGES.txt (changelog with JIRAs)

This is why we list links to these two files in the release announcements.

--
Kind regards,
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Reid Pinchback]

Hi Sergio,

We have a production cluster with vnodes=4 that is a bit larger than that, so 
yes it is possible to do so.  That said, we aren’t wedded to vnodes=4 and are 
paying attention to discussions happening around the 4.0 work and mulling the 
possibility of shifting to 16.

Note though, we didn’t just pick 4 based on blind faith.  There was work done 
to evaluate the state of our token distribution before and after.  It’s pretty 
much true of all the settings we have for C* (or anything else for that 
matter).  We started from a lot of helpful articles and docs and mail threads 
and JIRA issues, but then we evaluated to see what we thought of the results.  
It was the only way for us to start building up some understanding of the 
details.

From: Sergio 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, February 12, 2020 at 2:50 PM
To: "user@cassandra.apache.org" 
Subject: Re: [EXTERNAL] Cassandra 3.11.X upgrades

Message from External Sender
Thanks, everyone! @Jon 
https://lists.apache.org/thread.html/rd18814bfba487824ca95a58191f4dcdb86f15c9bb66cf2bcc29ddf0b%40%3Cuser.cassandra.apache.org%3E<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_rd18814bfba487824ca95a58191f4dcdb86f15c9bb66cf2bcc29ddf0b-2540-253Cuser.cassandra.apache.org-253E=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=GkUeet_u9KFI2OlzYvn34AhugRwTmQiTA-Rc9YYBFx8=Mwa2C1h8fwl0Qxfi75pISpAUVX18lhGVyHAjO8hwk3o=>
I have a side response to something that looks to be controversial with the 
response from Anthony.
So is it safe to go to production in a 1TB cluster with vnodes = 4?
Do we need to follow these steps 
https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__thelastpickle.com_blog_2019_02_21_set-2Dup-2Da-2Dcluster-2Dwith-2Deven-2Dtoken-2Ddistribution.html=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=GkUeet_u9KFI2OlzYvn34AhugRwTmQiTA-Rc9YYBFx8=jVKn-7nVIP3_zUh6LjYMG-X1KyUZPcIvV-m0Cy82Ttk=>?
 because from Anthony response what I got that this is just an example and 
vnodes = 4 it is not ready for production.
https://lists.apache.org/thread.html/r21cd99fa269076d186a82a8b466eb925681373302dd7aa6bb26e5bde%40%3Cuser.cassandra.apache.org%3E<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_r21cd99fa269076d186a82a8b466eb925681373302dd7aa6bb26e5bde-2540-253Cuser.cassandra.apache.org-253E=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=GkUeet_u9KFI2OlzYvn34AhugRwTmQiTA-Rc9YYBFx8=r5CPYag3chuqUfol7RR8NKj31MC27P-gONPzaKhkeJ4=>

Best,

Sergio


Il giorno mer 12 feb 2020 alle ore 11:42 Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> ha scritto:
>>A while ago, on my first cluster

Understatement used so effectively. Jon is a master.



On Wed, Feb 12, 2020 at 11:02 AM Sergio 
mailto:lapostadiser...@gmail.com>> wrote:
Thanks for your reply!

So unless the sstable format has not been changed I can avoid to do that.

Correct?

Best,

Sergio

On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
Check the readme.txt for any upgrade notes, but the basic procedure is to:

  *   Verify that nodetool upgradesstables has completed successfully on all 
nodes from any previous upgrade
  *   Turn off repairs and any other streaming operations (add/remove nodes)
  *   Stop an un-upgraded node (seeds first, preferably)
  *   Install new binaries and configs on the down node
  *   Restart that node and make sure it comes up clean (it will function 
normally in the cluster – even with mixed versions)
  *   Repeat for all nodes
  *   Run upgradesstables on each node (as many at a time as your load will 
allow). Minor upgrades usually don’t require this step (only if the sstable 
format has changed), but it is good to check.
  *   NOTE: in most cases applications can keep running and will not notice 
much impact – unless the cluster is overloaded and a single node down causes 
impact.



Sean Durity – Staff Systems Engineer, Cassandra

From: Sergio mailto:lapostadiser...@gmail.com>>
Sent: Wednesday, February 12, 2020 11:36 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Cassandra 3.11.X upgrades

Hi guys!

How do you usually upgrade your cluster for minor version upgrades?

I tried to add a node with 3.11.5 version to a test cluster with 3.11.4 nodes.

Is there any restriction?

Best,

Sergio



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitt

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Michael Shuler]

On 2/12/20 12:58 PM, Durity, Sean R wrote:

Check the readme.txt for any upgrade notes


Just a quick correction:

NEWS.txt (upgrade (and other important) notes)
CHANGES.txt (changelog with JIRAs)

This is why we list links to these two files in the release announcements.

--
Kind regards,
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Sergio]

Thanks, everyone! @Jon
https://lists.apache.org/thread.html/rd18814bfba487824ca95a58191f4dcdb86f15c9bb66cf2bcc29ddf0b%40%3Cuser.cassandra.apache.org%3E

I have a side response to something that looks to be controversial with the
response from Anthony.
So is it safe to go to production in a 1TB cluster with vnodes = 4?
Do we need to follow these steps
https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html?
because
from Anthony response what I got that this is just an example and vnodes =
4 it is not ready for production.
https://lists.apache.org/thread.html/r21cd99fa269076d186a82a8b466eb925681373302dd7aa6bb26e5bde%40%3Cuser.cassandra.apache.org%3E

Best,

Sergio



Il giorno mer 12 feb 2020 alle ore 11:42 Durity, Sean R <
sean_r_dur...@homedepot.com> ha scritto:

> >>A while ago, on my first cluster
>
>
>
> Understatement used so effectively. Jon is a master.
>
>
>
>
>
>
>
> On Wed, Feb 12, 2020 at 11:02 AM Sergio  wrote:
>
> Thanks for your reply!
>
>
>
> So unless the sstable format has not been changed I can avoid to do that.
>
>
>
> Correct?
>
>
>
> Best,
>
>
>
> Sergio
>
>
>
> On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
> wrote:
>
> Check the readme.txt for any upgrade notes, but the basic procedure is to:
>
>- Verify that nodetool upgradesstables has completed successfully on
>all nodes from any previous upgrade
>- Turn off repairs and any other streaming operations (add/remove
>nodes)
>- Stop an un-upgraded node (seeds first, preferably)
>- Install new binaries and configs on the down node
>- Restart that node and make sure it comes up clean (it will function
>normally in the cluster – even with mixed versions)
>- Repeat for all nodes
>- Run upgradesstables on each node (as many at a time as your load
>will allow). Minor upgrades usually don’t require this step (only if the
>sstable format has changed), but it is good to check.
>- NOTE: in most cases applications can keep running and will not
>notice much impact – unless the cluster is overloaded and a single node
>down causes impact.
>
>
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Sergio 
> *Sent:* Wednesday, February 12, 2020 11:36 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades
>
>
>
> Hi guys!
>
> How do you usually upgrade your cluster for minor version upgrades?
>
> I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
> nodes.
>
> Is there any restriction?
>
> Best,
>
> Sergio
>
>
> --
>
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] Cassandra 3.11.X upgrades

[Durity, Sean R]

>>A while ago, on my first cluster

Understatement used so effectively. Jon is a master.



On Wed, Feb 12, 2020 at 11:02 AM Sergio 
mailto:lapostadiser...@gmail.com>> wrote:
Thanks for your reply!

So unless the sstable format has not been changed I can avoid to do that.

Correct?

Best,

Sergio

On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
Check the readme.txt for any upgrade notes, but the basic procedure is to:

  *   Verify that nodetool upgradesstables has completed successfully on all 
nodes from any previous upgrade
  *   Turn off repairs and any other streaming operations (add/remove nodes)
  *   Stop an un-upgraded node (seeds first, preferably)
  *   Install new binaries and configs on the down node
  *   Restart that node and make sure it comes up clean (it will function 
normally in the cluster – even with mixed versions)
  *   Repeat for all nodes
  *   Run upgradesstables on each node (as many at a time as your load will 
allow). Minor upgrades usually don’t require this step (only if the sstable 
format has changed), but it is good to check.
  *   NOTE: in most cases applications can keep running and will not notice 
much impact – unless the cluster is overloaded and a single node down causes 
impact.



Sean Durity – Staff Systems Engineer, Cassandra

From: Sergio mailto:lapostadiser...@gmail.com>>
Sent: Wednesday, February 12, 2020 11:36 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra 3.11.X upgrades

Hi guys!

How do you usually upgrade your cluster for minor version upgrades?

I tried to add a node with 3.11.5 version to a test cluster with 3.11.4 nodes.

Is there any restriction?

Best,

Sergio



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Jon Haddad]

A while ago, on my first cluster, I decided to do an upgrade by adding
nodes running 1.2 to an existing cluster running version 1.1.  This was a
bad decision, and at that point I decided to always play it safe and always
stick to a single version, and never bootstrap in a node running different
version to a cluster.

I would do this even when it's a trivial upgrade, or a single commit is
different only touching a few lines of code.

Why?

I do this because in my opinion it's better to have a single way of doing
things.  Getting in the routine of doing minor upgrades by following Sean's
upgrade checklist will always work.  It's the supported, correct way of
doing things, and if you decide to always follow the same checklist, the
possibility of doing things wrong when there is an
incompatibility decreases.  Think of it as practice.

The only time you should even consider mixed version bootstrap upgrades is
if you know enough to not have to ask the list.  Even then, I go back to my
previous point about practice.  No point in practicing the less safe
version of things.

My 2 cents.

Jon



On Wed, Feb 12, 2020 at 11:02 AM Sergio  wrote:

> Thanks for your reply!
>
> So unless the sstable format has not been changed I can avoid to do that.
>
> Correct?
>
> Best,
>
> Sergio
>
> On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
> wrote:
>
>> Check the readme.txt for any upgrade notes, but the basic procedure is to:
>>
>>- Verify that nodetool upgradesstables has completed successfully on
>>all nodes from any previous upgrade
>>- Turn off repairs and any other streaming operations (add/remove
>>nodes)
>>- Stop an un-upgraded node (seeds first, preferably)
>>- Install new binaries and configs on the down node
>>- Restart that node and make sure it comes up clean (it will function
>>normally in the cluster – even with mixed versions)
>>- Repeat for all nodes
>>- Run upgradesstables on each node (as many at a time as your load
>>will allow). Minor upgrades usually don’t require this step (only if the
>>sstable format has changed), but it is good to check.
>>- NOTE: in most cases applications can keep running and will not
>>notice much impact – unless the cluster is overloaded and a single node
>>down causes impact.
>>
>>
>>
>>
>>
>>
>>
>> Sean Durity – Staff Systems Engineer, Cassandra
>>
>>
>>
>> *From:* Sergio 
>> *Sent:* Wednesday, February 12, 2020 11:36 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades
>>
>>
>>
>> Hi guys!
>>
>> How do you usually upgrade your cluster for minor version upgrades?
>>
>> I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
>> nodes.
>>
>> Is there any restriction?
>>
>> Best,
>>
>> Sergio
>>
>> --
>>
>> The information in this Internet Email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this Email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful. When addressed
>> to our clients any opinions or advice contained in this Email are subject
>> to the terms and conditions expressed in any applicable governing The Home
>> Depot terms of business or client engagement letter. The Home Depot
>> disclaims all responsibility and liability for the accuracy and content of
>> this attachment and for any damages or losses arising from any
>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>> items of a destructive nature, which may be contained in this attachment
>> and shall not be liable for direct, indirect, consequential or special
>> damages in connection with this e-mail message or its attachment.
>>
>

Re: [EXTERNAL] Cassandra 3.11.X upgrades

[Sergio]

Thanks for your reply!

So unless the sstable format has not been changed I can avoid to do that.

Correct?

Best,

Sergio

On Wed, Feb 12, 2020, 10:58 AM Durity, Sean R 
wrote:

> Check the readme.txt for any upgrade notes, but the basic procedure is to:
>
>- Verify that nodetool upgradesstables has completed successfully on
>all nodes from any previous upgrade
>- Turn off repairs and any other streaming operations (add/remove
>nodes)
>- Stop an un-upgraded node (seeds first, preferably)
>- Install new binaries and configs on the down node
>- Restart that node and make sure it comes up clean (it will function
>normally in the cluster – even with mixed versions)
>- Repeat for all nodes
>- Run upgradesstables on each node (as many at a time as your load
>will allow). Minor upgrades usually don’t require this step (only if the
>sstable format has changed), but it is good to check.
>- NOTE: in most cases applications can keep running and will not
>notice much impact – unless the cluster is overloaded and a single node
>down causes impact.
>
>
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Sergio 
> *Sent:* Wednesday, February 12, 2020 11:36 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra 3.11.X upgrades
>
>
>
> Hi guys!
>
> How do you usually upgrade your cluster for minor version upgrades?
>
> I tried to add a node with 3.11.5 version to a test cluster with 3.11.4
> nodes.
>
> Is there any restriction?
>
> Best,
>
> Sergio
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] Cassandra 3.11.X upgrades

[Durity, Sean R]

Check the readme.txt for any upgrade notes, but the basic procedure is to:

  *   Verify that nodetool upgradesstables has completed successfully on all 
nodes from any previous upgrade
  *   Turn off repairs and any other streaming operations (add/remove nodes)
  *   Stop an un-upgraded node (seeds first, preferably)
  *   Install new binaries and configs on the down node
  *   Restart that node and make sure it comes up clean (it will function 
normally in the cluster – even with mixed versions)
  *   Repeat for all nodes
  *   Run upgradesstables on each node (as many at a time as your load will 
allow). Minor upgrades usually don’t require this step (only if the sstable 
format has changed), but it is good to check.
  *   NOTE: in most cases applications can keep running and will not notice 
much impact – unless the cluster is overloaded and a single node down causes 
impact.



Sean Durity – Staff Systems Engineer, Cassandra

From: Sergio 
Sent: Wednesday, February 12, 2020 11:36 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra 3.11.X upgrades

Hi guys!

How do you usually upgrade your cluster for minor version upgrades?

I tried to add a node with 3.11.5 version to a test cluster with 3.11.4 nodes.

Is there any restriction?

Best,

Sergio



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] How to reduce vnodes without downtime

[Erick Ramirez]

>
> I am seeing some unbalancing and I was worried because I have 256 vnodes
> Weird stuff is related to this post where I don't find a match between the
> load and du -sh * for the node 10.1.31.60 and I was trying to figure out
> the reason, if it was due to the number of vnodes.


Out of curiosity, did you start with a smaller cluster then added new ones?
Just wondering if this is a case of not having ran nodetool cleanup
post-expansion.

Does Cassandra keep a copy of the data per rack so if I need to keep the
> things balanced and I would have to add 3 racks at the time in a single
> Datacenter keep the things balanced?


The output you posted shows that all nodes are in the same rack but yes, C*
will place a replica in each rack so that each rack has a full copy.
Caveats apply such as RF=3 and 3 racks in the DC.

Is it better to keep a single Rack with a single Datacenter in 3 different
> availability zones with replication factor = 3 or to have for each
> Datacenter: 1 Rack and 1 Availability Zone and eventually redirect the
> client to a fallback Datacenter in case one of the availability zone is not
> reachable


If you have RF=3 and EC2 instances in 3 AZs, you can choose to allocate the
instances to a logical C* rack based on their AZ. However, you would only
do this if each AZ has identical number of nodes in each. If the node count
in each rack is not identical, you will end up with some bloated nodes for
the same reason that C* will keep a full copy of data in each rack.

Using racks also means that when you want to expand the cluster, you need
to provision instances in each AZ. As above, if you only provision
instances in 2 of 3 AZs (for example) then nodes in 1 AZ will be "fatter"
than nodes in the other 2 AZs.

Failing over to another DC isn't really necessary if only 1 AZ is
unreachable if using a CL of LOCAL_QUORUM since 2 AZs is sufficient to
satisfy requests. But that really depends on your application's business
rules. Cheers!

>

Re: [EXTERNAL] How to reduce vnodes without downtime

[Sergio]

num_tokens default in Cassandra 4.0
>> <https://www.mail-archive.com/search?l=dev%40cassandra.apache.org=subject%3A%22%5C%5BDiscuss%5C%5D+num_tokens+default+in+Cassandra+4.0%22=oldest>
>> ".
>>
>> Regards,
>> Anthony
>>
>> On Sat, 1 Feb 2020 at 10:07, Sergio  wrote:
>>
>>>
>>> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>>>  This
>>> is the article with 4 token recommendations.
>>> @Erick Ramirez. which is the dev thread for the default 32 tokens
>>> recommendation?
>>>
>>> Thanks,
>>> Sergio
>>>
>>> Il giorno ven 31 gen 2020 alle ore 14:49 Erick Ramirez <
>>> flightc...@gmail.com> ha scritto:
>>>
>>>> There's an active discussion going on right now in a separate dev
>>>> thread. The current "default recommendation" is 32 tokens. But there's a
>>>> push for 4 in combination with allocate_tokens_for_keyspace from Jon
>>>> Haddad & co (based on a paper from Joe Lynch & Josh Snyder).
>>>>
>>>> If you're satisfied with the results from your own testing, go with 4
>>>> tokens. And that's the key -- you must test, test, TEST! Cheers!
>>>>
>>>> On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon 
>>>> wrote:
>>>>
>>>>> What is recommended vnodes now? I read 8 in later cassandra 3.x
>>>>> Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
>>>>> Thanks
>>>>>
>>>>> On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R <
>>>>> sean_r_dur...@homedepot.com> wrote:
>>>>>
>>>>>> These are good clarifications and expansions.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Sean Durity
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* Anthony Grasso 
>>>>>> *Sent:* Thursday, January 30, 2020 7:25 PM
>>>>>> *To:* user 
>>>>>> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Maxim,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Basically what Sean suggested is the way to do this without downtime.
>>>>>>
>>>>>>
>>>>>>
>>>>>> To clarify the, the *three* steps following the "Decommission each
>>>>>> node in the DC you are working on" step should be applied to *only*
>>>>>> the decommissioned nodes. So where it say "*all nodes*" or "*every
>>>>>> node*" it applies to only the decommissioned nodes.
>>>>>>
>>>>>>
>>>>>>
>>>>>> In addition, the step that says "Wipe data on all the nodes", I would
>>>>>> delete all files in the following directories on the decommissioned 
>>>>>> nodes.
>>>>>>
>>>>>>- data (usually located in /var/lib/cassandra/data)
>>>>>>- commitlogs (usually located in /var/lib/cassandra/commitlogs)
>>>>>>- hints (usually located in /var/lib/casandra/hints)
>>>>>>- saved_caches (usually located in
>>>>>>/var/lib/cassandra/saved_caches)
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Anthony
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <
>>>>>> sean_r_dur...@homedepot.com> wrote:
>>>>>>
>>>>>> Your procedure won’t work very well. On the first node, if you
>>>>>> switched to 4, you would end up with only a tiny fraction of the data
>>>>>> (because the other nodes would still be at 256). I updated a large 
>>>>>> cluster
>>>>>> (over 150 nodes – 2 DCs) to smaller number of vnodes. The basic outline 
>>>>>> was
>>>>>> this:
>>>>>>
>>>>>>
>>>>>>
>>>>>>- Stop all repairs
>>>>>>- Make sure the app is running against one DC only
>>>>>>- Change the replication settings on keyspaces to use only 1 DC
>>>>&

Re: [EXTERNAL] Re: Running select against cassandra

[Reid Pinchback]

I defer to Sean’s comment on materialized views.  I’m more familiar with 
DynamoDB on that front, where you do this pretty routinely.  I was curious so I 
went looking. This appears to be the C* Jira that points to many of the problem 
points:

https://issues.apache.org/jira/browse/CASSANDRA-13826

Abdul, you’d probably want to refer to that or similar info.  Could be that the 
more practical resolution is to just have the client write the data twice, if 
there are two very different query patterns to support.  Writes usually have 
quite low latency in C*, so double-writing may be less of a performance hit, 
and later drag on memory on I/O, than a query model that makes you browse 
through more data than necessary.

From: "Durity, Sean R" 
Reply-To: "user@cassandra.apache.org" 
Date: Thursday, February 6, 2020 at 4:24 PM
To: "user@cassandra.apache.org" 
Subject: RE: [EXTERNAL] Re: Running select against cassandra

Message from External Sender
Reid is right. You build the tables to easily answer the queries you want. So, 
start with the query! I inferred a query for you based on what you mentioned. 
If my inference is wrong, the table structure is likely wrong, too.

So, what kind of query do you want to run?

(NOTE: a select count(*) that is not restricted to within a single partition is 
a very bad option. Don’t do that)

The query for my table below is simply:
select user_count [, other columns] from users_by_day where date = ? and hour = 
? and minute = ?


Sean Durity

From: Reid Pinchback 
Sent: Thursday, February 6, 2020 4:10 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Re: Running select against cassandra

Abdul,

When in doubt, have a query model that immediately feeds you exactly what you 
are looking for. That’s kind of the data model philosophy that you want to 
shoot for as much as feasible with C*.

The point of Sean’s table isn’t the similarity to yours, it is how he has it 
keyed because it suits a partition structure much better aligned with what you 
want to request.  So I’d say yes, if a materialized view is how you want to 
achieve a denormalized state where the query model directly supports giving you 
want you want to query for, that sounds like an appropriate option to consider. 
 You might want a composite partition key for having an efficient selection of 
narrow time ranges.

From: Abdul Patel mailto:abd786...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Thursday, February 6, 2020 at 2:42 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: [EXTERNAL] Re: Running select against cassandra

Message from External Sender
this is the schema similar to what we have , they want to get user connected  - 
concurrent count for every say 1-5 minutes.
i am thinking will simple select will have performance issue or we can go for 
materialized views ?

CREATE TABLE  usr_session (
userid bigint,
session_usr text,
last_access_time timestamp,
login_time timestamp,
status int,
PRIMARY KEY (userid, session_usr)
) WITH CLUSTERING ORDER BY (session_usr ASC)


On Thu, Feb 6, 2020 at 2:09 PM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
Do you only need the current count or do you want to keep the historical counts 
also? By active users, does that mean some kind of user that the application 
tracks (as opposed to the Cassandra user connected to the cluster)?

I would consider a table like this for tracking active users through time:

Create table users_by_day (
app_date date,
hour integer,
minute integer,
user_count integer,
longest_login_user text,
longest_login_seconds integer,
last_login datetime,
last_login_user text )
primary key (app_date, hour, minute);

Then, your reporting can easily select full days or a specific, one-minute 
slice. Of course, the app would need to have a timer and write out the data. I 
would also suggest a TTL on the data so that you only keep what you need (a 
week, a year, whatever). Of course, if your reporting requires different 
granularities, you could consider a different time bucket for the table (by 
hour, by week, etc.)


Sean Durity – Staff Systems Engineer, Cassandra

From: Abdul Patel mailto:abd786...@gmail.com>>
Sent: Thursday, February 6, 2020 1:54 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Re: Running select against cassandra

Its sort of user connected, app team needa number of active users connected say 
 every 1 to 5 mins.
The timeout at app end is 120ms.



On Thursday, February 6, 2020, Michael Shuler 
mailto:mich...@pbandjelly.org>> wrote:
You'll have to be more specific. What is your table schema and what is the 
SELECT query? What is the normal response time?

As a basic guide for your general question, if

RE: [EXTERNAL] Re: Running select against cassandra

[Durity, Sean R]

Reid is right. You build the tables to easily answer the queries you want. So, 
start with the query! I inferred a query for you based on what you mentioned. 
If my inference is wrong, the table structure is likely wrong, too.

So, what kind of query do you want to run?

(NOTE: a select count(*) that is not restricted to within a single partition is 
a very bad option. Don’t do that)

The query for my table below is simply:
select user_count [, other columns] from users_by_day where date = ? and hour = 
? and minute = ?


Sean Durity

From: Reid Pinchback 
Sent: Thursday, February 6, 2020 4:10 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Re: Running select against cassandra

Abdul,

When in doubt, have a query model that immediately feeds you exactly what you 
are looking for. That’s kind of the data model philosophy that you want to 
shoot for as much as feasible with C*.

The point of Sean’s table isn’t the similarity to yours, it is how he has it 
keyed because it suits a partition structure much better aligned with what you 
want to request.  So I’d say yes, if a materialized view is how you want to 
achieve a denormalized state where the query model directly supports giving you 
want you want to query for, that sounds like an appropriate option to consider. 
 You might want a composite partition key for having an efficient selection of 
narrow time ranges.

From: Abdul Patel mailto:abd786...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Thursday, February 6, 2020 at 2:42 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: [EXTERNAL] Re: Running select against cassandra

Message from External Sender
this is the schema similar to what we have , they want to get user connected  - 
concurrent count for every say 1-5 minutes.
i am thinking will simple select will have performance issue or we can go for 
materialized views ?

CREATE TABLE  usr_session (
userid bigint,
session_usr text,
last_access_time timestamp,
login_time timestamp,
status int,
PRIMARY KEY (userid, session_usr)
) WITH CLUSTERING ORDER BY (session_usr ASC)


On Thu, Feb 6, 2020 at 2:09 PM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
Do you only need the current count or do you want to keep the historical counts 
also? By active users, does that mean some kind of user that the application 
tracks (as opposed to the Cassandra user connected to the cluster)?

I would consider a table like this for tracking active users through time:

Create table users_by_day (
app_date date,
hour integer,
minute integer,
user_count integer,
longest_login_user text,
longest_login_seconds integer,
last_login datetime,
last_login_user text )
primary key (app_date, hour, minute);

Then, your reporting can easily select full days or a specific, one-minute 
slice. Of course, the app would need to have a timer and write out the data. I 
would also suggest a TTL on the data so that you only keep what you need (a 
week, a year, whatever). Of course, if your reporting requires different 
granularities, you could consider a different time bucket for the table (by 
hour, by week, etc.)


Sean Durity – Staff Systems Engineer, Cassandra

From: Abdul Patel mailto:abd786...@gmail.com>>
Sent: Thursday, February 6, 2020 1:54 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Re: Running select against cassandra

Its sort of user connected, app team needa number of active users connected say 
 every 1 to 5 mins.
The timeout at app end is 120ms.



On Thursday, February 6, 2020, Michael Shuler 
mailto:mich...@pbandjelly.org>> wrote:
You'll have to be more specific. What is your table schema and what is the 
SELECT query? What is the normal response time?

As a basic guide for your general question, if the query is something sort of 
irrelevant that should be stored some other way, like a total row count, or 
most any SELECT that requires ALLOW FILTERING, you're doing it wrong and should 
re-evaluate your data model.

1 query per minute is a minuscule fraction of the basic capacity of queries per 
minute that a Cassandra cluster should be able to handle with good data 
modeling and table-relevant query. All depends on the data model and query.

Michael

On 2/6/20 12:20 PM, Abdul Patel wrote:
Hi,

Is it advisable to run select query to fetch every minute to grab data from 
cassandra for reporting purpose, if no then whats the alternative?

-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org<mailto:user-unsubscr...@cassandra.apache.org>
For additional commands, e-mail: 
user-h...@cassandra.apache.org<mailto:user-h...@cassandra.apache.org>

RE: [EXTERNAL] Re: Running select against cassandra

[Durity, Sean R]

From reports on this mailing list, I do not allow materialized views.


Sean Durity

From: Reid Pinchback 
Sent: Thursday, February 6, 2020 4:10 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Re: Running select against cassandra

Abdul,

When in doubt, have a query model that immediately feeds you exactly what you 
are looking for. That’s kind of the data model philosophy that you want to 
shoot for as much as feasible with C*.

The point of Sean’s table isn’t the similarity to yours, it is how he has it 
keyed because it suits a partition structure much better aligned with what you 
want to request.  So I’d say yes, if a materialized view is how you want to 
achieve a denormalized state where the query model directly supports giving you 
want you want to query for, that sounds like an appropriate option to consider. 
 You might want a composite partition key for having an efficient selection of 
narrow time ranges.

From: Abdul Patel mailto:abd786...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Thursday, February 6, 2020 at 2:42 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: [EXTERNAL] Re: Running select against cassandra

Message from External Sender
this is the schema similar to what we have , they want to get user connected  - 
concurrent count for every say 1-5 minutes.
i am thinking will simple select will have performance issue or we can go for 
materialized views ?

CREATE TABLE  usr_session (
userid bigint,
session_usr text,
last_access_time timestamp,
login_time timestamp,
status int,
PRIMARY KEY (userid, session_usr)
) WITH CLUSTERING ORDER BY (session_usr ASC)


On Thu, Feb 6, 2020 at 2:09 PM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
Do you only need the current count or do you want to keep the historical counts 
also? By active users, does that mean some kind of user that the application 
tracks (as opposed to the Cassandra user connected to the cluster)?

I would consider a table like this for tracking active users through time:

Create table users_by_day (
app_date date,
hour integer,
minute integer,
user_count integer,
longest_login_user text,
longest_login_seconds integer,
last_login datetime,
last_login_user text )
primary key (app_date, hour, minute);

Then, your reporting can easily select full days or a specific, one-minute 
slice. Of course, the app would need to have a timer and write out the data. I 
would also suggest a TTL on the data so that you only keep what you need (a 
week, a year, whatever). Of course, if your reporting requires different 
granularities, you could consider a different time bucket for the table (by 
hour, by week, etc.)


Sean Durity – Staff Systems Engineer, Cassandra

From: Abdul Patel mailto:abd786...@gmail.com>>
Sent: Thursday, February 6, 2020 1:54 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Re: Running select against cassandra

Its sort of user connected, app team needa number of active users connected say 
 every 1 to 5 mins.
The timeout at app end is 120ms.



On Thursday, February 6, 2020, Michael Shuler 
mailto:mich...@pbandjelly.org>> wrote:
You'll have to be more specific. What is your table schema and what is the 
SELECT query? What is the normal response time?

As a basic guide for your general question, if the query is something sort of 
irrelevant that should be stored some other way, like a total row count, or 
most any SELECT that requires ALLOW FILTERING, you're doing it wrong and should 
re-evaluate your data model.

1 query per minute is a minuscule fraction of the basic capacity of queries per 
minute that a Cassandra cluster should be able to handle with good data 
modeling and table-relevant query. All depends on the data model and query.

Michael

On 2/6/20 12:20 PM, Abdul Patel wrote:
Hi,

Is it advisable to run select query to fetch every minute to grab data from 
cassandra for reporting purpose, if no then whats the alternative?

-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org<mailto:user-unsubscr...@cassandra.apache.org>
For additional commands, e-mail: 
user-h...@cassandra.apache.org<mailto:user-h...@cassandra.apache.org>



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 

Re: [EXTERNAL] Re: Running select against cassandra

[Reid Pinchback]

Abdul,

When in doubt, have a query model that immediately feeds you exactly what you 
are looking for. That’s kind of the data model philosophy that you want to 
shoot for as much as feasible with C*.

The point of Sean’s table isn’t the similarity to yours, it is how he has it 
keyed because it suits a partition structure much better aligned with what you 
want to request.  So I’d say yes, if a materialized view is how you want to 
achieve a denormalized state where the query model directly supports giving you 
want you want to query for, that sounds like an appropriate option to consider. 
 You might want a composite partition key for having an efficient selection of 
narrow time ranges.

From: Abdul Patel 
Reply-To: "user@cassandra.apache.org" 
Date: Thursday, February 6, 2020 at 2:42 PM
To: "user@cassandra.apache.org" 
Subject: Re: [EXTERNAL] Re: Running select against cassandra

Message from External Sender
this is the schema similar to what we have , they want to get user connected  - 
concurrent count for every say 1-5 minutes.
i am thinking will simple select will have performance issue or we can go for 
materialized views ?

CREATE TABLE  usr_session (
userid bigint,
session_usr text,
last_access_time timestamp,
login_time timestamp,
status int,
PRIMARY KEY (userid, session_usr)
) WITH CLUSTERING ORDER BY (session_usr ASC)


On Thu, Feb 6, 2020 at 2:09 PM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
Do you only need the current count or do you want to keep the historical counts 
also? By active users, does that mean some kind of user that the application 
tracks (as opposed to the Cassandra user connected to the cluster)?

I would consider a table like this for tracking active users through time:

Create table users_by_day (
app_date date,
hour integer,
minute integer,
user_count integer,
longest_login_user text,
longest_login_seconds integer,
last_login datetime,
last_login_user text )
primary key (app_date, hour, minute);

Then, your reporting can easily select full days or a specific, one-minute 
slice. Of course, the app would need to have a timer and write out the data. I 
would also suggest a TTL on the data so that you only keep what you need (a 
week, a year, whatever). Of course, if your reporting requires different 
granularities, you could consider a different time bucket for the table (by 
hour, by week, etc.)


Sean Durity – Staff Systems Engineer, Cassandra

From: Abdul Patel mailto:abd786...@gmail.com>>
Sent: Thursday, February 6, 2020 1:54 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Re: Running select against cassandra

Its sort of user connected, app team needa number of active users connected say 
 every 1 to 5 mins.
The timeout at app end is 120ms.



On Thursday, February 6, 2020, Michael Shuler 
mailto:mich...@pbandjelly.org>> wrote:
You'll have to be more specific. What is your table schema and what is the 
SELECT query? What is the normal response time?

As a basic guide for your general question, if the query is something sort of 
irrelevant that should be stored some other way, like a total row count, or 
most any SELECT that requires ALLOW FILTERING, you're doing it wrong and should 
re-evaluate your data model.

1 query per minute is a minuscule fraction of the basic capacity of queries per 
minute that a Cassandra cluster should be able to handle with good data 
modeling and table-relevant query. All depends on the data model and query.

Michael

On 2/6/20 12:20 PM, Abdul Patel wrote:
Hi,

Is it advisable to run select query to fetch every minute to grab data from 
cassandra for reporting purpose, if no then whats the alternative?

-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org<mailto:user-unsubscr...@cassandra.apache.org>
For additional commands, e-mail: 
user-h...@cassandra.apache.org<mailto:user-h...@cassandra.apache.org>



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable

Re: [EXTERNAL] Re: Running select against cassandra

[Abdul Patel]

this is the schema similar to what we have , they want to get user
connected  - concurrent count for every say 1-5 minutes.
i am thinking will simple select will have performance issue or we can go
for materialized views ?

CREATE TABLE  usr_session (

userid bigint,

session_usr text,

last_access_time timestamp,

login_time timestamp,

status int,

PRIMARY KEY (userid, session_usr)

) WITH CLUSTERING ORDER BY (session_usr ASC)



On Thu, Feb 6, 2020 at 2:09 PM Durity, Sean R 
wrote:

> Do you only need the current count or do you want to keep the historical
> counts also? By active users, does that mean some kind of user that the
> application tracks (as opposed to the Cassandra user connected to the
> cluster)?
>
>
>
> I would consider a table like this for tracking active users through time:
>
>
>
> Create table users_by_day (
>
> app_date date,
>
> hour integer,
>
> minute integer,
>
> user_count integer,
>
> longest_login_user text,
>
> longest_login_seconds integer,
>
> last_login datetime,
>
> last_login_user text )
>
> primary key (app_date, hour, minute);
>
>
>
> Then, your reporting can easily select full days or a specific, one-minute
> slice. Of course, the app would need to have a timer and write out the
> data. I would also suggest a TTL on the data so that you only keep what you
> need (a week, a year, whatever). Of course, if your reporting requires
> different granularities, you could consider a different time bucket for the
> table (by hour, by week, etc.)
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Abdul Patel 
> *Sent:* Thursday, February 6, 2020 1:54 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: Running select against cassandra
>
>
>
> Its sort of user connected, app team needa number of active users
> connected say  every 1 to 5 mins.
>
> The timeout at app end is 120ms.
>
>
>
>
>
> On Thursday, February 6, 2020, Michael Shuler 
> wrote:
>
> You'll have to be more specific. What is your table schema and what is the
> SELECT query? What is the normal response time?
>
> As a basic guide for your general question, if the query is something sort
> of irrelevant that should be stored some other way, like a total row count,
> or most any SELECT that requires ALLOW FILTERING, you're doing it wrong and
> should re-evaluate your data model.
>
> 1 query per minute is a minuscule fraction of the basic capacity of
> queries per minute that a Cassandra cluster should be able to handle with
> good data modeling and table-relevant query. All depends on the data model
> and query.
>
> Michael
>
> On 2/6/20 12:20 PM, Abdul Patel wrote:
>
> Hi,
>
> Is it advisable to run select query to fetch every minute to grab data
> from cassandra for reporting purpose, if no then whats the alternative?
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] Re: Running select against cassandra

[Durity, Sean R]

Do you only need the current count or do you want to keep the historical counts 
also? By active users, does that mean some kind of user that the application 
tracks (as opposed to the Cassandra user connected to the cluster)?

I would consider a table like this for tracking active users through time:

Create table users_by_day (
app_date date,
hour integer,
minute integer,
user_count integer,
longest_login_user text,
longest_login_seconds integer,
last_login datetime,
last_login_user text )
primary key (app_date, hour, minute);

Then, your reporting can easily select full days or a specific, one-minute 
slice. Of course, the app would need to have a timer and write out the data. I 
would also suggest a TTL on the data so that you only keep what you need (a 
week, a year, whatever). Of course, if your reporting requires different 
granularities, you could consider a different time bucket for the table (by 
hour, by week, etc.)


Sean Durity – Staff Systems Engineer, Cassandra

From: Abdul Patel 
Sent: Thursday, February 6, 2020 1:54 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Running select against cassandra

Its sort of user connected, app team needa number of active users connected say 
 every 1 to 5 mins.
The timeout at app end is 120ms.



On Thursday, February 6, 2020, Michael Shuler 
mailto:mich...@pbandjelly.org>> wrote:
You'll have to be more specific. What is your table schema and what is the 
SELECT query? What is the normal response time?

As a basic guide for your general question, if the query is something sort of 
irrelevant that should be stored some other way, like a total row count, or 
most any SELECT that requires ALLOW FILTERING, you're doing it wrong and should 
re-evaluate your data model.

1 query per minute is a minuscule fraction of the basic capacity of queries per 
minute that a Cassandra cluster should be able to handle with good data 
modeling and table-relevant query. All depends on the data model and query.

Michael

On 2/6/20 12:20 PM, Abdul Patel wrote:
Hi,

Is it advisable to run select query to fetch every minute to grab data from 
cassandra for reporting purpose, if no then whats the alternative?


-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 
user-h...@cassandra.apache.org



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] How to reduce vnodes without downtime

[Sergio]

t;> Sergio
>>
>> Il giorno ven 31 gen 2020 alle ore 14:49 Erick Ramirez <
>> flightc...@gmail.com> ha scritto:
>>
>>> There's an active discussion going on right now in a separate dev
>>> thread. The current "default recommendation" is 32 tokens. But there's a
>>> push for 4 in combination with allocate_tokens_for_keyspace from Jon
>>> Haddad & co (based on a paper from Joe Lynch & Josh Snyder).
>>>
>>> If you're satisfied with the results from your own testing, go with 4
>>> tokens. And that's the key -- you must test, test, TEST! Cheers!
>>>
>>> On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon 
>>> wrote:
>>>
>>>> What is recommended vnodes now? I read 8 in later cassandra 3.x
>>>> Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
>>>> Thanks
>>>>
>>>> On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R <
>>>> sean_r_dur...@homedepot.com> wrote:
>>>>
>>>>> These are good clarifications and expansions.
>>>>>
>>>>>
>>>>>
>>>>> Sean Durity
>>>>>
>>>>>
>>>>>
>>>>> *From:* Anthony Grasso 
>>>>> *Sent:* Thursday, January 30, 2020 7:25 PM
>>>>> *To:* user 
>>>>> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime
>>>>>
>>>>>
>>>>>
>>>>> Hi Maxim,
>>>>>
>>>>>
>>>>>
>>>>> Basically what Sean suggested is the way to do this without downtime.
>>>>>
>>>>>
>>>>>
>>>>> To clarify the, the *three* steps following the "Decommission each
>>>>> node in the DC you are working on" step should be applied to *only*
>>>>> the decommissioned nodes. So where it say "*all nodes*" or "*every
>>>>> node*" it applies to only the decommissioned nodes.
>>>>>
>>>>>
>>>>>
>>>>> In addition, the step that says "Wipe data on all the nodes", I would
>>>>> delete all files in the following directories on the decommissioned nodes.
>>>>>
>>>>>- data (usually located in /var/lib/cassandra/data)
>>>>>- commitlogs (usually located in /var/lib/cassandra/commitlogs)
>>>>>- hints (usually located in /var/lib/casandra/hints)
>>>>>- saved_caches (usually located in /var/lib/cassandra/saved_caches)
>>>>>
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Anthony
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <
>>>>> sean_r_dur...@homedepot.com> wrote:
>>>>>
>>>>> Your procedure won’t work very well. On the first node, if you
>>>>> switched to 4, you would end up with only a tiny fraction of the data
>>>>> (because the other nodes would still be at 256). I updated a large cluster
>>>>> (over 150 nodes – 2 DCs) to smaller number of vnodes. The basic outline 
>>>>> was
>>>>> this:
>>>>>
>>>>>
>>>>>
>>>>>- Stop all repairs
>>>>>- Make sure the app is running against one DC only
>>>>>- Change the replication settings on keyspaces to use only 1 DC
>>>>>(basically cutting off the other DC)
>>>>>- Decommission each node in the DC you are working on. Because the
>>>>>replication setting are changed, no streaming occurs. But it releases 
>>>>> the
>>>>>token assignments
>>>>>- Wipe data on all the nodes
>>>>>- Update configuration on every node to your new settings,
>>>>>including auto_bootstrap = false
>>>>>- Start all nodes. They will choose tokens, but not stream any data
>>>>>- Update replication factor for all keyspaces to include the new DC
>>>>>- I disabled binary on those nodes to prevent app connections
>>>>>- Run nodetool reduild with -dc (other DC) on as many nodes as
>>>>>your system can safely handle until they are all rebuilt.
>>>>>- Re-enable binary (and app connections to the rebuilt DC)
>>>>>- Turn on repairs
>>>>>

Re: [EXTERNAL] How to reduce vnodes without downtime

[Sergio]

Thanks Erick!

Best,

Sergio

On Sun, Feb 2, 2020, 10:07 PM Erick Ramirez  wrote:

> If you are after more details about the trade-offs between different sized
>> token values, please see the discussion on the dev mailing list: "[Discuss]
>> num_tokens default in Cassandra 4.0
>> 
>> ".
>>
>> Regards,
>> Anthony
>>
>> On Sat, 1 Feb 2020 at 10:07, Sergio  wrote:
>>
>>>
>>> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>>>  This
>>> is the article with 4 token recommendations.
>>> @Erick Ramirez. which is the dev thread for the default 32 tokens
>>> recommendation?
>>>
>>> Thanks,
>>> Sergio
>>>
>>
> Sergio, my apologies for not replying. For some reason, your reply went to
> my spam folder and I didn't see it.
>
> Thanks, Anthony, for responding. I was indeed referring to that dev
> thread. Cheers!
>
>

Re: [EXTERNAL] How to reduce vnodes without downtime

[Maxim Parkachov]

Hi guys,

thanks a lot for useful tips. I obviously underestimated complexity of such
change.

Thanks again,
Maxim.

>

Re: [EXTERNAL] How to reduce vnodes without downtime

[Erick Ramirez]

>
> If you are after more details about the trade-offs between different sized
> token values, please see the discussion on the dev mailing list: "[Discuss]
> num_tokens default in Cassandra 4.0
> 
> ".
>
> Regards,
> Anthony
>
> On Sat, 1 Feb 2020 at 10:07, Sergio  wrote:
>
>>
>> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>>  This
>> is the article with 4 token recommendations.
>> @Erick Ramirez. which is the dev thread for the default 32 tokens
>> recommendation?
>>
>> Thanks,
>> Sergio
>>
>
Sergio, my apologies for not replying. For some reason, your reply went to
my spam folder and I didn't see it.

Thanks, Anthony, for responding. I was indeed referring to that dev thread.
Cheers!

Re: [EXTERNAL] How to reduce vnodes without downtime

[Sergio]

Thanks Anthony!

I will read more about it

Best,

Sergio



Il giorno dom 2 feb 2020 alle ore 18:36 Anthony Grasso <
anthony.gra...@gmail.com> ha scritto:

> Hi Sergio,
>
> There is a misunderstanding here. My post makes no recommendation for the
> value of num_tokens. Rather, it focuses on how to use
> the allocate_tokens_for_keyspace setting when creating a new cluster.
>
> Whilst a value of 4 is used for num_tokens in the post, it was chosen for
> demonstration purposes. Specifically it makes:
>
>- the uneven token distribution in a small cluster very obvious,
>- identifying the endpoints displayed in nodetool ring easy, and
>- the initial_token setup less verbose and easier to follow.
>
> I will add an editorial note to the post with the above information
> so there is no confusion about why 4 tokens were used.
>
> I would only consider moving a cluster to 4 tokens if it is larger than
> 100 nodes. If you read through the paper that Erick mentioned, written
> by Joe Lynch & Josh Snyder, they show that the num_tokens impacts the
> availability of large scale clusters.
>
> If you are after more details about the trade-offs between different sized
> token values, please see the discussion on the dev mailing list: "[Discuss]
> num_tokens default in Cassandra 4.0
> <https://www.mail-archive.com/search?l=dev%40cassandra.apache.org=subject%3A%22%5C%5BDiscuss%5C%5D+num_tokens+default+in+Cassandra+4.0%22=oldest>
> ".
>
> Regards,
> Anthony
>
> On Sat, 1 Feb 2020 at 10:07, Sergio  wrote:
>
>>
>> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>>  This
>> is the article with 4 token recommendations.
>> @Erick Ramirez. which is the dev thread for the default 32 tokens
>> recommendation?
>>
>> Thanks,
>> Sergio
>>
>> Il giorno ven 31 gen 2020 alle ore 14:49 Erick Ramirez <
>> flightc...@gmail.com> ha scritto:
>>
>>> There's an active discussion going on right now in a separate dev
>>> thread. The current "default recommendation" is 32 tokens. But there's a
>>> push for 4 in combination with allocate_tokens_for_keyspace from Jon
>>> Haddad & co (based on a paper from Joe Lynch & Josh Snyder).
>>>
>>> If you're satisfied with the results from your own testing, go with 4
>>> tokens. And that's the key -- you must test, test, TEST! Cheers!
>>>
>>> On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon 
>>> wrote:
>>>
>>>> What is recommended vnodes now? I read 8 in later cassandra 3.x
>>>> Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
>>>> Thanks
>>>>
>>>> On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R <
>>>> sean_r_dur...@homedepot.com> wrote:
>>>>
>>>>> These are good clarifications and expansions.
>>>>>
>>>>>
>>>>>
>>>>> Sean Durity
>>>>>
>>>>>
>>>>>
>>>>> *From:* Anthony Grasso 
>>>>> *Sent:* Thursday, January 30, 2020 7:25 PM
>>>>> *To:* user 
>>>>> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime
>>>>>
>>>>>
>>>>>
>>>>> Hi Maxim,
>>>>>
>>>>>
>>>>>
>>>>> Basically what Sean suggested is the way to do this without downtime.
>>>>>
>>>>>
>>>>>
>>>>> To clarify the, the *three* steps following the "Decommission each
>>>>> node in the DC you are working on" step should be applied to *only*
>>>>> the decommissioned nodes. So where it say "*all nodes*" or "*every
>>>>> node*" it applies to only the decommissioned nodes.
>>>>>
>>>>>
>>>>>
>>>>> In addition, the step that says "Wipe data on all the nodes", I would
>>>>> delete all files in the following directories on the decommissioned nodes.
>>>>>
>>>>>- data (usually located in /var/lib/cassandra/data)
>>>>>- commitlogs (usually located in /var/lib/cassandra/commitlogs)
>>>>>- hints (usually located in /var/lib/casandra/hints)
>>>>>- saved_caches (usually located in /var/lib/cassandra/saved_caches)
>>>>>
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Anthony
>>>>>
>>>&g

Re: [EXTERNAL] How to reduce vnodes without downtime

[Anthony Grasso]

Hi Sergio,

There is a misunderstanding here. My post makes no recommendation for the
value of num_tokens. Rather, it focuses on how to use
the allocate_tokens_for_keyspace setting when creating a new cluster.

Whilst a value of 4 is used for num_tokens in the post, it was chosen for
demonstration purposes. Specifically it makes:

   - the uneven token distribution in a small cluster very obvious,
   - identifying the endpoints displayed in nodetool ring easy, and
   - the initial_token setup less verbose and easier to follow.

I will add an editorial note to the post with the above information
so there is no confusion about why 4 tokens were used.

I would only consider moving a cluster to 4 tokens if it is larger than 100
nodes. If you read through the paper that Erick mentioned, written by Joe
Lynch & Josh Snyder, they show that the num_tokens impacts the availability
of large scale clusters.

If you are after more details about the trade-offs between different sized
token values, please see the discussion on the dev mailing list: "[Discuss]
num_tokens default in Cassandra 4.0
<https://www.mail-archive.com/search?l=dev%40cassandra.apache.org=subject%3A%22%5C%5BDiscuss%5C%5D+num_tokens+default+in+Cassandra+4.0%22=oldest>
".

Regards,
Anthony

On Sat, 1 Feb 2020 at 10:07, Sergio  wrote:

>
> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>  This
> is the article with 4 token recommendations.
> @Erick Ramirez. which is the dev thread for the default 32 tokens
> recommendation?
>
> Thanks,
> Sergio
>
> Il giorno ven 31 gen 2020 alle ore 14:49 Erick Ramirez <
> flightc...@gmail.com> ha scritto:
>
>> There's an active discussion going on right now in a separate dev thread.
>> The current "default recommendation" is 32 tokens. But there's a push for 4
>> in combination with allocate_tokens_for_keyspace from Jon Haddad & co
>> (based on a paper from Joe Lynch & Josh Snyder).
>>
>> If you're satisfied with the results from your own testing, go with 4
>> tokens. And that's the key -- you must test, test, TEST! Cheers!
>>
>> On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon 
>> wrote:
>>
>>> What is recommended vnodes now? I read 8 in later cassandra 3.x
>>> Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
>>> Thanks
>>>
>>> On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R <
>>> sean_r_dur...@homedepot.com> wrote:
>>>
>>>> These are good clarifications and expansions.
>>>>
>>>>
>>>>
>>>> Sean Durity
>>>>
>>>>
>>>>
>>>> *From:* Anthony Grasso 
>>>> *Sent:* Thursday, January 30, 2020 7:25 PM
>>>> *To:* user 
>>>> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime
>>>>
>>>>
>>>>
>>>> Hi Maxim,
>>>>
>>>>
>>>>
>>>> Basically what Sean suggested is the way to do this without downtime.
>>>>
>>>>
>>>>
>>>> To clarify the, the *three* steps following the "Decommission each
>>>> node in the DC you are working on" step should be applied to *only*
>>>> the decommissioned nodes. So where it say "*all nodes*" or "*every
>>>> node*" it applies to only the decommissioned nodes.
>>>>
>>>>
>>>>
>>>> In addition, the step that says "Wipe data on all the nodes", I would
>>>> delete all files in the following directories on the decommissioned nodes.
>>>>
>>>>- data (usually located in /var/lib/cassandra/data)
>>>>- commitlogs (usually located in /var/lib/cassandra/commitlogs)
>>>>- hints (usually located in /var/lib/casandra/hints)
>>>>- saved_caches (usually located in /var/lib/cassandra/saved_caches)
>>>>
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Anthony
>>>>
>>>>
>>>>
>>>> On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <
>>>> sean_r_dur...@homedepot.com> wrote:
>>>>
>>>> Your procedure won’t work very well. On the first node, if you switched
>>>> to 4, you would end up with only a tiny fraction of the data (because the
>>>> other nodes would still be at 256). I updated a large cluster (over 150
>>>> nodes – 2 DCs) to smaller number of vnodes. The basic outline was this:
>>>>
>>>>
>>>>
>>>>- Stop all repairs
&g

Re: [EXTERNAL] How to reduce vnodes without downtime

[Sergio]

https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
This
is the article with 4 token recommendations.
@Erick Ramirez. which is the dev thread for the default 32 tokens
recommendation?

Thanks,
Sergio

Il giorno ven 31 gen 2020 alle ore 14:49 Erick Ramirez 
ha scritto:

> There's an active discussion going on right now in a separate dev thread.
> The current "default recommendation" is 32 tokens. But there's a push for 4
> in combination with allocate_tokens_for_keyspace from Jon Haddad & co
> (based on a paper from Joe Lynch & Josh Snyder).
>
> If you're satisfied with the results from your own testing, go with 4
> tokens. And that's the key -- you must test, test, TEST! Cheers!
>
> On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon 
> wrote:
>
>> What is recommended vnodes now? I read 8 in later cassandra 3.x
>> Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
>> Thanks
>>
>> On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R <
>> sean_r_dur...@homedepot.com> wrote:
>>
>>> These are good clarifications and expansions.
>>>
>>>
>>>
>>> Sean Durity
>>>
>>>
>>>
>>> *From:* Anthony Grasso 
>>> *Sent:* Thursday, January 30, 2020 7:25 PM
>>> *To:* user 
>>> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime
>>>
>>>
>>>
>>> Hi Maxim,
>>>
>>>
>>>
>>> Basically what Sean suggested is the way to do this without downtime.
>>>
>>>
>>>
>>> To clarify the, the *three* steps following the "Decommission each node
>>> in the DC you are working on" step should be applied to *only* the
>>> decommissioned nodes. So where it say "*all nodes*" or "*every node*"
>>> it applies to only the decommissioned nodes.
>>>
>>>
>>>
>>> In addition, the step that says "Wipe data on all the nodes", I would
>>> delete all files in the following directories on the decommissioned nodes.
>>>
>>>- data (usually located in /var/lib/cassandra/data)
>>>- commitlogs (usually located in /var/lib/cassandra/commitlogs)
>>>- hints (usually located in /var/lib/casandra/hints)
>>>- saved_caches (usually located in /var/lib/cassandra/saved_caches)
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Anthony
>>>
>>>
>>>
>>> On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <
>>> sean_r_dur...@homedepot.com> wrote:
>>>
>>> Your procedure won’t work very well. On the first node, if you switched
>>> to 4, you would end up with only a tiny fraction of the data (because the
>>> other nodes would still be at 256). I updated a large cluster (over 150
>>> nodes – 2 DCs) to smaller number of vnodes. The basic outline was this:
>>>
>>>
>>>
>>>- Stop all repairs
>>>- Make sure the app is running against one DC only
>>>- Change the replication settings on keyspaces to use only 1 DC
>>>(basically cutting off the other DC)
>>>- Decommission each node in the DC you are working on. Because the
>>>replication setting are changed, no streaming occurs. But it releases the
>>>token assignments
>>>- Wipe data on all the nodes
>>>- Update configuration on every node to your new settings, including
>>>auto_bootstrap = false
>>>- Start all nodes. They will choose tokens, but not stream any data
>>>- Update replication factor for all keyspaces to include the new DC
>>>- I disabled binary on those nodes to prevent app connections
>>>- Run nodetool reduild with -dc (other DC) on as many nodes as your
>>>system can safely handle until they are all rebuilt.
>>>- Re-enable binary (and app connections to the rebuilt DC)
>>>- Turn on repairs
>>>- Rest for a bit, then reverse the process for the remaining DCs
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Sean Durity – Staff Systems Engineer, Cassandra
>>>
>>>
>>>
>>> *From:* Maxim Parkachov 
>>> *Sent:* Thursday, January 30, 2020 10:05 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* [EXTERNAL] How to reduce vnodes without downtime
>>>
>>>
>>>
>>> Hi everyone,
>>>
>>>
>>>
>>> with discussion about reduc

Re: [EXTERNAL] How to reduce vnodes without downtime

[Erick Ramirez]

There's an active discussion going on right now in a separate dev thread.
The current "default recommendation" is 32 tokens. But there's a push for 4
in combination with allocate_tokens_for_keyspace from Jon Haddad & co
(based on a paper from Joe Lynch & Josh Snyder).

If you're satisfied with the results from your own testing, go with 4
tokens. And that's the key -- you must test, test, TEST! Cheers!

On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon 
wrote:

> What is recommended vnodes now? I read 8 in later cassandra 3.x
> Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
> Thanks
>
> On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
>> These are good clarifications and expansions.
>>
>>
>>
>> Sean Durity
>>
>>
>>
>> *From:* Anthony Grasso 
>> *Sent:* Thursday, January 30, 2020 7:25 PM
>> *To:* user 
>> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime
>>
>>
>>
>> Hi Maxim,
>>
>>
>>
>> Basically what Sean suggested is the way to do this without downtime.
>>
>>
>>
>> To clarify the, the *three* steps following the "Decommission each node
>> in the DC you are working on" step should be applied to *only* the
>> decommissioned nodes. So where it say "*all nodes*" or "*every node*" it
>> applies to only the decommissioned nodes.
>>
>>
>>
>> In addition, the step that says "Wipe data on all the nodes", I would
>> delete all files in the following directories on the decommissioned nodes.
>>
>>- data (usually located in /var/lib/cassandra/data)
>>- commitlogs (usually located in /var/lib/cassandra/commitlogs)
>>- hints (usually located in /var/lib/casandra/hints)
>>- saved_caches (usually located in /var/lib/cassandra/saved_caches)
>>
>>
>>
>> Cheers,
>>
>> Anthony
>>
>>
>>
>> On Fri, 31 Jan 2020 at 03:05, Durity, Sean R 
>> wrote:
>>
>> Your procedure won’t work very well. On the first node, if you switched
>> to 4, you would end up with only a tiny fraction of the data (because the
>> other nodes would still be at 256). I updated a large cluster (over 150
>> nodes – 2 DCs) to smaller number of vnodes. The basic outline was this:
>>
>>
>>
>>- Stop all repairs
>>- Make sure the app is running against one DC only
>>- Change the replication settings on keyspaces to use only 1 DC
>>(basically cutting off the other DC)
>>- Decommission each node in the DC you are working on. Because the
>>replication setting are changed, no streaming occurs. But it releases the
>>token assignments
>>- Wipe data on all the nodes
>>- Update configuration on every node to your new settings, including
>>auto_bootstrap = false
>>- Start all nodes. They will choose tokens, but not stream any data
>>- Update replication factor for all keyspaces to include the new DC
>>- I disabled binary on those nodes to prevent app connections
>>- Run nodetool reduild with -dc (other DC) on as many nodes as your
>>system can safely handle until they are all rebuilt.
>>- Re-enable binary (and app connections to the rebuilt DC)
>>- Turn on repairs
>>- Rest for a bit, then reverse the process for the remaining DCs
>>
>>
>>
>>
>>
>>
>>
>> Sean Durity – Staff Systems Engineer, Cassandra
>>
>>
>>
>> *From:* Maxim Parkachov 
>> *Sent:* Thursday, January 30, 2020 10:05 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] How to reduce vnodes without downtime
>>
>>
>>
>> Hi everyone,
>>
>>
>>
>> with discussion about reducing default vnodes in version 4.0 I would like
>> to ask, what would be optimal procedure to perform reduction of vnodes in
>> existing 3.11.x cluster which was set up with default value 256. Cluster
>> has 2 DC with 5 nodes each and RF=3. There is one more restriction, I could
>> not add more servers, nor to create additional DC, everything is physical.
>> This should be done without downtime.
>>
>>
>>
>> My idea for such procedure would be
>>
>>
>>
>> for each node:
>>
>> - decommission node
>>
>> - set auto_bootstrap to true and vnodes to 4
>>
>> - start and wait till node joins cluster
>>
>> - run cleanup on rest of nodes in cluster
>>
>> - run repair on whole cluster (not sure 

Re: [EXTERNAL] How to reduce vnodes without downtime

[Arvinder Dhillon]

What is recommended vnodes now? I read 8 in later cassandra 3.x
Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
Thanks

On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R 
wrote:

> These are good clarifications and expansions.
>
>
>
> Sean Durity
>
>
>
> *From:* Anthony Grasso 
> *Sent:* Thursday, January 30, 2020 7:25 PM
> *To:* user 
> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime
>
>
>
> Hi Maxim,
>
>
>
> Basically what Sean suggested is the way to do this without downtime.
>
>
>
> To clarify the, the *three* steps following the "Decommission each node
> in the DC you are working on" step should be applied to *only* the
> decommissioned nodes. So where it say "*all nodes*" or "*every node*" it
> applies to only the decommissioned nodes.
>
>
>
> In addition, the step that says "Wipe data on all the nodes", I would
> delete all files in the following directories on the decommissioned nodes.
>
>- data (usually located in /var/lib/cassandra/data)
>- commitlogs (usually located in /var/lib/cassandra/commitlogs)
>- hints (usually located in /var/lib/casandra/hints)
>- saved_caches (usually located in /var/lib/cassandra/saved_caches)
>
>
>
> Cheers,
>
> Anthony
>
>
>
> On Fri, 31 Jan 2020 at 03:05, Durity, Sean R 
> wrote:
>
> Your procedure won’t work very well. On the first node, if you switched to
> 4, you would end up with only a tiny fraction of the data (because the
> other nodes would still be at 256). I updated a large cluster (over 150
> nodes – 2 DCs) to smaller number of vnodes. The basic outline was this:
>
>
>
>- Stop all repairs
>- Make sure the app is running against one DC only
>- Change the replication settings on keyspaces to use only 1 DC
>(basically cutting off the other DC)
>- Decommission each node in the DC you are working on. Because the
>replication setting are changed, no streaming occurs. But it releases the
>token assignments
>- Wipe data on all the nodes
>- Update configuration on every node to your new settings, including
>auto_bootstrap = false
>- Start all nodes. They will choose tokens, but not stream any data
>- Update replication factor for all keyspaces to include the new DC
>- I disabled binary on those nodes to prevent app connections
>- Run nodetool reduild with -dc (other DC) on as many nodes as your
>system can safely handle until they are all rebuilt.
>- Re-enable binary (and app connections to the rebuilt DC)
>- Turn on repairs
>- Rest for a bit, then reverse the process for the remaining DCs
>
>
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Maxim Parkachov 
> *Sent:* Thursday, January 30, 2020 10:05 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] How to reduce vnodes without downtime
>
>
>
> Hi everyone,
>
>
>
> with discussion about reducing default vnodes in version 4.0 I would like
> to ask, what would be optimal procedure to perform reduction of vnodes in
> existing 3.11.x cluster which was set up with default value 256. Cluster
> has 2 DC with 5 nodes each and RF=3. There is one more restriction, I could
> not add more servers, nor to create additional DC, everything is physical.
> This should be done without downtime.
>
>
>
> My idea for such procedure would be
>
>
>
> for each node:
>
> - decommission node
>
> - set auto_bootstrap to true and vnodes to 4
>
> - start and wait till node joins cluster
>
> - run cleanup on rest of nodes in cluster
>
> - run repair on whole cluster (not sure if needed after cleanup)
>
> - set auto_bootstrap to false
>
> repeat for each node
>
>
>
> rolling restart of cluster
>
> cluster repair
>
>
>
> Is this sounds right ? My concern is that after decommission, node will
> start on the same IP which could create some confusion.
>
>
>
> Regards,
>
> Maxim.
>
>
> --
>
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
>

RE: [EXTERNAL] How to reduce vnodes without downtime

[Durity, Sean R]

These are good clarifications and expansions.

Sean Durity

From: Anthony Grasso 
Sent: Thursday, January 30, 2020 7:25 PM
To: user 
Subject: Re: [EXTERNAL] How to reduce vnodes without downtime

Hi Maxim,

Basically what Sean suggested is the way to do this without downtime.

To clarify the, the three steps following the "Decommission each node in the DC 
you are working on" step should be applied to only the decommissioned nodes. So 
where it say "all nodes" or "every node" it applies to only the decommissioned 
nodes.

In addition, the step that says "Wipe data on all the nodes", I would delete 
all files in the following directories on the decommissioned nodes.

  *   data (usually located in /var/lib/cassandra/data)
  *   commitlogs (usually located in /var/lib/cassandra/commitlogs)
  *   hints (usually located in /var/lib/casandra/hints)
  *   saved_caches (usually located in /var/lib/cassandra/saved_caches)

Cheers,
Anthony

On Fri, 31 Jan 2020 at 03:05, Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
Your procedure won’t work very well. On the first node, if you switched to 4, 
you would end up with only a tiny fraction of the data (because the other nodes 
would still be at 256). I updated a large cluster (over 150 nodes – 2 DCs) to 
smaller number of vnodes. The basic outline was this:


  *   Stop all repairs
  *   Make sure the app is running against one DC only
  *   Change the replication settings on keyspaces to use only 1 DC (basically 
cutting off the other DC)
  *   Decommission each node in the DC you are working on. Because the 
replication setting are changed, no streaming occurs. But it releases the token 
assignments
  *   Wipe data on all the nodes
  *   Update configuration on every node to your new settings, including 
auto_bootstrap = false
  *   Start all nodes. They will choose tokens, but not stream any data
  *   Update replication factor for all keyspaces to include the new DC
  *   I disabled binary on those nodes to prevent app connections
  *   Run nodetool reduild with -dc (other DC) on as many nodes as your system 
can safely handle until they are all rebuilt.
  *   Re-enable binary (and app connections to the rebuilt DC)
  *   Turn on repairs
  *   Rest for a bit, then reverse the process for the remaining DCs



Sean Durity – Staff Systems Engineer, Cassandra

From: Maxim Parkachov mailto:lazy.gop...@gmail.com>>
Sent: Thursday, January 30, 2020 10:05 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] How to reduce vnodes without downtime

Hi everyone,

with discussion about reducing default vnodes in version 4.0 I would like to 
ask, what would be optimal procedure to perform reduction of vnodes in existing 
3.11.x cluster which was set up with default value 256. Cluster has 2 DC with 5 
nodes each and RF=3. There is one more restriction, I could not add more 
servers, nor to create additional DC, everything is physical. This should be 
done without downtime.

My idea for such procedure would be

for each node:
- decommission node
- set auto_bootstrap to true and vnodes to 4
- start and wait till node joins cluster
- run cleanup on rest of nodes in cluster
- run repair on whole cluster (not sure if needed after cleanup)
- set auto_bootstrap to false
repeat for each node

rolling restart of cluster
cluster repair

Is this sounds right ? My concern is that after decommission, node will start 
on the same IP which could create some confusion.

Regards,
Maxim.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and ma

Re: [EXTERNAL] How to reduce vnodes without downtime

[Anthony Grasso]

Hi Maxim,

Basically what Sean suggested is the way to do this without downtime.

To clarify the, the *three* steps following the "Decommission each node in
the DC you are working on" step should be applied to *only* the
decommissioned nodes. So where it say "*all nodes*" or "*every node*" it
applies to only the decommissioned nodes.

In addition, the step that says "Wipe data on all the nodes", I would
delete all files in the following directories on the decommissioned nodes.

   - data (usually located in /var/lib/cassandra/data)
   - commitlogs (usually located in /var/lib/cassandra/commitlogs)
   - hints (usually located in /var/lib/casandra/hints)
   - saved_caches (usually located in /var/lib/cassandra/saved_caches)


Cheers,
Anthony

On Fri, 31 Jan 2020 at 03:05, Durity, Sean R 
wrote:

> Your procedure won’t work very well. On the first node, if you switched to
> 4, you would end up with only a tiny fraction of the data (because the
> other nodes would still be at 256). I updated a large cluster (over 150
> nodes – 2 DCs) to smaller number of vnodes. The basic outline was this:
>
>
>
>- Stop all repairs
>- Make sure the app is running against one DC only
>- Change the replication settings on keyspaces to use only 1 DC
>(basically cutting off the other DC)
>- Decommission each node in the DC you are working on. Because the
>replication setting are changed, no streaming occurs. But it releases the
>token assignments
>- Wipe data on all the nodes
>- Update configuration on every node to your new settings, including
>auto_bootstrap = false
>- Start all nodes. They will choose tokens, but not stream any data
>- Update replication factor for all keyspaces to include the new DC
>- I disabled binary on those nodes to prevent app connections
>- Run nodetool reduild with -dc (other DC) on as many nodes as your
>system can safely handle until they are all rebuilt.
>- Re-enable binary (and app connections to the rebuilt DC)
>- Turn on repairs
>- Rest for a bit, then reverse the process for the remaining DCs
>
>
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Maxim Parkachov 
> *Sent:* Thursday, January 30, 2020 10:05 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] How to reduce vnodes without downtime
>
>
>
> Hi everyone,
>
>
>
> with discussion about reducing default vnodes in version 4.0 I would like
> to ask, what would be optimal procedure to perform reduction of vnodes in
> existing 3.11.x cluster which was set up with default value 256. Cluster
> has 2 DC with 5 nodes each and RF=3. There is one more restriction, I could
> not add more servers, nor to create additional DC, everything is physical.
> This should be done without downtime.
>
>
>
> My idea for such procedure would be
>
>
>
> for each node:
>
> - decommission node
>
> - set auto_bootstrap to true and vnodes to 4
>
> - start and wait till node joins cluster
>
> - run cleanup on rest of nodes in cluster
>
> - run repair on whole cluster (not sure if needed after cleanup)
>
> - set auto_bootstrap to false
>
> repeat for each node
>
>
>
> rolling restart of cluster
>
> cluster repair
>
>
>
> Is this sounds right ? My concern is that after decommission, node will
> start on the same IP which could create some confusion.
>
>
>
> Regards,
>
> Maxim.
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] How to reduce vnodes without downtime

[Durity, Sean R]

Your procedure won’t work very well. On the first node, if you switched to 4, 
you would end up with only a tiny fraction of the data (because the other nodes 
would still be at 256). I updated a large cluster (over 150 nodes – 2 DCs) to 
smaller number of vnodes. The basic outline was this:


  *   Stop all repairs
  *   Make sure the app is running against one DC only
  *   Change the replication settings on keyspaces to use only 1 DC (basically 
cutting off the other DC)
  *   Decommission each node in the DC you are working on. Because the 
replication setting are changed, no streaming occurs. But it releases the token 
assignments
  *   Wipe data on all the nodes
  *   Update configuration on every node to your new settings, including 
auto_bootstrap = false
  *   Start all nodes. They will choose tokens, but not stream any data
  *   Update replication factor for all keyspaces to include the new DC
  *   I disabled binary on those nodes to prevent app connections
  *   Run nodetool reduild with -dc (other DC) on as many nodes as your system 
can safely handle until they are all rebuilt.
  *   Re-enable binary (and app connections to the rebuilt DC)
  *   Turn on repairs
  *   Rest for a bit, then reverse the process for the remaining DCs



Sean Durity – Staff Systems Engineer, Cassandra

From: Maxim Parkachov 
Sent: Thursday, January 30, 2020 10:05 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] How to reduce vnodes without downtime

Hi everyone,

with discussion about reducing default vnodes in version 4.0 I would like to 
ask, what would be optimal procedure to perform reduction of vnodes in existing 
3.11.x cluster which was set up with default value 256. Cluster has 2 DC with 5 
nodes each and RF=3. There is one more restriction, I could not add more 
servers, nor to create additional DC, everything is physical. This should be 
done without downtime.

My idea for such procedure would be

for each node:
- decommission node
- set auto_bootstrap to true and vnodes to 4
- start and wait till node joins cluster
- run cleanup on rest of nodes in cluster
- run repair on whole cluster (not sure if needed after cleanup)
- set auto_bootstrap to false
repeat for each node

rolling restart of cluster
cluster repair

Is this sounds right ? My concern is that after decommission, node will start 
on the same IP which could create some confusion.

Regards,
Maxim.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Re: sstableloader & num_tokens change

[Voytek Jarnot]

Odd. Have you seen this behavior? I ran a test last week, loaded snapshots
from 4 nodes to 4 nodes (RF 3 on both ends) and did not notice a spike.
That's not to say that it didn't happen, but I think I'd have noticed as I
was loading approx 250GB x 4 (although sequentially rather than 4x
sstableloader in parallel).

Also, thanks to everyone for confirming no issue with num_tokens and
sstableloader; appreciate it.


On Mon, Jan 27, 2020 at 9:02 AM Durity, Sean R 
wrote:

> I would suggest to be aware of potential data size expansion. If you load
> (for example) three copies of the data into a new cluster (because the RF
> of the origin cluster is 3), it will also get written to the RF of the new
> cluster (3 more times). So, you could see data expansion of 9x the original
> data size (or, origin RF * target RF), until compaction can run.
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Erick Ramirez 
> *Sent:* Friday, January 24, 2020 11:03 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: sstableloader & num_tokens change
>
>
>
>
>
> If I may just loop this back to the question at hand:
>
> I'm curious if there are any gotchas with using sstableloader to restore
> snapshots taken from 256-token nodes into a cluster with 32-token (or your
> preferred number of tokens) nodes (otherwise same # of nodes and same RF).
>
>
>
> No, there isn't. It will work as designed so you're good to go. Cheers!
>
>
>
>
>
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] Re: sstableloader & num_tokens change

[Durity, Sean R]

I would suggest to be aware of potential data size expansion. If you load (for 
example) three copies of the data into a new cluster (because the RF of the 
origin cluster is 3), it will also get written to the RF of the new cluster (3 
more times). So, you could see data expansion of 9x the original data size (or, 
origin RF * target RF), until compaction can run.


Sean Durity – Staff Systems Engineer, Cassandra

From: Erick Ramirez 
Sent: Friday, January 24, 2020 11:03 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: sstableloader & num_tokens change


If I may just loop this back to the question at hand:

I'm curious if there are any gotchas with using sstableloader to restore 
snapshots taken from 256-token nodes into a cluster with 32-token (or your 
preferred number of tokens) nodes (otherwise same # of nodes and same RF).

No, there isn't. It will work as designed so you're good to go. Cheers!





The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Re: COPY command with where condition

[Jean Carlo]

Hello

Nobody has mentioned but you can use spark cassandra connector also.
Preferably if your data set is so big that a simple copy to csv cannot
handle it

Saludos

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay


On Fri, Jan 17, 2020 at 8:11 PM Durity, Sean R 
wrote:

> sstablekeys (in the tools directory?) can extract the actual keys from
> your sstables. You have to run it on each node and then combine and de-dupe
> the final results, but I have used this technique with a query generator to
> extract data more efficiently.
>
>
>
>
>
> Sean Durity
>
>
>
> *From:* Chris Splinter 
> *Sent:* Friday, January 17, 2020 1:47 PM
> *To:* adrien ruffie 
> *Cc:* user@cassandra.apache.org; Erick Ramirez 
> *Subject:* [EXTERNAL] Re: COPY command with where condition
>
>
>
> Do you know your partition keys?
>
>
>
> One option could be to enumerate that list of partition keys in separate
> cmds to make the individual operations less expensive for the cluster.
>
>
>
> For example:
>
> Say your partition key column is called id and the ids in your database
> are [1,2,3]
>
>
>
> You could do
>
> ./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT *
> FROM probe_sensors WHERE id = 1 AND localisation_id = 208812" -url
> /home/dump
>
> ./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT *
> FROM probe_sensors WHERE id = 2 AND localisation_id = 208812" -url
> /home/dump
>
> ./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT *
> FROM probe_sensors WHERE id = 3 AND localisation_id = 208812" -url
> /home/dump
>
>
>
>
>
> Does that option work for you?
>
>
>
>
>
>
>
> On Fri, Jan 17, 2020 at 12:17 PM adrien ruffie 
> wrote:
>
> I don't really know for the moment in production environment, but for
> developpment environment the table contains more than 10.000.000 rows.
>
> But we need just a sub dataset of this table not the entirety ...
> --
>
> *De :* Chris Splinter 
> *Envoyé :* vendredi 17 janvier 2020 17:40
> *À :* adrien ruffie 
> *Cc :* user@cassandra.apache.org ; Erick
> Ramirez 
> *Objet :* Re: COPY command with where condition
>
>
>
> What you are seeing there is a standard read timeout, how many rows do you
> expect back from that query?
>
>
>
> On Fri, Jan 17, 2020 at 9:50 AM adrien ruffie 
> wrote:
>
> Thank you very much,
>
>
>
>  so I do this request with for example -->
>
>
>
> ./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT *
> FROM probe_sensors WHERE localisation_id = 208812 ALLOW FILTERING" -url
> /home/dump
>
>
>
>
>
> But I get the following error
>
> com.datastax.dsbulk.executor.api.exception.BulkExecutionException:
> Statement execution failed: SELECT * FROM crt_sensors WHERE site_id =
> 208812 ALLOW FILTERING (Cassandra timeout during read query at consistency
> LOCAL_ONE (1 responses were required but only 0 replica responded))
>
>
>
> but I configured my driver with following driver.conf, but nothing work
> correctly. Do you know what is the problem ?
>
>
>
> datastax-java-driver {
>
> basic {
>
>
>
>
>
> contact-points = ["data1com:9042","data2.com:9042 [data2.com]
> 
> "]
>
>
>
> request {
>
> timeout = "200"
>
> consistency = "LOCAL_ONE"
>
>
>
> }
>
> }
>
> advanced {
>
>
>
> auth-provider {
>
> class = PlainTextAuthProvider
>
> username = "superuser"
>
> password = "mypass"
>
>
>
> }
>
> }
>
> }
> --
>
> *De :* Chris Splinter 
> *Envoyé :* vendredi 17 janvier 2020 16:17
> *À :* user@cassandra.apache.org 
> *Cc :* Erick Ramirez 
> *Objet :* Re: COPY command with where condition
>
>
>
> DSBulk has an option that lets you specify the query ( including a WHERE
> clause )
>
>
>
> See Example 19 in this blog post for details: 
> https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading
> [datastax.com]
> 
>
>
>
> On Fri, Jan 17, 2020 at 7:34 AM Jean Tremblay <
> jean.tremb...@zen-innovations.com> wrote:
>
> Did you think about using a Materialised View to generate what you want to
> keep, and then use DSBulk to extract the data?
>
>
>
> On 17 Jan 2020, at 14:30 , adrien ruffie 
> wrote:
>
>
>
> Sorry I come back to a quick question about the bulk loader ...
>
>
>
> https://www.datastax.com/blog/2018/05/introducing-datastax-bulk-loader
> [datastax.com]
> 
>
>
>
> I read this : "Operations such as converting strings 

Re: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster few having same keyspace/table names

[Dor Laor]

Another option instead of raw sstables is to use the Spark Migrator [1].
It reads a source cluster, can make some transformations (like
table/column naming) and
writes to a target cluster. It's a very convenient tool, OSS and free of charge.

[1] https://github.com/scylladb/scylla-migrator

On Fri, Jan 17, 2020 at 5:31 PM Erick Ramirez  wrote:
>>
>> In terms of speed, the sstableloader should be faster correct?
>> Maybe the DSE BulkLoader finds application when you want a slice of the data 
>> and not the entire cake. Is it correct?
>
>
> There's no real direct comparison because DSBulk is designed for operating on 
> data in CSV or JSON as a replacement for the COPY command. Cheers!
>
> On Sat, Jan 18, 2020 at 6:29 AM Sergio  wrote:
>>
>> Hi everyone,
>>
>> Is the DSE BulkLoader faster than the sstableloader?
>>
>> Sometimes I need to make a cluster snapshot and replicate a Cluster A to a 
>> Cluster B  with fewer performance capabilities but the same data size.
>>
>> In terms of speed, the sstableloader should be faster correct?
>>
>> Maybe the DSE BulkLoader finds application when you want a slice of the data 
>> and not the entire cake. Is it correct?
>>
>> Thanks,
>>
>> Sergio
>>
>>

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster few having same keyspace/table names

[Erick Ramirez]

>
>
> *In terms of speed, the sstableloader should be faster correct?Maybe the
> DSE BulkLoader finds application when you want a slice of the data and not
> the entire cake. Is it correct?*


There's no real direct comparison because DSBulk is designed for operating
on data in CSV or JSON as a replacement for the COPY command. Cheers!

On Sat, Jan 18, 2020 at 6:29 AM Sergio  wrote:

> Hi everyone,
>
> Is the DSE BulkLoader faster than the sstableloader?
>
> Sometimes I need to make a cluster snapshot and replicate a Cluster A to a
> Cluster B  with fewer performance capabilities but the same data size.
>
> In terms of speed, the sstableloader should be faster correct?
>
> Maybe the DSE BulkLoader finds application when you want a slice of the
> data and not the entire cake. Is it correct?
>
> Thanks,
>
> Sergio
>
>
>

Re: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster few having same keyspace/table names

[Sergio]

Hi everyone,

Is the DSE BulkLoader faster than the sstableloader?

Sometimes I need to make a cluster snapshot and replicate a Cluster A to a
Cluster B  with fewer performance capabilities but the same data size.

In terms of speed, the sstableloader should be faster correct?

Maybe the DSE BulkLoader finds application when you want a slice of the
data and not the entire cake. Is it correct?

Thanks,

Sergio

RE: [EXTERNAL] Re: COPY command with where condition

[Durity, Sean R]

sstablekeys (in the tools directory?) can extract the actual keys from your 
sstables. You have to run it on each node and then combine and de-dupe the 
final results, but I have used this technique with a query generator to extract 
data more efficiently.


Sean Durity

From: Chris Splinter 
Sent: Friday, January 17, 2020 1:47 PM
To: adrien ruffie 
Cc: user@cassandra.apache.org; Erick Ramirez 
Subject: [EXTERNAL] Re: COPY command with where condition

Do you know your partition keys?

One option could be to enumerate that list of partition keys in separate cmds 
to make the individual operations less expensive for the cluster.

For example:
Say your partition key column is called id and the ids in your database are 
[1,2,3]

You could do
./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT * FROM 
probe_sensors WHERE id = 1 AND localisation_id = 208812" -url /home/dump
./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT * FROM 
probe_sensors WHERE id = 2 AND localisation_id = 208812" -url /home/dump
./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT * FROM 
probe_sensors WHERE id = 3 AND localisation_id = 208812" -url /home/dump


Does that option work for you?



On Fri, Jan 17, 2020 at 12:17 PM adrien ruffie 
mailto:adriennolar...@hotmail.fr>> wrote:
I don't really know for the moment in production environment, but for 
developpment environment the table contains more than 10.000.000 rows.
But we need just a sub dataset of this table not the entirety ...

De : Chris Splinter 
mailto:chris.splinter...@gmail.com>>
Envoyé : vendredi 17 janvier 2020 17:40
À : adrien ruffie mailto:adriennolar...@hotmail.fr>>
Cc : user@cassandra.apache.org 
mailto:user@cassandra.apache.org>>; Erick Ramirez 
mailto:flightc...@gmail.com>>
Objet : Re: COPY command with where condition

What you are seeing there is a standard read timeout, how many rows do you 
expect back from that query?

On Fri, Jan 17, 2020 at 9:50 AM adrien ruffie 
mailto:adriennolar...@hotmail.fr>> wrote:
Thank you very much,

 so I do this request with for example -->

./dsbulk unload --dsbulk.schema.keyspace 'dev_keyspace' -query "SELECT * FROM 
probe_sensors WHERE localisation_id = 208812 ALLOW FILTERING" -url /home/dump


But I get the following error
com.datastax.dsbulk.executor.api.exception.BulkExecutionException: Statement 
execution failed: SELECT * FROM crt_sensors WHERE site_id = 208812 ALLOW 
FILTERING (Cassandra timeout during read query at consistency LOCAL_ONE (1 
responses were required but only 0 replica responded))

but I configured my driver with following driver.conf, but nothing work 
correctly. Do you know what is the problem ?

datastax-java-driver {
basic {


contact-points = ["data1com:9042","data2.com:9042 
[data2.com]"]

request {
timeout = "200"
consistency = "LOCAL_ONE"

}
}
advanced {

auth-provider {
class = PlainTextAuthProvider
username = "superuser"
password = "mypass"

}
}
}

De : Chris Splinter 
mailto:chris.splinter...@gmail.com>>
Envoyé : vendredi 17 janvier 2020 16:17
À : user@cassandra.apache.org 
mailto:user@cassandra.apache.org>>
Cc : Erick Ramirez mailto:flightc...@gmail.com>>
Objet : Re: COPY command with where condition

DSBulk has an option that lets you specify the query ( including a WHERE clause 
)

See Example 19 in this blog post for details: 
https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading 
[datastax.com]

On Fri, Jan 17, 2020 at 7:34 AM Jean Tremblay 
mailto:jean.tremb...@zen-innovations.com>> 
wrote:
Did you think about using a Materialised View to generate what you want to 
keep, and then use DSBulk to extract the data?


On 17 Jan 2020, at 14:30 , adrien ruffie 
mailto:adriennolar...@hotmail.fr>> wrote:

Sorry I come back to a quick question about the bulk loader ...

https://www.datastax.com/blog/2018/05/introducing-datastax-bulk-loader 
[datastax.com]

I read this : "Operations such as converting strings to lowercase, arithmetic 
on input columns, or filtering out rows based on some criteria, are not 
supported. "

Consequently, it's still not possible to use a WHERE clause with DSBulk, right ?

I don't really know how I can do it, in order to don't keep the wholeness of 

RE: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster few having same keyspace/table names

[Durity, Sean R]

Not sure what you mean by “online” migration. You can load data into the same 
name table in cluster B. If the primary keys match, data will be overwritten 
(effectively, not actually on disk). I think you can pipe the output of a 
dsbulk unload to a dsbulk load and make the data transfer very quick. Your 
clusters are very small, so this probably wouldn’t take long.

How you get the client apps to connect to the correct cluster/stop running/etc. 
is beyond the scope of Cassandra.



Sean Durity – Staff Systems Engineer, Cassandra

From: Ankit Gadhiya 
Sent: Friday, January 17, 2020 1:05 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Re: *URGENT* Migration across different Cassandra 
cluster few having same keyspace/table names

Hi Sean,

You got all valid points.

Please see my answers below -

1. Reason we want to move from 'A' to 'B' is to get rid of 'A' Azure region 
completely.

2. Cluster names in 'A' and 'B' are different.

3. DSbulk - Is there anyway I can do online migration? - I still need to get 
clarity on whether data for same keyspace/table names can be merged between A 
and B. So 2 cases -  1. If merge is not an issue - I guess DSBulk or 
SSTableloader would be an option? 2. If merge is an issue - I am guessing 
without app code change - this wont be possible ,right?


Thanks & Regards,
Ankit Gadhiya


On Fri, Jan 17, 2020 at 9:40 AM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
A couple things to consider:

  *   A separation of apps into their own clusters is typically a better model 
to avoid later entanglements
  *   Dsbulk (1.4.1) is now available for only open source clusters. It is a 
great tool for unloading/loading
  *   What data problem are you trying to solve with Cassandra and this move to 
another cluster? If it is high-availability, then trying to get to 2 DCs would 
be important. However, I think you will need at least a new keyspace if you 
can’t combine the data from the clusters. Whether this requires a code or 
config change depends on how configurable the developers made the connection 
and query details. (As a side rant: why is it that developers will write all 
kinds of new code, but don’t want to touch existing code?)
  *   Your migration requirements are quite stringent (“we don’t want to change 
anything, lose anything, or stop anything. Make it happen!”). There may be a 
solution, but you may end up with something even more fragile afterwards. I 
would push back to see what is negotiable.



Sean Durity – Staff Systems Engineer, Cassandra

From: Ankit Gadhiya mailto:ankitgadh...@gmail.com>>
Sent: Friday, January 17, 2020 8:50 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster 
few having same keyspace/table names

Hi Upasana,

Thanks for your response. I’d love to do that as a first strategy but since 
they are both separate clusters , how would I do that? Keyspaces already have 
networktopologystrategy with RF=3.


— Ankit

On Fri, Jan 17, 2020 at 8:45 AM Upasana Sharma 
<028upasana...@gmail.com<mailto:028upasana...@gmail.com>> wrote:
Hi,

Did you consider adding Cassandra nodes from cluster B,  into cluster A as a 
different data center ?

Your keyspace would than be on Network topology data strategy.

In this case, all data can be synced between both data centers by Cassandra 
using rebalancing.


At client/application level you will have to ensure local quorum/ local 
consistency  so that there is no impact on latencies.

Once you have moved data applications to new cluster , you can then remove the 
old data center (cluster A),  and cluster B would have fresh data.




On Fri, Jan 17, 2020, 6:59 PM Ankit Gadhiya 
mailto:ankitgadh...@gmail.com>> wrote:
Thanks but there’s no DSE License.
Wondering how sstableloader will help as some oh the Keyspace and tables names 
are same. Also how do i sync few system keyspaces.


Thanks & Regards,
Ankit

On Fri, Jan 17, 2020 at 1:11 AM Vova Shelgunov 
mailto:vvs...@gmail.com>> wrote:
Loader*

https://www.datastax.com/blog/2018/05/introducing-datastax-bulk-loader 
[datastax.com]<https://urldefense.com/v3/__https:/www.datastax.com/blog/2018/05/introducing-datastax-bulk-loader__;!!M-nmYVHPHQ!ZYeKjPXZF1wl9Nz0tJN8gy3m46Qf4nw7EmJX_Wd5ecuSBeP0V8GyjQhTiQh8hnDvcRk_RUg$>

On Fri, Jan 17, 2020, 09:09 Vova Shelgunov 
mailto:vvs...@gmail.com>> wrote:
DataStax bulk loaded can be an option if data is large.

On Fri, Jan 17, 2020, 07:33 Nitan Kainth 
mailto:nitankai...@gmail.com>> wrote:
If the keyspace already exist, use copy command or sstableloader to merge data. 
If data volume it too big, consider spark or a custom java program

Regards,
Nitan
Cell: 510 449 9629

On Jan 16, 2020, at 10:26 PM, Ankit Gadhiya 
mailto:ankitgadh...@gmail.com>> wrote:

Any leads on this ?

— Ankit

On Thu, Jan 16, 2020 at 8:51 PM Ankit Gadhiya 
mailto:ankitgadh...@g

Re: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster few having same keyspace/table names

[Ankit Gadhiya]

Hi Sean,

You got all valid points.

Please see my answers below -

1. Reason we want to move from 'A' to 'B' is to get rid of 'A' Azure region
completely.

2. Cluster names in 'A' and 'B' are different.

3. DSbulk - Is there anyway I can do online migration? - I still need to
get clarity on whether data for same keyspace/table names can be merged
between A and B. So 2 cases -  1. If merge is not an issue - I guess DSBulk
or SSTableloader would be an option? 2. If merge is an issue - I am
guessing without app code change - this wont be possible ,right?


*Thanks & Regards,*
*Ankit Gadhiya*



On Fri, Jan 17, 2020 at 9:40 AM Durity, Sean R 
wrote:

> A couple things to consider:
>
>- A separation of apps into their own clusters is typically a better
>model to avoid later entanglements
>- Dsbulk (1.4.1) is now available for only open source clusters. It is
>a great tool for unloading/loading
>- What data problem are you trying to solve with Cassandra and this
>move to another cluster? If it is high-availability, then trying to get to
>2 DCs would be important. However, I think you will need at least a new
>keyspace if you can’t combine the data from the clusters. Whether this
>requires a code or config change depends on how configurable the developers
>made the connection and query details. (As a side rant: why is it that
>developers will write all kinds of new code, but don’t want to touch
>existing code?)
>- Your migration requirements are quite stringent (“we don’t want to
>change anything, lose anything, or stop anything. Make it happen!”). There
>may be a solution, but you may end up with something even more fragile
>afterwards. I would push back to see what is negotiable.
>
>
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Ankit Gadhiya 
> *Sent:* Friday, January 17, 2020 8:50 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: *URGENT* Migration across different Cassandra
> cluster few having same keyspace/table names
>
>
>
> Hi Upasana,
>
>
>
> Thanks for your response. I’d love to do that as a first strategy but
> since they are both separate clusters , how would I do that? Keyspaces
> already have networktopologystrategy with RF=3.
>
>
>
>
>
> — Ankit
>
>
>
> On Fri, Jan 17, 2020 at 8:45 AM Upasana Sharma <028upasana...@gmail.com>
> wrote:
>
> Hi,
>
>
>
> Did you consider adding Cassandra nodes from cluster B,  into cluster A as
> a different data center ?
>
>
>
> Your keyspace would than be on Network topology data strategy.
>
>
>
> In this case, all data can be synced between both data centers by
> Cassandra using rebalancing.
>
>
>
>
>
> At client/application level you will have to ensure local quorum/ local
> consistency  so that there is no impact on latencies.
>
>
>
> Once you have moved data applications to new cluster , you can then remove
> the old data center (cluster A),  and cluster B would have fresh data.
>
>
>
>
>
>
>
>
>
> On Fri, Jan 17, 2020, 6:59 PM Ankit Gadhiya 
> wrote:
>
> Thanks but there’s no DSE License.
>
> Wondering how sstableloader will help as some oh the Keyspace and tables
> names are same. Also how do i sync few system keyspaces.
>
>
>
>
>
> Thanks & Regards,
>
> Ankit
>
>
>
> On Fri, Jan 17, 2020 at 1:11 AM Vova Shelgunov  wrote:
>
> Loader*
>
>
>
> https://www.datastax.com/blog/2018/05/introducing-datastax-bulk-loader
> [datastax.com]
> 
>
>
>
> On Fri, Jan 17, 2020, 09:09 Vova Shelgunov  wrote:
>
> DataStax bulk loaded can be an option if data is large.
>
>
>
> On Fri, Jan 17, 2020, 07:33 Nitan Kainth  wrote:
>
> If the keyspace already exist, use copy command or sstableloader to merge
> data. If data volume it too big, consider spark or a custom java program
>
>
>
> Regards,
>
> Nitan
>
> Cell: 510 449 9629
>
>
>
> On Jan 16, 2020, at 10:26 PM, Ankit Gadhiya 
> wrote:
>
> 
>
> Any leads on this ?
>
>
>
> — Ankit
>
>
>
> On Thu, Jan 16, 2020 at 8:51 PM Ankit Gadhiya 
> wrote:
>
> Hi Arvinder,
>
>
>
> Thanks for your response.
>
>
>
> Yes - Cluster B already has some data. Tables/KS names are identical ; for
> data - I still haven't got the clarity if it has identical data or no - I
> am assuming no since it's for different customers but need the confirmation.
>
>
>
> *Thanks & Regards,*
>
> *Ankit Gadhiya*
>
>
>
>
>
> On Thu, Jan 16, 2020 at 8:49 PM Arvinder Dhillon 
> wrote:
>
> So as I understand, Cluster B already has some data and not an empty
> cluster.
>
>
>
> When you say, clusters share same keyspace and table names, do you mean
> both clusters have identical data on those ks/tables?
>
>
>
> -Arvi
>
>
>
> On Thu, Jan 16, 2020, 5:27 PM Ankit Gadhiya 
> wrote:
>
> Hello Group,
>
>
>
> I have a requirement in one of the production systems where I need to be
> able to 

Re: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster few having same keyspace/table names

[Jeff Jirsa]

The migration requirements are impossible given the current state of the 
database

You probably can’t join two distinct clusters without app changes and without 
downtime unless you’re very lucky (same cluster name, app using quorum but not 
local quorum, both clusters using NetworkTopologyStrategy, neither app using 
serial reads or writes), and trying to do it with conflicting keyspace and 
table names makes it impossible 

Would just assume this isn’t possible and look for alternate plans, like 
downtime or code changes. 


> On Jan 17, 2020, at 6:40 AM, Durity, Sean R  
> wrote:
> 
> 
> A couple things to consider:
> A separation of apps into their own clusters is typically a better model to 
> avoid later entanglements
> Dsbulk (1.4.1) is now available for only open source clusters. It is a great 
> tool for unloading/loading
> What data problem are you trying to solve with Cassandra and this move to 
> another cluster? If it is high-availability, then trying to get to 2 DCs 
> would be important. However, I think you will need at least a new keyspace if 
> you can’t combine the data from the clusters. Whether this requires a code or 
> config change depends on how configurable the developers made the connection 
> and query details. (As a side rant: why is it that developers will write all 
> kinds of new code, but don’t want to touch existing code?)
> Your migration requirements are quite stringent (“we don’t want to change 
> anything, lose anything, or stop anything. Make it happen!”). There may be a 
> solution, but you may end up with something even more fragile afterwards. I 
> would push back to see what is negotiable.
>  
>  
>  
> Sean Durity – Staff Systems Engineer, Cassandra
>  
> From: Ankit Gadhiya  
> Sent: Friday, January 17, 2020 8:50 AM
> To: user@cassandra.apache.org
> Subject: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster 
> few having same keyspace/table names
>  
> Hi Upasana,
>  
> Thanks for your response. I’d love to do that as a first strategy but since 
> they are both separate clusters , how would I do that? Keyspaces already have 
> networktopologystrategy with RF=3.
>  
>  
> — Ankit
>  
> On Fri, Jan 17, 2020 at 8:45 AM Upasana Sharma <028upasana...@gmail.com> 
> wrote:
> Hi,
>  
> Did you consider adding Cassandra nodes from cluster B,  into cluster A as a 
> different data center ? 
>  
> Your keyspace would than be on Network topology data strategy. 
>  
> In this case, all data can be synced between both data centers by Cassandra 
> using rebalancing.
>  
>  
> At client/application level you will have to ensure local quorum/ local 
> consistency  so that there is no impact on latencies.
>  
> Once you have moved data applications to new cluster , you can then remove 
> the old data center (cluster A),  and cluster B would have fresh data.
>  
>  
>  
>  
> On Fri, Jan 17, 2020, 6:59 PM Ankit Gadhiya  wrote:
> Thanks but there’s no DSE License.
> Wondering how sstableloader will help as some oh the Keyspace and tables 
> names are same. Also how do i sync few system keyspaces.
>  
>  
> Thanks & Regards,
> Ankit
>  
> On Fri, Jan 17, 2020 at 1:11 AM Vova Shelgunov  wrote:
> Loader*
>  
> https://www.datastax.com/blog/2018/05/introducing-datastax-bulk-loader 
> [datastax.com]
>  
> On Fri, Jan 17, 2020, 09:09 Vova Shelgunov  wrote:
> DataStax bulk loaded can be an option if data is large. 
>  
> On Fri, Jan 17, 2020, 07:33 Nitan Kainth  wrote:
> If the keyspace already exist, use copy command or sstableloader to merge 
> data. If data volume it too big, consider spark or a custom java program 
> 
>  
> Regards,
> Nitan
> Cell: 510 449 9629
> 
> 
> On Jan 16, 2020, at 10:26 PM, Ankit Gadhiya  wrote:
> 
> 
> Any leads on this ?
>  
> — Ankit
>  
> On Thu, Jan 16, 2020 at 8:51 PM Ankit Gadhiya  wrote:
> Hi Arvinder,
>  
> Thanks for your response.
>  
> Yes - Cluster B already has some data. Tables/KS names are identical ; for 
> data - I still haven't got the clarity if it has identical data or no - I am 
> assuming no since it's for different customers but need the confirmation.
>  
> Thanks & Regards,
> Ankit Gadhiya
> 
>  
>  
> On Thu, Jan 16, 2020 at 8:49 PM Arvinder Dhillon  
> wrote:
> So as I understand, Cluster B already has some data and not an empty cluster.
>  
> When you say, clusters share same keyspace and table names, do you mean both 
> clusters have identical data on those ks/tables?
>  
> 
> -Arvi
>  
> On Thu, Jan 16, 2020, 5:27 PM Ankit Gadhiya  wrote:
> Hello Group,
>  
> I have a requirement in one of the production systems where I need to be able 
> to migrate entire dataset from Cluster A (Azure Region A) to Cluster B (Azure 
> Region B). 
>  
> Each cluster have 3 Cassandra nodes (RF=3) running used by different 
> applications. Few of the applications are common is Cluster A and Cluster B 
> thereby sharing same keyspace/table names. 
> Need suggestion for the best possible migration strategy here considering - 
> 

RE: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster few having same keyspace/table names

[Durity, Sean R]

A couple things to consider:

  *   A separation of apps into their own clusters is typically a better model 
to avoid later entanglements
  *   Dsbulk (1.4.1) is now available for only open source clusters. It is a 
great tool for unloading/loading
  *   What data problem are you trying to solve with Cassandra and this move to 
another cluster? If it is high-availability, then trying to get to 2 DCs would 
be important. However, I think you will need at least a new keyspace if you 
can’t combine the data from the clusters. Whether this requires a code or 
config change depends on how configurable the developers made the connection 
and query details. (As a side rant: why is it that developers will write all 
kinds of new code, but don’t want to touch existing code?)
  *   Your migration requirements are quite stringent (“we don’t want to change 
anything, lose anything, or stop anything. Make it happen!”). There may be a 
solution, but you may end up with something even more fragile afterwards. I 
would push back to see what is negotiable.



Sean Durity – Staff Systems Engineer, Cassandra

From: Ankit Gadhiya 
Sent: Friday, January 17, 2020 8:50 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: *URGENT* Migration across different Cassandra cluster 
few having same keyspace/table names

Hi Upasana,

Thanks for your response. I’d love to do that as a first strategy but since 
they are both separate clusters , how would I do that? Keyspaces already have 
networktopologystrategy with RF=3.


— Ankit

On Fri, Jan 17, 2020 at 8:45 AM Upasana Sharma 
<028upasana...@gmail.com> wrote:
Hi,

Did you consider adding Cassandra nodes from cluster B,  into cluster A as a 
different data center ?

Your keyspace would than be on Network topology data strategy.

In this case, all data can be synced between both data centers by Cassandra 
using rebalancing.


At client/application level you will have to ensure local quorum/ local 
consistency  so that there is no impact on latencies.

Once you have moved data applications to new cluster , you can then remove the 
old data center (cluster A),  and cluster B would have fresh data.




On Fri, Jan 17, 2020, 6:59 PM Ankit Gadhiya 
mailto:ankitgadh...@gmail.com>> wrote:
Thanks but there’s no DSE License.
Wondering how sstableloader will help as some oh the Keyspace and tables names 
are same. Also how do i sync few system keyspaces.


Thanks & Regards,
Ankit

On Fri, Jan 17, 2020 at 1:11 AM Vova Shelgunov 
mailto:vvs...@gmail.com>> wrote:
Loader*

https://www.datastax.com/blog/2018/05/introducing-datastax-bulk-loader 
[datastax.com]

On Fri, Jan 17, 2020, 09:09 Vova Shelgunov 
mailto:vvs...@gmail.com>> wrote:
DataStax bulk loaded can be an option if data is large.

On Fri, Jan 17, 2020, 07:33 Nitan Kainth 
mailto:nitankai...@gmail.com>> wrote:
If the keyspace already exist, use copy command or sstableloader to merge data. 
If data volume it too big, consider spark or a custom java program

Regards,
Nitan
Cell: 510 449 9629


On Jan 16, 2020, at 10:26 PM, Ankit Gadhiya 
mailto:ankitgadh...@gmail.com>> wrote:

Any leads on this ?

— Ankit

On Thu, Jan 16, 2020 at 8:51 PM Ankit Gadhiya 
mailto:ankitgadh...@gmail.com>> wrote:
Hi Arvinder,

Thanks for your response.

Yes - Cluster B already has some data. Tables/KS names are identical ; for data 
- I still haven't got the clarity if it has identical data or no - I am 
assuming no since it's for different customers but need the confirmation.

Thanks & Regards,
Ankit Gadhiya


On Thu, Jan 16, 2020 at 8:49 PM Arvinder Dhillon 
mailto:dhillona...@gmail.com>> wrote:
So as I understand, Cluster B already has some data and not an empty cluster.

When you say, clusters share same keyspace and table names, do you mean both 
clusters have identical data on those ks/tables?

-Arvi

On Thu, Jan 16, 2020, 5:27 PM Ankit Gadhiya 
mailto:ankitgadh...@gmail.com>> wrote:
Hello Group,

I have a requirement in one of the production systems where I need to be able 
to migrate entire dataset from Cluster A (Azure Region A) to Cluster B (Azure 
Region B).

Each cluster have 3 Cassandra nodes (RF=3) running used by different 
applications. Few of the applications are common is Cluster A and Cluster B 
thereby sharing same keyspace/table names.
Need suggestion for the best possible migration strategy here considering - 1. 
No Application code changes possible - Minor config/infra changes can be 
considered. 2. Zero data loss. 3. No/Minimal downtime.

It'd be great to hear ideas from all of you based on your experiences.

Cassandra Version - Cassandra 3.0.13 on both sides.
Total Data size - Cluster A: 70 GB, Cluster B: 15 GB

Thanks & Regards,
Ankit Gadhiya
--
Thanks & Regards,
Ankit Gadhiya
--
Thanks & Regards,
Ankit Gadhiya
--

RE: [EXTERNAL] Re: Log output when Cassandra is "up"?

[Durity, Sean R]

I use a script that calls nodetool info. If nodetool info returns an error 
(instance isn’t up, on the way up, etc.) then I return that same error code 
(and I know the node is NOT OK). If nodetool info succeeds, I then parse the 
output for each protocol to be up. A node can be up, but have gossip or cql 
down/unavailable. That node is also NOT OK.

If all this passes, then the node is OK.

I would love if there was something simpler, like the Informix onstat - | grep 
“On-line” …

Sean Durity

From: Deepak Vohra 
Sent: Wednesday, January 8, 2020 4:12 PM
To: user@cassandra.apache.org; Voytek Jarnot 
Subject: [EXTERNAL] Re: Log output when Cassandra is "up"?


Use the nodetool status command.


nodetool status

Datacenter: us-east-1

=

Status=Up/Down|/ State=Normal/Leaving/Joining/Moving--  Address Load
Tokens  Owns (effective)  Host ID   Rack

UN  10.0.1.115  205.6 KiB   256 66.9% 
b64cb32a-b32a-46b4-9eeb-e123fa8fc287  us-east-1b

UN  10.0.3.206  182.67 KiB  256 63.5% 
74863177-684b-45f4-99f7-d1006625dc9e  us-east-1d

UN  10.0.2.238  240.81 KiB  256 69.6% 
4dcdadd2-41f9-4f34-9892-1f20868b27c7  us-east-1c


UN in 1st column implies status is Up and Normal.


On Wednesday, January 8, 2020, 08:37:42 p.m. UTC, Voytek Jarnot 
mailto:voytek.jar...@gmail.com>> wrote:


Needing to know when Cassandra is finished initializing and is up & running.

Had some scripts which were looking through system.log for "No gossip backlog; 
proceeding", but that turns out not to be 100% reliable.

Is looking for "Starting listening for CQL clients" considered definitive? 
I.E., always gets output on success, and not on failure?

Thanks



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Re: How bottom of cassandra save data efficiently?

[lampahome]

Thank you all.

I found doc in datastax and it said the compression is default to enabled
and set it as LZ4Compressor.

RE: [EXTERNAL] Re: How bottom of cassandra save data efficiently?

[Durity, Sean R]

100,000 rows is pretty small. Import your data to your cluster, do a nodetool 
flush on each node, then you can see how much disk space is actually used.

There are different compression tools available to you when you create the 
table. It also matters if the rows are in separate partitions or you have many 
rows per partition. In one exercise I have done, individual partitions can 
cause the data to expand from 0.3 MB (with many rows per partition) to 20 MB 
(one row per partition) – all from the same data set. Your compaction settings 
can also change the size of data on disk.

Bottom line – precise math requires more parameters than you have given. Actual 
experimentation is easier.


Sean Durity

From: lampahome 
Sent: Wednesday, January 1, 2020 8:33 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: How bottom of cassandra save data efficiently?



Dipan Shah mailto:dipan@hotmail.com>> 於 2019年12月31日 
週二 下午5:34寫道：
Hello lampahome,

Data will be compressed but you will also have to account for the replication 
factor that you will be using.


Thanks,

Dipan Shah


The key factor about efficiency is replication factor. Are there other factors?




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: [EXTERNAL] Re: Facing issues while starting Cassandra

[Durity, Sean R]

Any read-only file systems? Have you tried to start from the command line 
(instead of a service)? Sometimes that will give a more helpful error when 
start-up can’t complete.

If your error is literally what you included, it looks like the executable 
can’t find the cassandra.yaml file.

I will agree with Jeff, though. When I have seen a similar error it has usually 
been a yaml violation, such as having a tab (instead of spaces) in the yaml 
file. Check that specific node’s file with a yaml lint detector?

Sean Durity

From: Inquistive allen 
Sent: Tuesday, December 24, 2019 2:01 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Facing issues while starting Cassandra

Hello Osman,

Thanks for the suggestion.
I did try "export LC_ALL=C"
It didn't help.

Thanks

On Tue, 24 Dec, 2019, 12:05 PM Osman Yozgatlıoğlu, 
mailto:osman.yozgatlio...@gmail.com>> wrote:
I faced similar issues with different locale settings.
Could you try following command before running?
export LC_ALL=C;

Regards,
Osman

On Tue, 24 Dec 2019 at 09:01, Inquistive allen 
mailto:inquial...@gmail.com>> wrote:
>
> Hello Jeff,
>
> Thanks for responding.
> I have validated the cassandra.yaml file with other hosts in the cluster.
> There is no difference. I copied a yaml file from other node to this node and 
> changed the required configs. Still facing the same issue.
> The server went down for patching and after coming back up, Cassandra dosent 
> seem to start.
> Having looked for solutions on google, I found that it might be a problem 
> with the /tmp directory where the classes are stored.
> Each time I try starting Cassandra, in the /tmp directory a new directory is 
> created, but nothing is inside the directory. After some time, the node goes 
> down.
>
> I believe there is something to do with the /tmp directory.
> Request you to comment on the same.
>
> Thanks
>
> On Tue, 24 Dec, 2019, 3:42 AM Jeff Jirsa, 
> mailto:jji...@gmail.com>> wrote:
>>
>> Are you able to share the yaml? Almost certainly something in it that’s 
>> invalid.
>>
>> On Dec 23, 2019, at 12:51 PM, Inquistive allen 
>> mailto:inquial...@gmail.com>> wrote:
>>
>> 
>> Hello Team,
>>
>> I am facing issues while starting Cassandra.
>>
>> Caused by: org.apache.cassandra.exceptions.ConfigurationException : Invalid 
>> yaml: file: /path/to/yaml
>> Error: null ; can't construct a java object for tag: yaml.org 
>> [yaml.org],2002:org.apache.cassandra.config.Config;
>>  exception= java.lang.reflect.InvocationTargetException
>>
>> Request to comment on how to resolve the issue.
>>
>> Thanks & Regards
>> Allen

-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 
user-h...@cassandra.apache.org



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Re: Connection Pooling in v4.x Java Driver

[Caravaggio, Kevin]

Hi Alexandre,


Thank you for the explanation. I understand that reasoning very well now.

Jon, appreciate the link, and will follow up there for this sort of thing then.


Thanks,


Kevin
From: Alexandre Dutra 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, December 11, 2019 at 3:33 AM
To: "user@cassandra.apache.org" 
Subject: [EXTERNAL] Re: Connection Pooling in v4.x Java Driver


*EXTERNAL SENDER*

Hi,

In driver 4.x, pools do not resize dynamically anymore because the ratio 
between concrete benefits brought by this feature and the maintenance burden it 
caused was largely unfavorable: most bugs related to connection pooling in 
driver 3.x were caused by the dynamic pool resizing. Having a fixed pool size 
made driver 4.x pool implementation much more straightforward and robust.

Thanks,

Alex Dutra


On Tue, Dec 10, 2019 at 7:13 PM Jon Haddad 
mailto:j...@jonhaddad.com>> wrote:
I'm not sure how closely the driver maintainers are following this list. You 
might want to ask on the Java Driver mailing list: 
https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user




On Tue, Dec 10, 2019 at 5:10 PM Caravaggio, Kevin 
mailto:kevin.caravag...@lowes.com>> wrote:
Hello,


When integrating with DataStax OSS Cassandra Java driver v4.x, I noticed 
“Unlike previous versions of the driver, pools do not resize 
dynamically”
 in reference to the connection pool configuration. Is anyone aware of the 
reasoning for this departure from dynamic pool sizing, which I believe was 
available in v3.x?


Thanks,


Kevin


NOTICE: All information in and attached to the e-mails below may be 
proprietary, confidential, privileged and otherwise protected from improper or 
erroneous disclosure. If you are not the sender's intended recipient, you are 
not authorized to intercept, read, print, retain, copy, forward, or disseminate 
this message. If you have erroneously received this communication, please 
notify the sender immediately by phone (704-758-1000) or by e-mail and destroy 
all copies of this message electronic, paper, or otherwise. By transmitting 
documents via this email: Users, Customers, Suppliers and Vendors collectively 
acknowledge and agree the transmittal of information via email is voluntary, is 
offered as a convenience, and is not a secured method of communication; Not to 
transmit any payment information E.G. credit card, debit card, checking 
account, wire transfer information, passwords, or sensitive and personal 
information E.G. Driver's license, DOB, social security, or any other 
information the user wishes to remain confidential; To transmit only 
non-confidential information such as plans, pictures and drawings and to assume 
all risk and liability for and indemnify Lowe's from any claims, losses or 
damages that may arise from the transmittal of documents or including 
non-confidential information in the body of an email transmittal. Thank you.


--

Alexandre Dutra  |  Technical Manager, Drivers

alexandre.du...@datastax.com  |  
datastax.com

[Image removed by 
sender.][Image
 removed by 
sender.][Image
 removed by 

RE: [EXTERNAL] Migration a Keyspace from 3.0.X to 3.11.2 Cluster which already have keyspaces

[Durity, Sean R]

The size of the data matters here. Copy to/from is ok if the data is a few 
million rows per table, but not billions. It is also relatively slow (but with 
small data or a decent outage window, it could be fine). If the data is large 
and the outage time matters, you may need custom code to read from one cluster 
and write to another. If this is DataStax, the dsbulk utility would be ideal.


Sean Durity
-Original Message-
From: slmnjobs - 
Sent: Sunday, December 1, 2019 4:41 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Migration a Keyspace from 3.0.X to 3.11.2 Cluster which 
already have keyspaces

Hi everyone,

I have a question about migrate a keyspace another cluster. The main problem 
for us, our new cluster already have 2 keyspaces and we using it in production. 
Because of we don't sure how token ranges will be change, we would like the 
share our migration plan here and take back your comments.
 We have two Cassandra clusters which one of them:

CLUSTER-A :
- Cassandra version 3.0.10
- describe keyspace:
CREATE KEYSPACE mykeyspace WITH replication = {'class': 
'NetworkTopologyStrategy', 'DC1': '3', 'DC2': '3', 'DC3': '1'}  AND 
durable_writes = true;
- DC1 : 6 nodes
- DC2 : 6 nodes
- DC3 : 1 node (backup node, have all data)

CLUSTER-B :
- Cassandra version 3.11.2- DC1 : 5 nodes
- DC2 : 5 nodes- DC3 : 1 node
- Already have 2 keyspaces and write/read traffic

We want to migrate a keyspace which on CLUSTER-A to CLUSTER-B. There're some 
solutions for restore or migrate a keyspace on a new cluster but I haven't seen 
any safety way about how we can migrate a keyspace on existing cluster which 
already have keyspaces.

Replication Factor won't change.
We think about two ways : one of them using sstableloader and other one using 
COPY TO/COPY FROM commands.

Our migration plan is:

- export of keyspace schema structure with DESC keyspace on CLUSTER-A
- create keyspace schema on CLUSTER-B
- disable writing traffic on application layer
- load data from CLUSTER-A, DC3 backup node (which have all data) to CLUSTER-B, 
DC1 with sstableloader or COPY command (each table wiil be copy one by one).
- update cluster IP addresses in application configuration
- enable writing traffic on application layer
So do you see any risk or any suggestion for this plan? Thanks a lot.

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: [EXTERNAL] Re: Upgrade strategy for high number of nodes

[Durity, Sean R]

All my upgrades are without downtime for the application. Yes, do the binary 
upgrade one node at a time. Then run upgradesstables on as many nodes as your 
app load can handle (maybe you can point the app to a different DC, while 
another DC is doing upgradesstables). Upgradesstables doesn’t cause downtime – 
it just increases the IO load on the nodes executing the upgradesstables. I try 
to get it done as quickly as possible, because I suspend streaming operations 
(repairs, etc.) until the sstable rewrites are completed.

Sean Durity

From: Shishir Kumar 
Sent: Saturday, November 30, 2019 1:00 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Upgrade strategy for high number of nodes

Thanks for pointer. We haven't much changed data model since long, so before 
workarounds (scrub) worth understanding root cause of problem.
This might be reason why running upgradesstables in parallel was not 
recommended.
-Shishir
On Sat, 30 Nov 2019, 10:37 Jeff Jirsa, 
mailto:jji...@gmail.com>> wrote:
Scrub really shouldn’t be required here.

If there’s ever a step that reports corruption, it’s either a very very old 
table where you dropped columns previously or did something “wrong” in the past 
or a software bug. The old dropped column really should be obvious in the stack 
trace - anything else deserves a bug report.

It’s unfortunate that people jump to just scrubbing the unreadable data - would 
appreciate an anonymized JIRA if possible. Alternatively work with your vendor 
to make sure they don’t have bugs in their readers somehow.





On Nov 29, 2019, at 8:58 PM, Shishir Kumar 
mailto:shishirroy2...@gmail.com>> wrote:

Some more background. We are planning (tested) binary upgrade across all nodes 
without downtime. As next step running upgradesstables. As C* file format and 
version (from format big, version mc to format bti, version aa (Refer 
https://docs.datastax.com/en/dse/6.0/dse-admin/datastax_enterprise/tools/toolsSStables/ToolsSSTableupgrade.html
 
[docs.datastax.com]
 - upgrade from DSE 5.1 to 6.x). Underlying changes explains why it takes too 
much time to upgrade.
Running  upgradesstables  in parallel across RAC - This is where I am not sure 
on impact of running in parallel (document recommends to run one node at time). 
During upgradesstables there are scenario's where it report file corruption, 
hence require corrective step I.e. scrub. Due to file corruption at times nodes 
goes down due to sstable corruption or result in high CPU usage ~100%. 
Performing above in parallel without downtime might result in more 
inconsistency across nodes. This scenario have not tested, so will need group 
help in case they have done similar upgrade in past (I.e. scenario's/complexity 
which needs to be considered and why guideline recommend to run upgradesstable 
one node at time).
-Shishir

On Fri, Nov 29, 2019 at 11:52 PM Josh Snyder 
mailto:j...@code406.com>> wrote:
Hello Shishir,

It shouldn't be necessary to take downtime to perform upgrades of a Cassandra 
cluster. It sounds like the biggest issue you're facing is the upgradesstables 
step. upgradesstables is not strictly necessary before a Cassandra node 
re-enters the cluster to serve traffic; in my experience it is purely for 
optimizing the performance of the database once the software upgrade is 
complete. I recommend trying out an upgrade in a test environment without using 
upgradesstables, which should bring the 5 hours per node down to just a few 
minutes.

If you're running NetworkTopologyStrategy and you want to optimize further, you 
could consider performing the upgrade on multiple nodes within the same rack in 
parallel. When correctly configured, NetworkTopologyStrategy can protect your 
database from an outage of an entire rack. So performing an upgrade on a few 
nodes at a time within a rack is the same as a partial rack outage, from the 
database's perspective.

Have a nice upgrade!

Josh

On Fri, Nov 29, 2019 at 7:22 AM Shishir Kumar 
mailto:shishirroy2...@gmail.com>> wrote:
Hi,

Need input on cassandra upgrade strategy for below:
1. We have Datacenter across 4 geography (multiple isolated deployments in each 
DC).
2. Number of Cassandra nodes in each deployment is between 6 to 24
3. Data volume on each nodes between 150 to 400 GB
4. All production environment has DR set up
5. During upgrade we do not want downtime

We are planning to go for stack upgrade but upgradesstables is taking approx. 5 
hours per node (if data volume is approx 200 GB).
Options-
No downtime - As per recommendation (DataStax documentation) if we plan to 
upgrade one node at time I.e. in sequence upgrade cycle for one environment 
will take weeks, so DevOps concern.
Read Only (No downtime) - Route read only load to DR system. We have 

RE: [EXTERNAL] performance

[Durity, Sean R]

I’m not sure this is the fully correct question to ask. The size of the data 
will matter. The importance of high availability matters. Performance can be 
tuned by taking advantage of Cassandra’s design strengths. In general, you 
should not be doing queries with a where clause on non-key columns. Secondary 
indexes are not what you would expect from a relational background (and should 
normally be avoided).

In short, choose Cassandra if you need high-availability and low latency on 
KNOWN access patterns (on which you base your table design).

If you want an opinion – I would never put data over a few hundred GB that I 
care about into mysql. I don’t like the engine, the history, the company, or 
anything about it. But that’s just my opinion. I know many companies have 
successfully used mysql.


Sean Durity

From: hahaha sc 
Sent: Friday, November 29, 2019 3:27 AM
To: cassandra-user 
Subject: [EXTERNAL] performance

Query based on a field with a non-primary key and a secondary index, and then 
update based on the primary key. Can it be  more efficient than mysql?



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: [EXTERNAL] Re: Cassandra 3.11.4 Node the load starts to increase after few minutes to 40 on 4 CPU machine

[Durity, Sean R]

There is definitely a resource risk to having thousands of open connections to 
each node. Some of the drivers have (had?) less than optimal default settings, 
like acquiring 50 connections per Cassandra node. This is usually overkill. I 
think 5-10/node is much more reasonable. It depends on your app architecture 
and cluster node count. If there are lots of small micro-services, maybe they 
only need 2 connections per node.


Sean Durity – Staff Systems Engineer, Cassandra

From: Sergio 
Sent: Wednesday, October 30, 2019 5:39 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Cassandra 3.11.4 Node the load starts to increase after 
few minutes to 40 on 4 CPU machine

Hi Reid,

I don't have anymore this loading problem.
I solved by changing the Cassandra Driver Configuration.
Now my cluster is pretty stable and I don't have machines with crazy CPU Load.
The only thing not urgent but I need to investigate is the number of 
ESTABLISHED TCP connections. I see just one node having 7K TCP connections 
ESTABLISHED while the others are having around 4-6K connection opened. So the 
newest nodes added into the cluster have a higher number of ESTABLISHED TCP 
connections.

default['cassandra']['sysctl'] = {
'net.ipv4.tcp_keepalive_time' => 60,
'net.ipv4.tcp_keepalive_probes' => 3,
'net.ipv4.tcp_keepalive_intvl' => 10,
'net.core.rmem_max' => 16777216,
'net.core.wmem_max' => 16777216,
'net.core.rmem_default' => 16777216,
'net.core.wmem_default' => 16777216,
'net.core.optmem_max' => 40960,
'net.ipv4.tcp_rmem' => '4096 87380 16777216',
'net.ipv4.tcp_wmem' => '4096 65536 16777216',
'net.ipv4.ip_local_port_range' => '1 65535',
'net.ipv4.tcp_window_scaling' => 1,
  'net.core.netdev_max_backlog' => 2500,
  'net.core.somaxconn' => 65000,
'vm.max_map_count' => 1048575,
'vm.swappiness' => 0
}

These are my tweaked value and I used the values recommended from datastax.

Do you have something different?

Best,
Sergio

Il giorno mer 30 ott 2019 alle ore 13:27 Reid Pinchback 
mailto:rpinchb...@tripadvisor.com>> ha scritto:
Oh nvm, didn't see the later msg about just posting what your fix was.

R


On 10/30/19, 4:24 PM, "Reid Pinchback" 
mailto:rpinchb...@tripadvisor.com>> wrote:

 Message from External Sender

Hi Sergio,

Assuming nobody is actually mounting a SYN flood attack, then this sounds 
like you're either being hammered with connection requests in very short 
periods of time, or your TCP backlog tuning is off.   At least, that's where 
I'd start looking.  If you take that log message and google it (Possible SYN 
flooding... Sending cookies") you'll find explanations.  Or just googling "TCP 
backlog tuning".

R


On 10/30/19, 3:29 PM, "Sergio Bilello" 
mailto:lapostadiser...@gmail.com>> wrote:

>
>Oct 17 00:23:03 prod-personalization-live-data-cassandra-08 kernel: 
TCP: request_sock_TCP: Possible SYN flooding on port 9042. Sending cookies. 
Check SNMP counters.




-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 
user-h...@cassandra.apache.org




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: [EXTERNAL] n00b q re UPDATE v. INSERT in CQL

[Durity, Sean R]

Everything in Cassandra is an insert. So, an update and an insert are 
functionally equivalent. An update doesn't go update the existing data on disk; 
it is a new write of the columns involved. So, the difference in your scenario 
is that with the "targeted" update, you are writing less of the columns again.

So, instead of inserting 10 values (for example), you are inserting 3 (pk1, 
pk2, and col1). This would mean less disk space used for your data cleanup. 
Once Cassandra runs its compaction across your data (with a simplifying 
assumption that all data gets compacted), there would be no disk space 
difference for the final result.

I would do the updates, if the size/scope of the data involved is significant.


Sean Durity – Staff Systems Engineer, Cassandra

-Original Message-
From: James A. Robinson 
Sent: Friday, October 25, 2019 10:49 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] n00b q re UPDATE v. INSERT in CQL

Hi folks,

I'm working on a clean-up task for some bad data in a cassandra db.
The bad data in this case are values with mixed case that will need to
be lowercased.  In some tables the value that needs to be changed is a
primary key, in other cases it is not.

From the reading I've done, the situations where I need to change a
primary key column to lowercase will mean I need to perform an INSERT
of the entire row using the new primary key values merged with the old
non-primary-key values, followed by a DELETE of the old primary key
row.

My question is, on a table where I need to update a column that isn't
primary key, should I perform a limited UPDATE in that situation like
I would in SQL:

UPDATE ks.table SET col1 = ? WHERE pk1 = ? AND pk2 = ?

or will there be any downsides to that over an INSERT where I specify
all columns?

INSERT INTO ks.table (pk1, pk2, col1, col2, ...) VALUES (?,?,?,?, ...)

In SQL I'd never question just using the update but my impression
reading the blogosphere is that Cassandra has subtleties that I might
not be grasping when it comes to UPDATE v. INSERT behavior...

Jim

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org




The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: [EXTERNAL] Cassandra Export error in COPY command

[Hossein Ghiyasi Mehr]

I tested dsbulk too. But there are many errors:

"[1710949318] Error writing cancel request. This is not critical (the
request will eventually time out server-side)."
"Forcing termination of Connection[/127.0.0.1:9042-14, inFlight=1,
closed=true]. This should not happen and is likely a bug, please report."

I tried with "--executor.maxPerSecond 1000" and
"--driver.socket.readTimeout 3600" options but errors didn't resolve.

How can I export a table data?

On Mon, Sep 23, 2019 at 6:30 AM Durity, Sean R 
wrote:

> Copy command tries to export all rows in the table, not just the ones on
> the node. It will eventually timeout if the table is large. It is really
> built for something under 5 million rows or so. Dsbulk (from DataStax) is
> great for this, if you are a customer. Otherwise, you will probably need to
> write an extract of some kind. You can get keys from the sstables, then
> dedupe, then export rows one by one using the keys (kind of painful). How
> large is the table you are trying to export?
>
>
>
> Sean Durity
>
>
>
> *From:* Hossein Ghiyasi Mehr 
> *Sent:* Saturday, September 21, 2019 8:02 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra Export error in COPY command
>
>
>
> Hi all members,
>
> I want to export (pk, another_int_column) from single node using COPY
> command. But after about 1h 45m, I've got a lot of read errors:
>
>
>
>
>
> I tried this action many times but after maximum 2h, it failed with the
> errors.
>
>
>
> Any idea may help me!
>
> Thanks.
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

[Sergio]

Thanks Jon!

I used that tool and I did a test to compare LCS and STCS and it works
great. However, I was referring to the JVM flags that you use since there
are a lot of flags that I found as default and I would like to exclude the
unused or wrong ones from the current configuration.

I have also another thread opened where I am trying to figure out Kernel
Settings for TCP
https://lists.apache.org/thread.html/7708c22a1d95882598cbcc29bc34fa54c01fcb33c40bb616dcd3956d@%3Cuser.cassandra.apache.org%3E

Do you have anything to add to that?

Thanks,

Sergio

Il giorno lun 21 ott 2019 alle ore 15:09 Jon Haddad  ha
scritto:

> tlp-stress comes with workloads pre-baked, so there's not much
> configuration to do.  The main flags you'll want are going to be:
>
> -d : duration, I highly recommend running your test for a few days
> --compaction
> --compression
> -p: number of partitions
> -r: % of reads, 0-1
>
> For example, you might run:
>
> tlp-stress run KeyValue -d 24h --compaction lcs -p 10m -r .9
>
> for a basic key value table, running for 24 hours, using LCS, 10 million
> partitions, 90% reads.
>
> There's a lot of options. I won't list them all here, it's why I wrote the
> manual :)
>
> Jon
>
>
> On Mon, Oct 21, 2019 at 1:16 PM Sergio  wrote:
>
>> Thanks, guys!
>> I just copied and paste what I found on our test machines but I can
>> confirm that we have the same settings except for 8GB in production.
>> I didn't select these settings and I need to verify why these settings
>> are there.
>> If any of you want to share your flags for a read-heavy workload it would
>> be appreciated, so I would replace and test those flags with TLP-STRESS.
>> I am thinking about different approaches (G1GC vs ParNew + CMS)
>> How many GB for RAM do you dedicate to the OS in percentage or in an
>> exact number?
>> Can you share the flags for ParNew + CMS that I can play with it and
>> perform a test?
>>
>> Best,
>> Sergio
>>
>>
>> Il giorno lun 21 ott 2019 alle ore 09:27 Reid Pinchback <
>> rpinchb...@tripadvisor.com> ha scritto:
>>
>>> Since the instance size is < 32gb, hopefully swap isn’t being used, so
>>> it should be moot.
>>>
>>>
>>>
>>> Sergio, also be aware that  -XX:+CMSClassUnloadingEnabled probably
>>> doesn’t do anything for you.  I believe that only applies to CMS, not
>>> G1GC.  I also wouldn’t take it as gospel truth that  -XX:+UseNUMA is a good
>>> thing on AWS (or anything virtualized), you’d have to run your own tests
>>> and find out.
>>>
>>>
>>>
>>> R
>>>
>>> *From: *Jon Haddad 
>>> *Reply-To: *"user@cassandra.apache.org" 
>>> *Date: *Monday, October 21, 2019 at 12:06 PM
>>> *To: *"user@cassandra.apache.org" 
>>> *Subject: *Re: [EXTERNAL] Re: GC Tuning
>>> https://thelastpickle.com/blog/2018/04/11/gc-tuning.html
>>>
>>>
>>>
>>> *Message from External Sender*
>>>
>>> One thing to note, if you're going to use a big heap, cap it at 31GB,
>>> not 32.  Once you go to 32GB, you don't get to use compressed pointers [1],
>>> so you get less addressable space than at 31GB.
>>>
>>>
>>>
>>> [1]
>>> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.codecentric.de_en_2014_02_35gb-2Dheap-2Dless-2D32gb-2Djava-2Djvm-2Dmemory-2Doddities_=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=e9Ahs5XXRBicgUhMZQaboxsqb6jXpjvo48kEojUWaQc=Q7jI4ZEqVMFZIMPoSXTvMebG5fWOUJ6lhDOgWGxiHg8=>
>>>
>>>
>>>
>>> On Mon, Oct 21, 2019 at 11:39 AM Durity, Sean R <
>>> sean_r_dur...@homedepot.com> wrote:
>>>
>>> I don’t disagree with Jon, who has all kinds of performance tuning
>>> experience. But for ease of operation, we only use G1GC (on Java 8),
>>> because the tuning of ParNew+CMS requires a high degree of knowledge and
>>> very repeatable testing harnesses. It isn’t worth our time. As a previous
>>> writer mentioned, there is usually better return on our time tuning the
>>> schema (aka helping developers understand Cassandra’s strengths).
>>>
>>>
>>>
>>> We use 16 – 32 GB heaps, nothing smaller than that.
>>>
>>>
>>>
>>> Sean Durity
>>>
>>>
>>>
>>> *From:* Jon Haddad 
>>> *Sent:* Monday, October 21, 2019 10:43 AM
>>> *To:* user@

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

[Jon Haddad]

tlp-stress comes with workloads pre-baked, so there's not much
configuration to do.  The main flags you'll want are going to be:

-d : duration, I highly recommend running your test for a few days
--compaction
--compression
-p: number of partitions
-r: % of reads, 0-1

For example, you might run:

tlp-stress run KeyValue -d 24h --compaction lcs -p 10m -r .9

for a basic key value table, running for 24 hours, using LCS, 10 million
partitions, 90% reads.

There's a lot of options. I won't list them all here, it's why I wrote the
manual :)

Jon


On Mon, Oct 21, 2019 at 1:16 PM Sergio  wrote:

> Thanks, guys!
> I just copied and paste what I found on our test machines but I can
> confirm that we have the same settings except for 8GB in production.
> I didn't select these settings and I need to verify why these settings are
> there.
> If any of you want to share your flags for a read-heavy workload it would
> be appreciated, so I would replace and test those flags with TLP-STRESS.
> I am thinking about different approaches (G1GC vs ParNew + CMS)
> How many GB for RAM do you dedicate to the OS in percentage or in an exact
> number?
> Can you share the flags for ParNew + CMS that I can play with it and
> perform a test?
>
> Best,
> Sergio
>
>
> Il giorno lun 21 ott 2019 alle ore 09:27 Reid Pinchback <
> rpinchb...@tripadvisor.com> ha scritto:
>
>> Since the instance size is < 32gb, hopefully swap isn’t being used, so it
>> should be moot.
>>
>>
>>
>> Sergio, also be aware that  -XX:+CMSClassUnloadingEnabled probably
>> doesn’t do anything for you.  I believe that only applies to CMS, not
>> G1GC.  I also wouldn’t take it as gospel truth that  -XX:+UseNUMA is a good
>> thing on AWS (or anything virtualized), you’d have to run your own tests
>> and find out.
>>
>>
>>
>> R
>>
>> *From: *Jon Haddad 
>> *Reply-To: *"user@cassandra.apache.org" 
>> *Date: *Monday, October 21, 2019 at 12:06 PM
>> *To: *"user@cassandra.apache.org" 
>> *Subject: *Re: [EXTERNAL] Re: GC Tuning
>> https://thelastpickle.com/blog/2018/04/11/gc-tuning.html
>>
>>
>>
>> *Message from External Sender*
>>
>> One thing to note, if you're going to use a big heap, cap it at 31GB, not
>> 32.  Once you go to 32GB, you don't get to use compressed pointers [1], so
>> you get less addressable space than at 31GB.
>>
>>
>>
>> [1]
>> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.codecentric.de_en_2014_02_35gb-2Dheap-2Dless-2D32gb-2Djava-2Djvm-2Dmemory-2Doddities_=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=e9Ahs5XXRBicgUhMZQaboxsqb6jXpjvo48kEojUWaQc=Q7jI4ZEqVMFZIMPoSXTvMebG5fWOUJ6lhDOgWGxiHg8=>
>>
>>
>>
>> On Mon, Oct 21, 2019 at 11:39 AM Durity, Sean R <
>> sean_r_dur...@homedepot.com> wrote:
>>
>> I don’t disagree with Jon, who has all kinds of performance tuning
>> experience. But for ease of operation, we only use G1GC (on Java 8),
>> because the tuning of ParNew+CMS requires a high degree of knowledge and
>> very repeatable testing harnesses. It isn’t worth our time. As a previous
>> writer mentioned, there is usually better return on our time tuning the
>> schema (aka helping developers understand Cassandra’s strengths).
>>
>>
>>
>> We use 16 – 32 GB heaps, nothing smaller than that.
>>
>>
>>
>> Sean Durity
>>
>>
>>
>> *From:* Jon Haddad 
>> *Sent:* Monday, October 21, 2019 10:43 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] Re: GC Tuning
>> https://thelastpickle.com/blog/2018/04/11/gc-tuning.html
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__thelastpickle.com_blog_2018_04_11_gc-2Dtuning.html=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=e9Ahs5XXRBicgUhMZQaboxsqb6jXpjvo48kEojUWaQc=YFRUQ6Rdb5mcFf6GqguRYCsrcAcP6KzjozIgYp56riE=>
>>
>>
>>
>> I still use ParNew + CMS over G1GC with Java 8.  I haven't done a
>> comparison with JDK 11 yet, so I'm not sure if it's any better.  I've heard
>> it is, but I like to verify first.  The pause times with ParNew + CMS are
>> generally lower than G1 when tuned right, but as Chris said it can be
>> tricky.  If you aren't willing to spend the time understanding how it works
>> and why each setting matters, G1 is a better option.
>>
>>
>>
>> I wouldn't run Cassandra in production on less than 8GB of heap - I
>> consider it the absol

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

[Reid Pinchback]

Think of GB to OS as something intended to support file caching.  As such the 
amount is whatever suits your usage.  If your use is almost exclusively 
reading, then file cache memory doesn’t matter that much if you’re operating 
with your storage as those nvme ssd drives that the i3’s come with.  There is 
already a chunk cache that you should be tuning in C* instead, and feeding fast 
from the O/S file cache, assuming compressed SSTables, maybe turns out to be 
less of a concern.

If you have moderate write activity then your situation changes because then 
that same file cache is how your dirty background pages turn into eventual 
flushes to disk, and so you have to watch the impact of read stalls when the 
I/O fills with write requests.  You might not see this so obviously on nvme 
drives, but that could depend a lot on the distro and kernels and how the 
filesystem is mounted.

My super strong advice on issues like this is to not cargo-cult other people’s 
tunings.  Look at them for ideas, sure. But learn how to do your own 
investigations, and budget the time for it into your project.  Budget a LOT of 
time for it if your measure of “good performance” is based on latency; when 
“good” is defined in terms of throughput your life is easier.  Also, everything 
is always a little different in virtualization, and lord knows you can have 
screwball things appear in AWS. The good news is you don’t need a perfect 
configuration out of the gate; you need a configuration you understand and can 
refine; understanding comes from knowing how to do your own performance 
monitoring.


From: Sergio 
Reply-To: "user@cassandra.apache.org" 
Date: Monday, October 21, 2019 at 1:16 PM
To: "user@cassandra.apache.org" 
Subject: Re: [EXTERNAL] Re: GC Tuning 
https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

Message from External Sender
Thanks, guys!
I just copied and paste what I found on our test machines but I can confirm 
that we have the same settings except for 8GB in production.
I didn't select these settings and I need to verify why these settings are 
there.
If any of you want to share your flags for a read-heavy workload it would be 
appreciated, so I would replace and test those flags with TLP-STRESS.
I am thinking about different approaches (G1GC vs ParNew + CMS)
How many GB for RAM do you dedicate to the OS in percentage or in an exact 
number?
Can you share the flags for ParNew + CMS that I can play with it and perform a 
test?

Best,
Sergio

Il giorno lun 21 ott 2019 alle ore 09:27 Reid Pinchback 
mailto:rpinchb...@tripadvisor.com>> ha scritto:
Since the instance size is < 32gb, hopefully swap isn’t being used, so it 
should be moot.

Sergio, also be aware that  -XX:+CMSClassUnloadingEnabled probably doesn’t do 
anything for you.  I believe that only applies to CMS, not G1GC.  I also 
wouldn’t take it as gospel truth that  -XX:+UseNUMA is a good thing on AWS (or 
anything virtualized), you’d have to run your own tests and find out.

R
From: Jon Haddad mailto:j...@jonhaddad.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Monday, October 21, 2019 at 12:06 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: [EXTERNAL] Re: GC Tuning 
https://thelastpickle.com/blog/2018/04/11/gc-tuning.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__thelastpickle.com_blog_2018_04_11_gc-2Dtuning.html=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=UvvIpm6RP7FYRQH6S5EPXTsxAMsezbm6QzHNB0zmMG0=jmk5lyXeQ6gwlVWF86TKWUbIhy57G5tOnlLEps8-DQw=>

Message from External Sender
One thing to note, if you're going to use a big heap, cap it at 31GB, not 32.  
Once you go to 32GB, you don't get to use compressed pointers [1], so you get 
less addressable space than at 31GB.

[1] 
https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/<https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.codecentric.de_en_2014_02_35gb-2Dheap-2Dless-2D32gb-2Djava-2Djvm-2Dmemory-2Doddities_=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=e9Ahs5XXRBicgUhMZQaboxsqb6jXpjvo48kEojUWaQc=Q7jI4ZEqVMFZIMPoSXTvMebG5fWOUJ6lhDOgWGxiHg8=>

On Mon, Oct 21, 2019 at 11:39 AM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
I don’t disagree with Jon, who has all kinds of performance tuning experience. 
But for ease of operation, we only use G1GC (on Java 8), because the tuning of 
ParNew+CMS requires a high degree of knowledge and very repeatable testing 
harnesses. It isn’t worth our time. As a previous writer mentioned, there is 
usually better return on our time tuning the schema (aka helping developers 
understand Cassandra’s strengths).

We use 16 – 32 GB heaps, nothing smaller than that.

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

[Sergio]

Thanks, guys!
I just copied and paste what I found on our test machines but I can confirm
that we have the same settings except for 8GB in production.
I didn't select these settings and I need to verify why these settings are
there.
If any of you want to share your flags for a read-heavy workload it would
be appreciated, so I would replace and test those flags with TLP-STRESS.
I am thinking about different approaches (G1GC vs ParNew + CMS)
How many GB for RAM do you dedicate to the OS in percentage or in an exact
number?
Can you share the flags for ParNew + CMS that I can play with it and
perform a test?

Best,
Sergio


Il giorno lun 21 ott 2019 alle ore 09:27 Reid Pinchback <
rpinchb...@tripadvisor.com> ha scritto:

> Since the instance size is < 32gb, hopefully swap isn’t being used, so it
> should be moot.
>
>
>
> Sergio, also be aware that  -XX:+CMSClassUnloadingEnabled probably
> doesn’t do anything for you.  I believe that only applies to CMS, not
> G1GC.  I also wouldn’t take it as gospel truth that  -XX:+UseNUMA is a good
> thing on AWS (or anything virtualized), you’d have to run your own tests
> and find out.
>
>
>
> R
>
> *From: *Jon Haddad 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Monday, October 21, 2019 at 12:06 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: [EXTERNAL] Re: GC Tuning
> https://thelastpickle.com/blog/2018/04/11/gc-tuning.html
>
>
>
> *Message from External Sender*
>
> One thing to note, if you're going to use a big heap, cap it at 31GB, not
> 32.  Once you go to 32GB, you don't get to use compressed pointers [1], so
> you get less addressable space than at 31GB.
>
>
>
> [1]
> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.codecentric.de_en_2014_02_35gb-2Dheap-2Dless-2D32gb-2Djava-2Djvm-2Dmemory-2Doddities_=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=e9Ahs5XXRBicgUhMZQaboxsqb6jXpjvo48kEojUWaQc=Q7jI4ZEqVMFZIMPoSXTvMebG5fWOUJ6lhDOgWGxiHg8=>
>
>
>
> On Mon, Oct 21, 2019 at 11:39 AM Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
> I don’t disagree with Jon, who has all kinds of performance tuning
> experience. But for ease of operation, we only use G1GC (on Java 8),
> because the tuning of ParNew+CMS requires a high degree of knowledge and
> very repeatable testing harnesses. It isn’t worth our time. As a previous
> writer mentioned, there is usually better return on our time tuning the
> schema (aka helping developers understand Cassandra’s strengths).
>
>
>
> We use 16 – 32 GB heaps, nothing smaller than that.
>
>
>
> Sean Durity
>
>
>
> *From:* Jon Haddad 
> *Sent:* Monday, October 21, 2019 10:43 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: GC Tuning
> https://thelastpickle.com/blog/2018/04/11/gc-tuning.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__thelastpickle.com_blog_2018_04_11_gc-2Dtuning.html=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=e9Ahs5XXRBicgUhMZQaboxsqb6jXpjvo48kEojUWaQc=YFRUQ6Rdb5mcFf6GqguRYCsrcAcP6KzjozIgYp56riE=>
>
>
>
> I still use ParNew + CMS over G1GC with Java 8.  I haven't done a
> comparison with JDK 11 yet, so I'm not sure if it's any better.  I've heard
> it is, but I like to verify first.  The pause times with ParNew + CMS are
> generally lower than G1 when tuned right, but as Chris said it can be
> tricky.  If you aren't willing to spend the time understanding how it works
> and why each setting matters, G1 is a better option.
>
>
>
> I wouldn't run Cassandra in production on less than 8GB of heap - I
> consider it the absolute minimum.  For G1 I'd use 16GB, and never 4GB with
> Cassandra unless you're rarely querying it.
>
>
>
> I typically use the following as a starting point now:
>
>
>
> ParNew + CMS
>
> 16GB heap
>
> 10GB new gen
>
> 2GB memtable cap, otherwise you'll spend a bunch of time copying around
> memtables (cassandra.yaml)
>
> Max tenuring threshold: 2
>
> survivor ratio 6
>
>
>
> I've also done some tests with a 30GB heap, 24 GB of which was new gen.
> This worked surprisingly well in my tests since it essentially keeps
> everything out of the old gen.  New gen allocations are just a pointer bump
> and are pretty fast, so in my (limited) tests of this I was seeing really
> good p99 times.  I was seeing a 200-400 ms pause roughly once a minute
> running a workload that deliberately wasn't hitting a resource limit
> (testing real world looking stress vs overwhelming the cluster).
>
>
>
&g

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

[Reid Pinchback]

Since the instance size is < 32gb, hopefully swap isn’t being used, so it 
should be moot.

Sergio, also be aware that  -XX:+CMSClassUnloadingEnabled probably doesn’t do 
anything for you.  I believe that only applies to CMS, not G1GC.  I also 
wouldn’t take it as gospel truth that  -XX:+UseNUMA is a good thing on AWS (or 
anything virtualized), you’d have to run your own tests and find out.

R

From: Jon Haddad 
Reply-To: "user@cassandra.apache.org" 
Date: Monday, October 21, 2019 at 12:06 PM
To: "user@cassandra.apache.org" 
Subject: Re: [EXTERNAL] Re: GC Tuning 
https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

Message from External Sender
One thing to note, if you're going to use a big heap, cap it at 31GB, not 32.  
Once you go to 32GB, you don't get to use compressed pointers [1], so you get 
less addressable space than at 31GB.

[1] 
https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/<https://urldefense.proofpoint.com/v2/url?u=https-3A__blog.codecentric.de_en_2014_02_35gb-2Dheap-2Dless-2D32gb-2Djava-2Djvm-2Dmemory-2Doddities_=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=e9Ahs5XXRBicgUhMZQaboxsqb6jXpjvo48kEojUWaQc=Q7jI4ZEqVMFZIMPoSXTvMebG5fWOUJ6lhDOgWGxiHg8=>

On Mon, Oct 21, 2019 at 11:39 AM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
I don’t disagree with Jon, who has all kinds of performance tuning experience. 
But for ease of operation, we only use G1GC (on Java 8), because the tuning of 
ParNew+CMS requires a high degree of knowledge and very repeatable testing 
harnesses. It isn’t worth our time. As a previous writer mentioned, there is 
usually better return on our time tuning the schema (aka helping developers 
understand Cassandra’s strengths).

We use 16 – 32 GB heaps, nothing smaller than that.

Sean Durity

From: Jon Haddad mailto:j...@jonhaddad.com>>
Sent: Monday, October 21, 2019 10:43 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Re: GC Tuning 
https://thelastpickle.com/blog/2018/04/11/gc-tuning.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__thelastpickle.com_blog_2018_04_11_gc-2Dtuning.html=DwMFaQ=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc=e9Ahs5XXRBicgUhMZQaboxsqb6jXpjvo48kEojUWaQc=YFRUQ6Rdb5mcFf6GqguRYCsrcAcP6KzjozIgYp56riE=>

I still use ParNew + CMS over G1GC with Java 8.  I haven't done a comparison 
with JDK 11 yet, so I'm not sure if it's any better.  I've heard it is, but I 
like to verify first.  The pause times with ParNew + CMS are generally lower 
than G1 when tuned right, but as Chris said it can be tricky.  If you aren't 
willing to spend the time understanding how it works and why each setting 
matters, G1 is a better option.

I wouldn't run Cassandra in production on less than 8GB of heap - I consider it 
the absolute minimum.  For G1 I'd use 16GB, and never 4GB with Cassandra unless 
you're rarely querying it.

I typically use the following as a starting point now:

ParNew + CMS
16GB heap
10GB new gen
2GB memtable cap, otherwise you'll spend a bunch of time copying around 
memtables (cassandra.yaml)
Max tenuring threshold: 2
survivor ratio 6

I've also done some tests with a 30GB heap, 24 GB of which was new gen.  This 
worked surprisingly well in my tests since it essentially keeps everything out 
of the old gen.  New gen allocations are just a pointer bump and are pretty 
fast, so in my (limited) tests of this I was seeing really good p99 times.  I 
was seeing a 200-400 ms pause roughly once a minute running a workload that 
deliberately wasn't hitting a resource limit (testing real world looking stress 
vs overwhelming the cluster).

We built tlp-cluster [1] and tlp-stress [2] to help figure these things out.

[1] https://thelastpickle.com/tlp-cluster/ 
[thelastpickle.com]<https://urldefense.com/v3/__https:/thelastpickle.com/tlp-cluster/__;!OYIaWQQGbnA!ZhiXAdRaL49J8nBlh0F_5MQ97Z1QNTUuTSMvksmEmxan3d65D6ATmQO1ig58W52u_EmQ1GM$>
[2] http://thelastpickle.com/tlp-stress 
[thelastpickle.com]<https://urldefense.com/v3/__http:/thelastpickle.com/tlp-stress__;!OYIaWQQGbnA!ZhiXAdRaL49J8nBlh0F_5MQ97Z1QNTUuTSMvksmEmxan3d65D6ATmQO1ig58W52uuCUZYKw$>

Jon




On Mon, Oct 21, 2019 at 10:24 AM Reid Pinchback 
mailto:rpinchb...@tripadvisor.com>> wrote:
An i3x large has 30.5 gb of RAM but you’re using less than 4gb for C*.  So 
minus room for other uses of jvm memory and for kernel activity, that’s about 
25 gb for file cache.  You’ll have to see if you either want a bigger heap to 
allow for less frequent gc cycles, or you could save money on the instance 
size.  C* generates a lot of medium-length lifetime objects which can easily 
end up in old gen.  A larger heap will reduce the burn of more old-gen 
collections.  There are no magic numbers to just give because it’ll depend on 
your usage pat

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

[Jon Haddad]

One thing to note, if you're going to use a big heap, cap it at 31GB, not
32.  Once you go to 32GB, you don't get to use compressed pointers [1], so
you get less addressable space than at 31GB.

[1]
https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/

On Mon, Oct 21, 2019 at 11:39 AM Durity, Sean R 
wrote:

> I don’t disagree with Jon, who has all kinds of performance tuning
> experience. But for ease of operation, we only use G1GC (on Java 8),
> because the tuning of ParNew+CMS requires a high degree of knowledge and
> very repeatable testing harnesses. It isn’t worth our time. As a previous
> writer mentioned, there is usually better return on our time tuning the
> schema (aka helping developers understand Cassandra’s strengths).
>
>
>
> We use 16 – 32 GB heaps, nothing smaller than that.
>
>
>
> Sean Durity
>
>
>
> *From:* Jon Haddad 
> *Sent:* Monday, October 21, 2019 10:43 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: GC Tuning
> https://thelastpickle.com/blog/2018/04/11/gc-tuning.html
>
>
>
> I still use ParNew + CMS over G1GC with Java 8.  I haven't done a
> comparison with JDK 11 yet, so I'm not sure if it's any better.  I've heard
> it is, but I like to verify first.  The pause times with ParNew + CMS are
> generally lower than G1 when tuned right, but as Chris said it can be
> tricky.  If you aren't willing to spend the time understanding how it works
> and why each setting matters, G1 is a better option.
>
>
>
> I wouldn't run Cassandra in production on less than 8GB of heap - I
> consider it the absolute minimum.  For G1 I'd use 16GB, and never 4GB with
> Cassandra unless you're rarely querying it.
>
>
>
> I typically use the following as a starting point now:
>
>
>
> ParNew + CMS
>
> 16GB heap
>
> 10GB new gen
>
> 2GB memtable cap, otherwise you'll spend a bunch of time copying around
> memtables (cassandra.yaml)
>
> Max tenuring threshold: 2
>
> survivor ratio 6
>
>
>
> I've also done some tests with a 30GB heap, 24 GB of which was new gen.
> This worked surprisingly well in my tests since it essentially keeps
> everything out of the old gen.  New gen allocations are just a pointer bump
> and are pretty fast, so in my (limited) tests of this I was seeing really
> good p99 times.  I was seeing a 200-400 ms pause roughly once a minute
> running a workload that deliberately wasn't hitting a resource limit
> (testing real world looking stress vs overwhelming the cluster).
>
>
>
> We built tlp-cluster [1] and tlp-stress [2] to help figure these things
> out.
>
>
>
> [1] https://thelastpickle.com/tlp-cluster/ [thelastpickle.com]
> 
>
> [2] http://thelastpickle.com/tlp-stress [thelastpickle.com]
> 
>
>
>
> Jon
>
>
>
>
>
>
>
>
>
> On Mon, Oct 21, 2019 at 10:24 AM Reid Pinchback <
> rpinchb...@tripadvisor.com> wrote:
>
> An i3x large has 30.5 gb of RAM but you’re using less than 4gb for C*.  So
> minus room for other uses of jvm memory and for kernel activity, that’s
> about 25 gb for file cache.  You’ll have to see if you either want a bigger
> heap to allow for less frequent gc cycles, or you could save money on the
> instance size.  C* generates a lot of medium-length lifetime objects which
> can easily end up in old gen.  A larger heap will reduce the burn of more
> old-gen collections.  There are no magic numbers to just give because it’ll
> depend on your usage patterns.
>
>
>
> *From: *Sergio 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Sunday, October 20, 2019 at 2:51 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: GC Tuning 
> https://thelastpickle.com/blog/2018/04/11/gc-tuning.html
> [thelastpickle.com]
> 
>
>
>
> *Message from External Sender*
>
> Thanks for the answer.
>
> This is the JVM version that I have right now.
>
> openjdk version "1.8.0_161"
> OpenJDK Runtime Environment (build 1.8.0_161-b14)
> OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
>
> These are the current flags. Would you change anything in a i3x.large aws
> node?
>
> java -Xloggc:/var/log/cassandra/gc.log
> -Dcassandra.max_queued_native_transport_requests=4096 -ea
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB
> -XX:+UseNUMA -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:+UseG1GC
> -XX:G1RSetUpdatingPauseTimePercent=5 -XX:MaxGCPauseMillis=200
> 

RE: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

[Durity, Sean R]

I don’t disagree with Jon, who has all kinds of performance tuning experience. 
But for ease of operation, we only use G1GC (on Java 8), because the tuning of 
ParNew+CMS requires a high degree of knowledge and very repeatable testing 
harnesses. It isn’t worth our time. As a previous writer mentioned, there is 
usually better return on our time tuning the schema (aka helping developers 
understand Cassandra’s strengths).

We use 16 – 32 GB heaps, nothing smaller than that.

Sean Durity

From: Jon Haddad 
Sent: Monday, October 21, 2019 10:43 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: GC Tuning 
https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

I still use ParNew + CMS over G1GC with Java 8.  I haven't done a comparison 
with JDK 11 yet, so I'm not sure if it's any better.  I've heard it is, but I 
like to verify first.  The pause times with ParNew + CMS are generally lower 
than G1 when tuned right, but as Chris said it can be tricky.  If you aren't 
willing to spend the time understanding how it works and why each setting 
matters, G1 is a better option.

I wouldn't run Cassandra in production on less than 8GB of heap - I consider it 
the absolute minimum.  For G1 I'd use 16GB, and never 4GB with Cassandra unless 
you're rarely querying it.

I typically use the following as a starting point now:

ParNew + CMS
16GB heap
10GB new gen
2GB memtable cap, otherwise you'll spend a bunch of time copying around 
memtables (cassandra.yaml)
Max tenuring threshold: 2
survivor ratio 6

I've also done some tests with a 30GB heap, 24 GB of which was new gen.  This 
worked surprisingly well in my tests since it essentially keeps everything out 
of the old gen.  New gen allocations are just a pointer bump and are pretty 
fast, so in my (limited) tests of this I was seeing really good p99 times.  I 
was seeing a 200-400 ms pause roughly once a minute running a workload that 
deliberately wasn't hitting a resource limit (testing real world looking stress 
vs overwhelming the cluster).

We built tlp-cluster [1] and tlp-stress [2] to help figure these things out.

[1] https://thelastpickle.com/tlp-cluster/ 
[thelastpickle.com]
[2] http://thelastpickle.com/tlp-stress 
[thelastpickle.com]

Jon




On Mon, Oct 21, 2019 at 10:24 AM Reid Pinchback 
mailto:rpinchb...@tripadvisor.com>> wrote:
An i3x large has 30.5 gb of RAM but you’re using less than 4gb for C*.  So 
minus room for other uses of jvm memory and for kernel activity, that’s about 
25 gb for file cache.  You’ll have to see if you either want a bigger heap to 
allow for less frequent gc cycles, or you could save money on the instance 
size.  C* generates a lot of medium-length lifetime objects which can easily 
end up in old gen.  A larger heap will reduce the burn of more old-gen 
collections.  There are no magic numbers to just give because it’ll depend on 
your usage patterns.

From: Sergio mailto:lapostadiser...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Sunday, October 20, 2019 at 2:51 PM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html 
[thelastpickle.com]

Message from External Sender
Thanks for the answer.

This is the JVM version that I have right now.

openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

These are the current flags. Would you change anything in a i3x.large aws node?

java -Xloggc:/var/log/cassandra/gc.log 
-Dcassandra.max_queued_native_transport_requests=4096 -ea 
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 
-XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
-XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB 
-XX:+UseNUMA -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true 
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:+UseG1GC 
-XX:G1RSetUpdatingPauseTimePercent=5 -XX:MaxGCPauseMillis=200 
-XX:InitiatingHeapOccupancyPercent=45 -XX:G1HeapRegionSize=0 
-XX:-ParallelRefProcEnabled -Xms3821M -Xmx3821M 
-XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler 
-Dcom.sun.management.jmxremote.port=7199 
-Dcom.sun.management.jmxremote.rmi.port=7199 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false 

Re: [EXTERNAL] Cassandra Export error in COPY command

[Hossein Ghiyasi Mehr]

The table has more than 10 M rows. I used COPY command in a cluster with
five machine for this table and everything was OK.
I took a backup to a single machine using sstableloader.
Now I want to extract rows using COPY command but I can't!

On Mon, Sep 23, 2019 at 6:30 AM Durity, Sean R 
wrote:

> Copy command tries to export all rows in the table, not just the ones on
> the node. It will eventually timeout if the table is large. It is really
> built for something under 5 million rows or so. Dsbulk (from DataStax) is
> great for this, if you are a customer. Otherwise, you will probably need to
> write an extract of some kind. You can get keys from the sstables, then
> dedupe, then export rows one by one using the keys (kind of painful). How
> large is the table you are trying to export?
>
>
>
> Sean Durity
>
>
>
> *From:* Hossein Ghiyasi Mehr 
> *Sent:* Saturday, September 21, 2019 8:02 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra Export error in COPY command
>
>
>
> Hi all members,
>
> I want to export (pk, another_int_column) from single node using COPY
> command. But after about 1h 45m, I've got a lot of read errors:
>
>
>
>
>
> I tried this action many times but after maximum 2h, it failed with the
> errors.
>
>
>
> Any idea may help me!
>
> Thanks.
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

RE: [EXTERNAL] Cassandra Export error in COPY command

[Durity, Sean R]

Copy command tries to export all rows in the table, not just the ones on the 
node. It will eventually timeout if the table is large. It is really built for 
something under 5 million rows or so. Dsbulk (from DataStax) is great for this, 
if you are a customer. Otherwise, you will probably need to write an extract of 
some kind. You can get keys from the sstables, then dedupe, then export rows 
one by one using the keys (kind of painful). How large is the table you are 
trying to export?

Sean Durity

From: Hossein Ghiyasi Mehr 
Sent: Saturday, September 21, 2019 8:02 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra Export error in COPY command

Hi all members,
I want to export (pk, another_int_column) from single node using COPY command. 
But after about 1h 45m, I've got a lot of read errors:

[cid:image002.png@01D57199.74B91800]

I tried this action many times but after maximum 2h, it failed with the errors.

Any idea may help me!
Thanks.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: [EXTERNAL] Re: loading big amount of data to Cassandra

[Amanda Moran]

With DataStax bulkloader you can only export from a Cassandra table but not 
import into Cassandra (only load into DSE cluster). 

And +1 on the confusing name of batches ... yes it’s for writes but not for 
loading data. 

Amanda 

> On Aug 5, 2019, at 8:14 AM, Durity, Sean R  
> wrote:
> 
> DataStax has a very fast bulk load tool - dsebulk. Not sure if it is 
> available for open source or not. In my experience so far, I am very 
> impressed with it.
> 
> 
> 
> Sean Durity – Staff Systems Engineer, Cassandra
> 
> -Original Message-
> From: p...@xvalheru.org 
> Sent: Saturday, August 3, 2019 6:06 AM
> To: user@cassandra.apache.org
> Cc: Dimo Velev 
> Subject: [EXTERNAL] Re: loading big amount of data to Cassandra
> 
> Thanks to all,
> 
> I'll try the SSTables.
> 
> Thanks
> 
> Pat
> 
>> On 2019-08-03 09:54, Dimo Velev wrote:
>> Check out the CQLSSTableWriter java class -
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_trunk_src_java_org_apache_cassandra_io_sstable_CQLSSTableWriter.java=DwIDaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=0F8VMU_BKNwicZFDQ0Nx54JvvS3MHT92_W1RRwF3deA=F43aPz7NPfAfs5c_oRJQvUiTMJjDmpB_BXAHKhPfW2A=
>> . You use it to generate sstables - you need to write a small program
>> for that. You can then stream them over the network using the
>> sstableloader (either use the utility or use the underlying classes to
>> embed it in your program).
>> 
>>> On 3. Aug 2019, at 07:17, Ayub M  wrote:
>>> 
>>> Dimo, how do you generate sstables? Do you mean load data locally on
>>> a cassandra node and use sstableloader?
>>> 
>>> On Fri, Aug 2, 2019, 5:48 PM Dimo Velev 
>>> wrote:
>>> 
 Hi,
 
 Batches will actually slow down the process because they mean a
 different thing in C* - as you read they are just grouping changes
 together that you want executed atomically.
 
 Cassandra does not really have indices so that is different than a
 relational DB. However, after writing stuff to Cassandra it
 generates many smallish partitions of the data. These are then
 joined in the background together to improve read performance.
 
 You have two options from my experience:
 
 Option 1: use normal CQL api in async mode. This will create a
 high CPU load on your cluster. Depending on whether that is fine
 for you that might be the easiest solution.
 
 Option 2: generate sstables locally and use the sstableloader to
 upload them into the cluster. The streaming does not generate high
 cpu load so it is a viable option for clusters with other
 operational load.
 
 Option 2 scales with the number of cores of the machine generating
 the sstables. If you can split your data you can generate sstables
 on multiple machines. In contrast, option 1 scales with your
 cluster. If you have a large cluster that is idling, it would be
 better to use option 1.
 
 With both options I was able to write at about 50-100K rows / sec
 on my laptop and local Cassandra. The speed heavily depends on the
 size of your rows.
 
 Back to your question — I guess option2 is similar to what you
 are used to from tools like sqlloader for relational DBMSes
 
 I had a requirement of loading a few 100 mio rows per day into an
 operational cluster so I went with option 2 to offload the cpu
 load to reduce impact on the reading side during the loads.
 
 Cheers,
 Dimo
 
 Sent from my iPad
 
> On 2. Aug 2019, at 18:59, p...@xvalheru.org wrote:
> 
> Hi,
> 
> I need to upload to Cassandra about 7 billions of records. What
 is the best setup of Cassandra for this task? Will usage of batch
 speeds up the upload (I've read somewhere that batch in Cassandra
 is dedicated to atomicity not to speeding up communication)? How
 Cassandra internally works related to indexing? In SQL databases
 when uploading such amount of data is suggested to turn off
 indexing and then turn on. Is something simmillar possible in
 Cassandra?
> 
> Thanks for all suggestions.
> 
> Pat
> 
> 
> Freehosting PIPNI - 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pipni.cz_=DwIDaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=0F8VMU_BKNwicZFDQ0Nx54JvvS3MHT92_W1RRwF3deA=nccgCDZwHe3qri11l3VV1if5GR1iqcWR5gjf6-J1C5U=
> 
> 
> 
 
>>> 
>> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 
 
 
>>> 
>> -
 To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
>> 

Re: [EXTERNAL] Re: loading big amount of data to Cassandra

[Hiroyuki Yamada]

cassandra-loader is also useful because you don't need to create sstables.
https://github.com/brianmhess/cassandra-loader

Hiro

On Tue, Aug 6, 2019 at 12:15 AM Durity, Sean R
 wrote:
>
> DataStax has a very fast bulk load tool - dsebulk. Not sure if it is 
> available for open source or not. In my experience so far, I am very 
> impressed with it.
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
> -Original Message-
> From: p...@xvalheru.org 
> Sent: Saturday, August 3, 2019 6:06 AM
> To: user@cassandra.apache.org
> Cc: Dimo Velev 
> Subject: [EXTERNAL] Re: loading big amount of data to Cassandra
>
> Thanks to all,
>
> I'll try the SSTables.
>
> Thanks
>
> Pat
>
> On 2019-08-03 09:54, Dimo Velev wrote:
> > Check out the CQLSSTableWriter java class -
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_trunk_src_java_org_apache_cassandra_io_sstable_CQLSSTableWriter.java=DwIDaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=0F8VMU_BKNwicZFDQ0Nx54JvvS3MHT92_W1RRwF3deA=F43aPz7NPfAfs5c_oRJQvUiTMJjDmpB_BXAHKhPfW2A=
> > . You use it to generate sstables - you need to write a small program
> > for that. You can then stream them over the network using the
> > sstableloader (either use the utility or use the underlying classes to
> > embed it in your program).
> >
> > On 3. Aug 2019, at 07:17, Ayub M  wrote:
> >
> >> Dimo, how do you generate sstables? Do you mean load data locally on
> >> a cassandra node and use sstableloader?
> >>
> >> On Fri, Aug 2, 2019, 5:48 PM Dimo Velev 
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Batches will actually slow down the process because they mean a
> >>> different thing in C* - as you read they are just grouping changes
> >>> together that you want executed atomically.
> >>>
> >>> Cassandra does not really have indices so that is different than a
> >>> relational DB. However, after writing stuff to Cassandra it
> >>> generates many smallish partitions of the data. These are then
> >>> joined in the background together to improve read performance.
> >>>
> >>> You have two options from my experience:
> >>>
> >>> Option 1: use normal CQL api in async mode. This will create a
> >>> high CPU load on your cluster. Depending on whether that is fine
> >>> for you that might be the easiest solution.
> >>>
> >>> Option 2: generate sstables locally and use the sstableloader to
> >>> upload them into the cluster. The streaming does not generate high
> >>> cpu load so it is a viable option for clusters with other
> >>> operational load.
> >>>
> >>> Option 2 scales with the number of cores of the machine generating
> >>> the sstables. If you can split your data you can generate sstables
> >>> on multiple machines. In contrast, option 1 scales with your
> >>> cluster. If you have a large cluster that is idling, it would be
> >>> better to use option 1.
> >>>
> >>> With both options I was able to write at about 50-100K rows / sec
> >>> on my laptop and local Cassandra. The speed heavily depends on the
> >>> size of your rows.
> >>>
> >>> Back to your question — I guess option2 is similar to what you
> >>> are used to from tools like sqlloader for relational DBMSes
> >>>
> >>> I had a requirement of loading a few 100 mio rows per day into an
> >>> operational cluster so I went with option 2 to offload the cpu
> >>> load to reduce impact on the reading side during the loads.
> >>>
> >>> Cheers,
> >>> Dimo
> >>>
> >>> Sent from my iPad
> >>>
>  On 2. Aug 2019, at 18:59, p...@xvalheru.org wrote:
> 
>  Hi,
> 
>  I need to upload to Cassandra about 7 billions of records. What
> >>> is the best setup of Cassandra for this task? Will usage of batch
> >>> speeds up the upload (I've read somewhere that batch in Cassandra
> >>> is dedicated to atomicity not to speeding up communication)? How
> >>> Cassandra internally works related to indexing? In SQL databases
> >>> when uploading such amount of data is suggested to turn off
> >>> indexing and then turn on. Is something simmillar possible in
> >>> Cassandra?
> 
>  Thanks for all suggestions.
> 
>  Pat
> 
>  
>  Freehosting PIPNI - 
>  https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pipni.cz_=DwIDaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=0F8VMU_BKNwicZFDQ0Nx54JvvS3MHT92_W1RRwF3deA=nccgCDZwHe3qri11l3VV1if5GR1iqcWR5gjf6-J1C5U=
> 
> 
> 
> >>>
> >>
> > -
>  To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>  For additional commands, e-mail: user-h...@cassandra.apache.org
> 
> >>>
> >>>
> >>
> > -
> >>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: user-h...@cassandra.apache.org
> >
> > 

RE: [EXTERNAL] Re: loading big amount of data to Cassandra

[Durity, Sean R]

DataStax has a very fast bulk load tool - dsebulk. Not sure if it is available 
for open source or not. In my experience so far, I am very impressed with it.



Sean Durity – Staff Systems Engineer, Cassandra

-Original Message-
From: p...@xvalheru.org 
Sent: Saturday, August 3, 2019 6:06 AM
To: user@cassandra.apache.org
Cc: Dimo Velev 
Subject: [EXTERNAL] Re: loading big amount of data to Cassandra

Thanks to all,

I'll try the SSTables.

Thanks

Pat

On 2019-08-03 09:54, Dimo Velev wrote:
> Check out the CQLSSTableWriter java class -
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_trunk_src_java_org_apache_cassandra_io_sstable_CQLSSTableWriter.java=DwIDaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=0F8VMU_BKNwicZFDQ0Nx54JvvS3MHT92_W1RRwF3deA=F43aPz7NPfAfs5c_oRJQvUiTMJjDmpB_BXAHKhPfW2A=
> . You use it to generate sstables - you need to write a small program
> for that. You can then stream them over the network using the
> sstableloader (either use the utility or use the underlying classes to
> embed it in your program).
>
> On 3. Aug 2019, at 07:17, Ayub M  wrote:
>
>> Dimo, how do you generate sstables? Do you mean load data locally on
>> a cassandra node and use sstableloader?
>>
>> On Fri, Aug 2, 2019, 5:48 PM Dimo Velev 
>> wrote:
>>
>>> Hi,
>>>
>>> Batches will actually slow down the process because they mean a
>>> different thing in C* - as you read they are just grouping changes
>>> together that you want executed atomically.
>>>
>>> Cassandra does not really have indices so that is different than a
>>> relational DB. However, after writing stuff to Cassandra it
>>> generates many smallish partitions of the data. These are then
>>> joined in the background together to improve read performance.
>>>
>>> You have two options from my experience:
>>>
>>> Option 1: use normal CQL api in async mode. This will create a
>>> high CPU load on your cluster. Depending on whether that is fine
>>> for you that might be the easiest solution.
>>>
>>> Option 2: generate sstables locally and use the sstableloader to
>>> upload them into the cluster. The streaming does not generate high
>>> cpu load so it is a viable option for clusters with other
>>> operational load.
>>>
>>> Option 2 scales with the number of cores of the machine generating
>>> the sstables. If you can split your data you can generate sstables
>>> on multiple machines. In contrast, option 1 scales with your
>>> cluster. If you have a large cluster that is idling, it would be
>>> better to use option 1.
>>>
>>> With both options I was able to write at about 50-100K rows / sec
>>> on my laptop and local Cassandra. The speed heavily depends on the
>>> size of your rows.
>>>
>>> Back to your question — I guess option2 is similar to what you
>>> are used to from tools like sqlloader for relational DBMSes
>>>
>>> I had a requirement of loading a few 100 mio rows per day into an
>>> operational cluster so I went with option 2 to offload the cpu
>>> load to reduce impact on the reading side during the loads.
>>>
>>> Cheers,
>>> Dimo
>>>
>>> Sent from my iPad
>>>
 On 2. Aug 2019, at 18:59, p...@xvalheru.org wrote:

 Hi,

 I need to upload to Cassandra about 7 billions of records. What
>>> is the best setup of Cassandra for this task? Will usage of batch
>>> speeds up the upload (I've read somewhere that batch in Cassandra
>>> is dedicated to atomicity not to speeding up communication)? How
>>> Cassandra internally works related to indexing? In SQL databases
>>> when uploading such amount of data is suggested to turn off
>>> indexing and then turn on. Is something simmillar possible in
>>> Cassandra?

 Thanks for all suggestions.

 Pat

 
 Freehosting PIPNI - 
 https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pipni.cz_=DwIDaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=0F8VMU_BKNwicZFDQ0Nx54JvvS3MHT92_W1RRwF3deA=nccgCDZwHe3qri11l3VV1if5GR1iqcWR5gjf6-J1C5U=



>>>
>>
> -
 To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: user-h...@cassandra.apache.org

>>>
>>>
>>
> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>
> ---
>
> Freehosting PIPNI - 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pipni.cz_=DwIDaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=0F8VMU_BKNwicZFDQ0Nx54JvvS3MHT92_W1RRwF3deA=nccgCDZwHe3qri11l3VV1if5GR1iqcWR5gjf6-J1C5U=


Freehosting PIPNI - 

Re: [EXTERNAL] Apache Cassandra upgrade path

[Jai Bheemsen Rao Dhanwada]

Thank you Romain

On Sat, Jul 27, 2019 at 1:42 AM Romain Hardouin 
wrote:

> Hi,
>
> Here are some upgrade options:
>   - Standard rolling upgrade: node by node
>
>   - Fast rolling upgrade: rack by rack.
> If clients use CL=LOCAL_ONE then it's OK as long as one rack is UP.
> For higher CL it's possible assuming you have no more than one replica per
> rack e.g. CL=LOCAL_QUORUM with RF=3 and 2 racks is a *BAD* setup. But RF=3
> with 3 rack is OK.
>   - Double write in another cluster: easy for short TTL data (e.g. TTL of
> few days)
> When possible, this option is not only the safest but also allows major
> change (e.g. Partitioner for legacy clusters).
> And of course it's a good opportunity to use new cloud instance type,
> change number of vnodes, etc.
>
> As Sean said, it's not possible for C* servers to stream data with other
> versions when Streaming versions are different. There is no workaround.
> You can check that here
> https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/streaming/messages/StreamMessage.java#L35
> The community plans to work on this limitation to make streaming possible
> between different major versions starting from C*4.x
>
> Last but not least, don't forget to take snapshots (+ backup) and to
> prepare a rollback script.
> System keyspace will be automatically snapshotted by Cassandra when the
> new version will start: the rollback script should be based on that
> snapshot for the system part.
> New data (both commitlog and sstables flushed in 3.11 format) will be lost
> even with such a script but it's useful to test it and to have it ready for
> the D day.
> (See also snapshot_before_compaction setting but it might be useless
> depending on your procedure.)
>
> Romain
>
>
>
> Le vendredi 26 juillet 2019 à 23:51:52 UTC+2, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> a écrit :
>
>
> yes correct, it doesn't work for the servers. trying to see if any had any
> workaround for this issue? (may be changing the protocol version during the
> upgrade time?)
>
> On Fri, Jul 26, 2019 at 1:11 PM Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
> This would handle client protocol, but not streaming protocol between
> nodes.
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Alok Dwivedi 
> *Sent:* Friday, July 26, 2019 3:21 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: [EXTERNAL] Apache Cassandra upgrade path
>
>
>
> Hi Sean
>
> The recommended practice for upgrade is to explicitly control protocol
> version in your application during upgrade process. Basically the protocol
> version is negotiated on first connection and based on chance it can talk
> to an already upgraded node first which means it will negotiate a higher
> version that will not be compatible with those nodes which are still one
> lower Cassandra version. So initially you set it a lower version that is
> like lower common denominator for mixed mode cluster and then remove the
> call to explicit setting once upgrade has completed.
>
>
>
> Cluster cluster = Cluster.builder()
>
> .addContactPoint("127.0.0.1")
>
> .withProtocolVersion(ProtocolVersion.V2)
>
> .build();
>
>
>
> Refer here for more information if using Java driver
>
>
> https://docs.datastax.com/en/developer/java-driver/3.7/manual/native_protocol/#protocol-version-with-mixed-clusters
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_developer_java-2Ddriver_3.7_manual_native-5Fprotocol_-23protocol-2Dversion-2Dwith-2Dmixed-2Dclusters=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=JUUAJpaOGj5fhLX2uWOwUVqUcHN3c24hEaDC1T8RZVQ=WLqlcmEjAYjj7TAAmvYA3NyPqe7ZqgFTNuRNZXryUQE=>
>
>
>
> Same thing applies to drivers in other languages.
>
>
>
> Thanks
>
> Alok Dwivedi
>
> Senior Consultant
>
> https://www.instaclustr.com/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=JUUAJpaOGj5fhLX2uWOwUVqUcHN3c24hEaDC1T8RZVQ=gQuE9u1lRiSA9uZsshvcKIuYih5Rvz3v6lhUOLZzvw4=>
>
>
>
>
>
> On Fri, 26 Jul 2019 at 20:03, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Thanks Sean,
>
>
>
> In my use case all my clusters are multi DC, and I am trying my best
> effort to upgrade ASAP, however there is a chance since all machines are
> VMs. Also my key spaces are not uniform across DCs. some are replicated to
> all DCs and some of them are just one DC, so I am worried there.
>
>
>
> Is there a way

Re: [EXTERNAL] Apache Cassandra upgrade path

[Romain Hardouin]

 Hi,
Here are some upgrade options:  - Standard rolling upgrade: node by node    - 
Fast rolling upgrade: rack by rack.  If clients use CL=LOCAL_ONE then it's OK 
as long as one rack is UP. For higher CL it's possible assuming you have no 
more than one replica per rack e.g. CL=LOCAL_QUORUM with RF=3 and 2 racks is a 
*BAD* setup. But RF=3 with 3 rack is OK.   - Double write in another cluster: 
easy for short TTL data (e.g. TTL of few days) When possible, this option is 
not only the safest but also allows major change (e.g. Partitioner for legacy 
clusters). And of course it's a good opportunity to use new cloud instance 
type, change number of vnodes, etc.
As Sean said, it's not possible for C* servers to stream data with other 
versions when Streaming versions are different. There is no workaround.You can 
check that here 
https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/streaming/messages/StreamMessage.java#L35The
 community plans to work on this limitation to make streaming possible between 
different major versions starting from C*4.x
Last but not least, don't forget to take snapshots (+ backup) and to prepare a 
rollback script.System keyspace will be automatically snapshotted by Cassandra 
when the new version will start: the rollback script should be based on that 
snapshot for the system part.New data (both commitlog and sstables flushed in 
3.11 format) will be lost even with such a script but it's useful to test it 
and to have it ready for the D day.(See also snapshot_before_compaction setting 
but it might be useless depending on your procedure.)
Romain


Le vendredi 26 juillet 2019 à 23:51:52 UTC+2, Jai Bheemsen Rao Dhanwada 
 a écrit :  
 
 yes correct, it doesn't work for the servers. trying to see if any had any 
workaround for this issue? (may be changing the protocol version during the 
upgrade time?)

On Fri, Jul 26, 2019 at 1:11 PM Durity, Sean R  
wrote:


This would handle client protocol, but not streaming protocol between nodes.

 

 

Sean Durity – Staff Systems Engineer, Cassandra

 

From: Alok Dwivedi  
Sent: Friday, July 26, 2019 3:21 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Apache Cassandra upgrade path

 

Hi Sean

The recommended practice for upgrade is to explicitly control protocol version 
in your application during upgrade process. Basically the protocol version is 
negotiated on first connection and based on chance it can talk to an already 
upgraded node first which means it will negotiate a higher version that will 
not be compatible with those nodes which are still one lower Cassandra version. 
So initially you set it a lower version that is like lower common denominator 
for mixed mode cluster and then remove the call to explicit setting once 
upgrade has completed. 

 

Clustercluster= Cluster.builder()

   .addContactPoint("127.0.0.1")

   .withProtocolVersion(ProtocolVersion.V2)

   .build();

 

Refer here for more information if using Java driver

https://docs.datastax.com/en/developer/java-driver/3.7/manual/native_protocol/#protocol-version-with-mixed-clusters

 

Same thing applies to drivers in other languages. 

 

Thanks

Alok Dwivedi

Senior Consultant 

https://www.instaclustr.com/

 

 

On Fri, 26 Jul 2019 at 20:03, Jai Bheemsen Rao Dhanwada  
wrote:


Thanks Sean,

 

In my use case all my clusters are multi DC, and I am trying my best effort to 
upgrade ASAP, however there is a chance since all machines are VMs. Also my key 
spaces are not uniform across DCs. some are replicated to all DCs and some of 
them are just one DC, so I am worried there.

 

Is there a way to override the protocol version until the upgrade is done and 
then change it back once the upgrade is completed?

 

On Fri, Jul 26, 2019 at 11:42 AM Durity, Sean R  
wrote:


What you have seen is totally expected. You can’t stream between different 
major versions of Cassandra. Get the upgrade done, then worry about any down 
hardware. If you are using DCs, upgrade one DC at a time, so that there is an 
available environment in case of any disasters.

 

My advice, though, is to get through the rolling upgrade process as quickly as 
possible. Don’t stay in a mixed state very long. The cluster will function fine 
in a mixed state – except for those streaming operations. No repairs, no 
bootstraps. 

 

 

Sean Durity – Staff Systems Engineer, Cassandra

 

From: Jai Bheemsen Rao Dhanwada 
Sent: Friday, July 26, 2019 2:24 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Apache Cassandra upgrade path

 

Hello,

 

I am trying to upgrade Apache Cassandra from 2.1.16 to 3.11.3, the regular 
rolling upgrade process works fine without any issues.

 

However, I am running into an issue where if there is a node with older version 
dies (hardware failure) and a new node comes up and tries to bootstrap, it's 
failing.

 

I tried two combinations:

 

1. Joining replacement node with 2.1.16 version of cassandra 

In

Re: [EXTERNAL] Apache Cassandra upgrade path

[Jai Bheemsen Rao Dhanwada]

yes correct, it doesn't work for the servers. trying to see if any had any
workaround for this issue? (may be changing the protocol version during the
upgrade time?)

On Fri, Jul 26, 2019 at 1:11 PM Durity, Sean R 
wrote:

> This would handle client protocol, but not streaming protocol between
> nodes.
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Alok Dwivedi 
> *Sent:* Friday, July 26, 2019 3:21 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: [EXTERNAL] Apache Cassandra upgrade path
>
>
>
> Hi Sean
>
> The recommended practice for upgrade is to explicitly control protocol
> version in your application during upgrade process. Basically the protocol
> version is negotiated on first connection and based on chance it can talk
> to an already upgraded node first which means it will negotiate a higher
> version that will not be compatible with those nodes which are still one
> lower Cassandra version. So initially you set it a lower version that is
> like lower common denominator for mixed mode cluster and then remove the
> call to explicit setting once upgrade has completed.
>
>
>
> Cluster cluster = Cluster.builder()
>
> .addContactPoint("127.0.0.1")
>
> .withProtocolVersion(ProtocolVersion.V2)
>
> .build();
>
>
>
> Refer here for more information if using Java driver
>
>
> https://docs.datastax.com/en/developer/java-driver/3.7/manual/native_protocol/#protocol-version-with-mixed-clusters
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_developer_java-2Ddriver_3.7_manual_native-5Fprotocol_-23protocol-2Dversion-2Dwith-2Dmixed-2Dclusters=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=JUUAJpaOGj5fhLX2uWOwUVqUcHN3c24hEaDC1T8RZVQ=WLqlcmEjAYjj7TAAmvYA3NyPqe7ZqgFTNuRNZXryUQE=>
>
>
>
> Same thing applies to drivers in other languages.
>
>
>
> Thanks
>
> Alok Dwivedi
>
> Senior Consultant
>
> https://www.instaclustr.com/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=JUUAJpaOGj5fhLX2uWOwUVqUcHN3c24hEaDC1T8RZVQ=gQuE9u1lRiSA9uZsshvcKIuYih5Rvz3v6lhUOLZzvw4=>
>
>
>
>
>
> On Fri, 26 Jul 2019 at 20:03, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Thanks Sean,
>
>
>
> In my use case all my clusters are multi DC, and I am trying my best
> effort to upgrade ASAP, however there is a chance since all machines are
> VMs. Also my key spaces are not uniform across DCs. some are replicated to
> all DCs and some of them are just one DC, so I am worried there.
>
>
>
> Is there a way to override the protocol version until the upgrade is done
> and then change it back once the upgrade is completed?
>
>
>
> On Fri, Jul 26, 2019 at 11:42 AM Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
> What you have seen is totally expected. You can’t stream between different
> major versions of Cassandra. Get the upgrade done, then worry about any
> down hardware. If you are using DCs, upgrade one DC at a time, so that
> there is an available environment in case of any disasters.
>
>
>
> My advice, though, is to get through the rolling upgrade process as
> quickly as possible. Don’t stay in a mixed state very long. The cluster
> will function fine in a mixed state – except for those streaming
> operations. No repairs, no bootstraps.
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Jai Bheemsen Rao Dhanwada 
> *Sent:* Friday, July 26, 2019 2:24 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Apache Cassandra upgrade path
>
>
>
> Hello,
>
>
>
> I am trying to upgrade Apache Cassandra from 2.1.16 to 3.11.3, the regular
> rolling upgrade process works fine without any issues.
>
>
>
> However, I am running into an issue where if there is a node with older
> version dies (hardware failure) and a new node comes up and tries to
> bootstrap, it's failing.
>
>
>
> I tried two combinations:
>
>
>
> 1. Joining replacement node with 2.1.16 version of cassandra
>
> In this case nodes with 2.1.16 version are able to stream data to the new
> node, but the nodes with 3.11.3 version are failing with the below error.
>
>
>
> ERROR [STREAM-INIT-/10.x.x.x:40296] 2019-07-26 17:45:17,775
> IncomingStreamingConnection.java:80 - Error while reading from socket from
> /10.y.y.y:40296.
> java.io.IOException: Received stream using protocol version 2 (my version
> 4). Terminating connection
>
> 2. Join

RE: [EXTERNAL] Apache Cassandra upgrade path

[Durity, Sean R]

This would handle client protocol, but not streaming protocol between nodes.


Sean Durity – Staff Systems Engineer, Cassandra

From: Alok Dwivedi 
Sent: Friday, July 26, 2019 3:21 PM
To: user@cassandra.apache.org
Subject: Re: [EXTERNAL] Apache Cassandra upgrade path

Hi Sean
The recommended practice for upgrade is to explicitly control protocol version 
in your application during upgrade process. Basically the protocol version is 
negotiated on first connection and based on chance it can talk to an already 
upgraded node first which means it will negotiate a higher version that will 
not be compatible with those nodes which are still one lower Cassandra version. 
So initially you set it a lower version that is like lower common denominator 
for mixed mode cluster and then remove the call to explicit setting once 
upgrade has completed.

Cluster cluster = Cluster.builder()
.addContactPoint("127.0.0.1")
.withProtocolVersion(ProtocolVersion.V2)
.build();

Refer here for more information if using Java driver
https://docs.datastax.com/en/developer/java-driver/3.7/manual/native_protocol/#protocol-version-with-mixed-clusters<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_developer_java-2Ddriver_3.7_manual_native-5Fprotocol_-23protocol-2Dversion-2Dwith-2Dmixed-2Dclusters=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=JUUAJpaOGj5fhLX2uWOwUVqUcHN3c24hEaDC1T8RZVQ=WLqlcmEjAYjj7TAAmvYA3NyPqe7ZqgFTNuRNZXryUQE=>

Same thing applies to drivers in other languages.

Thanks
Alok Dwivedi
Senior Consultant
https://www.instaclustr.com/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_=DwMFaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=JUUAJpaOGj5fhLX2uWOwUVqUcHN3c24hEaDC1T8RZVQ=gQuE9u1lRiSA9uZsshvcKIuYih5Rvz3v6lhUOLZzvw4=>


On Fri, 26 Jul 2019 at 20:03, Jai Bheemsen Rao Dhanwada 
mailto:jaibheem...@gmail.com>> wrote:
Thanks Sean,

In my use case all my clusters are multi DC, and I am trying my best effort to 
upgrade ASAP, however there is a chance since all machines are VMs. Also my key 
spaces are not uniform across DCs. some are replicated to all DCs and some of 
them are just one DC, so I am worried there.

Is there a way to override the protocol version until the upgrade is done and 
then change it back once the upgrade is completed?

On Fri, Jul 26, 2019 at 11:42 AM Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:
What you have seen is totally expected. You can’t stream between different 
major versions of Cassandra. Get the upgrade done, then worry about any down 
hardware. If you are using DCs, upgrade one DC at a time, so that there is an 
available environment in case of any disasters.

My advice, though, is to get through the rolling upgrade process as quickly as 
possible. Don’t stay in a mixed state very long. The cluster will function fine 
in a mixed state – except for those streaming operations. No repairs, no 
bootstraps.


Sean Durity – Staff Systems Engineer, Cassandra

From: Jai Bheemsen Rao Dhanwada 
mailto:jaibheem...@gmail.com>>
Sent: Friday, July 26, 2019 2:24 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] Apache Cassandra upgrade path

Hello,

I am trying to upgrade Apache Cassandra from 2.1.16 to 3.11.3, the regular 
rolling upgrade process works fine without any issues.

However, I am running into an issue where if there is a node with older version 
dies (hardware failure) and a new node comes up and tries to bootstrap, it's 
failing.

I tried two combinations:

1. Joining replacement node with 2.1.16 version of cassandra
In this case nodes with 2.1.16 version are able to stream data to the new node, 
but the nodes with 3.11.3 version are failing with the below error.

ERROR [STREAM-INIT-/10.x.x.x:40296] 2019-07-26 17:45:17,775 
IncomingStreamingConnection.java:80 - Error while reading from socket from 
/10.y.y.y:40296.
java.io.IOException: Received stream using protocol version 2 (my version 4). 
Terminating connection
2. Joining replacement node with 3.11.3 version of cassandra
In this case the nodes with 3.11.3 version of cassandra are able to stream the 
data but it's not able to stream data from the 2.1.16 nodes and failing with 
the below error.

ERROR [STREAM-IN-/10.z.z.z:7000] 2019-07-26 18:08:10,380 StreamSession.java:593 
- [Stream #538c6900-afd0-11e9-a649-ab2e045ee53b] Streaming error occurred on 
session with peer 10.z.z.z
java.io.IOException: Connection reset by peer
   at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_151]
   at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) 
~[na:1.8.0_151]
   at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) 
~[na:1.8.0_151]
   at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_151]
   at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) 
~[na:1.8.

Re: [EXTERNAL] Apache Cassandra upgrade path

[Alok Dwivedi]

Hi Sean
The recommended practice for upgrade is to explicitly control protocol
version in your application during upgrade process. Basically the protocol
version is negotiated on first connection and based on chance it can talk
to an already upgraded node first which means it will negotiate a higher
version that will not be compatible with those nodes which are still one
lower Cassandra version. So initially you set it a lower version that is
like lower common denominator for mixed mode cluster and then remove the
call to explicit setting once upgrade has completed.

Cluster cluster = Cluster.builder() .addContactPoint("127.0.0.1") .
withProtocolVersion(ProtocolVersion.V2) .build();

Refer here for more information if using Java driver
https://docs.datastax.com/en/developer/java-driver/3.7/manual/native_protocol/#protocol-version-with-mixed-clusters

Same thing applies to drivers in other languages.

Thanks
Alok Dwivedi
Senior Consultant
https://www.instaclustr.com/


On Fri, 26 Jul 2019 at 20:03, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Thanks Sean,
>
> In my use case all my clusters are multi DC, and I am trying my best
> effort to upgrade ASAP, however there is a chance since all machines are
> VMs. Also my key spaces are not uniform across DCs. some are replicated to
> all DCs and some of them are just one DC, so I am worried there.
>
> Is there a way to override the protocol version until the upgrade is done
> and then change it back once the upgrade is completed?
>
> On Fri, Jul 26, 2019 at 11:42 AM Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
>> What you have seen is totally expected. You can’t stream between
>> different major versions of Cassandra. Get the upgrade done, then worry
>> about any down hardware. If you are using DCs, upgrade one DC at a time, so
>> that there is an available environment in case of any disasters.
>>
>>
>>
>> My advice, though, is to get through the rolling upgrade process as
>> quickly as possible. Don’t stay in a mixed state very long. The cluster
>> will function fine in a mixed state – except for those streaming
>> operations. No repairs, no bootstraps.
>>
>>
>>
>>
>>
>> Sean Durity – Staff Systems Engineer, Cassandra
>>
>>
>>
>> *From:* Jai Bheemsen Rao Dhanwada 
>> *Sent:* Friday, July 26, 2019 2:24 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] Apache Cassandra upgrade path
>>
>>
>>
>> Hello,
>>
>>
>>
>> I am trying to upgrade Apache Cassandra from 2.1.16 to 3.11.3, the
>> regular rolling upgrade process works fine without any issues.
>>
>>
>>
>> However, I am running into an issue where if there is a node with older
>> version dies (hardware failure) and a new node comes up and tries to
>> bootstrap, it's failing.
>>
>>
>>
>> I tried two combinations:
>>
>>
>>
>> 1. Joining replacement node with 2.1.16 version of cassandra
>>
>> In this case nodes with 2.1.16 version are able to stream data to the new
>> node, but the nodes with 3.11.3 version are failing with the below error.
>>
>>
>>
>> ERROR [STREAM-INIT-/10.x.x.x:40296] 2019-07-26 17:45:17,775
>> IncomingStreamingConnection.java:80 - Error while reading from socket from
>> /10.y.y.y:40296.
>> java.io.IOException: Received stream using protocol version 2 (my version
>> 4). Terminating connection
>>
>> 2. Joining replacement node with 3.11.3 version of cassandra
>>
>> In this case the nodes with 3.11.3 version of cassandra are able to
>> stream the data but it's not able to stream data from the 2.1.16 nodes and
>> failing with the below error.
>>
>>
>>
>> ERROR [STREAM-IN-/10.z.z.z:7000] 2019-07-26 18:08:10,380
>> StreamSession.java:593 - [Stream #538c6900-afd0-11e9-a649-ab2e045ee53b]
>> Streaming error occurred on session with peer 10.z.z.z
>> java.io.IOException: Connection reset by peer
>>at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> ~[na:1.8.0_151]
>>at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> ~[na:1.8.0_151]
>>at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> ~[na:1.8.0_151]
>>at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_151]
>>at
>> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>> ~[na:1.8.0_151]
>>at
>> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
>> ~[na:1.8.0_151]
>>at
>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>> ~[na:1.8.0_151]
>>at
>> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>> ~[na:1.8.0_151]
>>at
>> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56)
>> ~[apache-cassandra-3.11.3.jar:3.11.3]
>>at
>> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:311)
>> ~[apache-cassandra-3.11.3.jar:3.11.3]
>>at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
>>
>>
>>
>> Note: In both cases I 

Re: [EXTERNAL] Apache Cassandra upgrade path

[Jai Bheemsen Rao Dhanwada]

Thanks Sean,

In my use case all my clusters are multi DC, and I am trying my best effort
to upgrade ASAP, however there is a chance since all machines are VMs. Also
my key spaces are not uniform across DCs. some are replicated to all DCs
and some of them are just one DC, so I am worried there.

Is there a way to override the protocol version until the upgrade is done
and then change it back once the upgrade is completed?

On Fri, Jul 26, 2019 at 11:42 AM Durity, Sean R 
wrote:

> What you have seen is totally expected. You can’t stream between different
> major versions of Cassandra. Get the upgrade done, then worry about any
> down hardware. If you are using DCs, upgrade one DC at a time, so that
> there is an available environment in case of any disasters.
>
>
>
> My advice, though, is to get through the rolling upgrade process as
> quickly as possible. Don’t stay in a mixed state very long. The cluster
> will function fine in a mixed state – except for those streaming
> operations. No repairs, no bootstraps.
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Jai Bheemsen Rao Dhanwada 
> *Sent:* Friday, July 26, 2019 2:24 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Apache Cassandra upgrade path
>
>
>
> Hello,
>
>
>
> I am trying to upgrade Apache Cassandra from 2.1.16 to 3.11.3, the regular
> rolling upgrade process works fine without any issues.
>
>
>
> However, I am running into an issue where if there is a node with older
> version dies (hardware failure) and a new node comes up and tries to
> bootstrap, it's failing.
>
>
>
> I tried two combinations:
>
>
>
> 1. Joining replacement node with 2.1.16 version of cassandra
>
> In this case nodes with 2.1.16 version are able to stream data to the new
> node, but the nodes with 3.11.3 version are failing with the below error.
>
>
>
> ERROR [STREAM-INIT-/10.x.x.x:40296] 2019-07-26 17:45:17,775
> IncomingStreamingConnection.java:80 - Error while reading from socket from
> /10.y.y.y:40296.
> java.io.IOException: Received stream using protocol version 2 (my version
> 4). Terminating connection
>
> 2. Joining replacement node with 3.11.3 version of cassandra
>
> In this case the nodes with 3.11.3 version of cassandra are able to stream
> the data but it's not able to stream data from the 2.1.16 nodes and failing
> with the below error.
>
>
>
> ERROR [STREAM-IN-/10.z.z.z:7000] 2019-07-26 18:08:10,380
> StreamSession.java:593 - [Stream #538c6900-afd0-11e9-a649-ab2e045ee53b]
> Streaming error occurred on session with peer 10.z.z.z
> java.io.IOException: Connection reset by peer
>at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> ~[na:1.8.0_151]
>at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> ~[na:1.8.0_151]
>at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> ~[na:1.8.0_151]
>at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_151]
>at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> ~[na:1.8.0_151]
>at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
> ~[na:1.8.0_151]
>at
> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_151]
>at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_151]
>at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56)
> ~[apache-cassandra-3.11.3.jar:3.11.3]
>at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:311)
> ~[apache-cassandra-3.11.3.jar:3.11.3]
>at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
>
>
>
> Note: In both cases I am using replace_address to replace dead node, as I
> am running into some issues with "nodetool removenode" . I use ephemeral
> disk, so replacement node always comes up with empty data dir and bootstrap.
>
>
>
> Any other work around to mitigate this problem? I am worried about any
> nodes going down while we are in the process of upgrade, as it could take
> several hours to upgrade depending on the cluster size.
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, 

RE: [EXTERNAL] Apache Cassandra upgrade path

[Durity, Sean R]

What you have seen is totally expected. You can’t stream between different 
major versions of Cassandra. Get the upgrade done, then worry about any down 
hardware. If you are using DCs, upgrade one DC at a time, so that there is an 
available environment in case of any disasters.

My advice, though, is to get through the rolling upgrade process as quickly as 
possible. Don’t stay in a mixed state very long. The cluster will function fine 
in a mixed state – except for those streaming operations. No repairs, no 
bootstraps.


Sean Durity – Staff Systems Engineer, Cassandra

From: Jai Bheemsen Rao Dhanwada 
Sent: Friday, July 26, 2019 2:24 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Apache Cassandra upgrade path

Hello,

I am trying to upgrade Apache Cassandra from 2.1.16 to 3.11.3, the regular 
rolling upgrade process works fine without any issues.

However, I am running into an issue where if there is a node with older version 
dies (hardware failure) and a new node comes up and tries to bootstrap, it's 
failing.

I tried two combinations:

1. Joining replacement node with 2.1.16 version of cassandra
In this case nodes with 2.1.16 version are able to stream data to the new node, 
but the nodes with 3.11.3 version are failing with the below error.

ERROR [STREAM-INIT-/10.x.x.x:40296] 2019-07-26 17:45:17,775 
IncomingStreamingConnection.java:80 - Error while reading from socket from 
/10.y.y.y:40296.
java.io.IOException: Received stream using protocol version 2 (my version 4). 
Terminating connection
2. Joining replacement node with 3.11.3 version of cassandra
In this case the nodes with 3.11.3 version of cassandra are able to stream the 
data but it's not able to stream data from the 2.1.16 nodes and failing with 
the below error.

ERROR [STREAM-IN-/10.z.z.z:7000] 2019-07-26 18:08:10,380 StreamSession.java:593 
- [Stream #538c6900-afd0-11e9-a649-ab2e045ee53b] Streaming error occurred on 
session with peer 10.z.z.z
java.io.IOException: Connection reset by peer
   at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_151]
   at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) 
~[na:1.8.0_151]
   at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) 
~[na:1.8.0_151]
   at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_151]
   at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) 
~[na:1.8.0_151]
   at 
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206) 
~[na:1.8.0_151]
   at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) 
~[na:1.8.0_151]
   at 
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) 
~[na:1.8.0_151]
   at 
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56)
 ~[apache-cassandra-3.11.3.jar:3.11.3]
   at 
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:311)
 ~[apache-cassandra-3.11.3.jar:3.11.3]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

Note: In both cases I am using replace_address to replace dead node, as I am 
running into some issues with "nodetool removenode" . I use ephemeral disk, so 
replacement node always comes up with empty data dir and bootstrap.

Any other work around to mitigate this problem? I am worried about any nodes 
going down while we are in the process of upgrade, as it could take several 
hours to upgrade depending on the cluster size.



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: [EXTERNAL] Re: Bursts of Thrift threads make cluster unresponsive

[Durity, Sean R]

This sounds like a bad query or large partition. If a large partition is 
requested on multiple nodes (because of consistency level), it will pressure 
all those replica nodes. Then, as the cluster tries to adjust the rest of the 
load, the other nodes can get overwhelmed, too.

Look at cfstats to see if you have some large partitions. You may also see them 
as warnings in the system.log when they are getting compacted.

Also check for any ALLOW FILTERING queries in the code (or slow query stats, if 
you have them)

Sean


From: Dmitry Simonov 
Sent: Thursday, June 27, 2019 5:22 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Bursts of Thrift threads make cluster unresponsive

> Is there an order in which the events you described happened, or is the order 
> with which you presented them the order you notice things going wrong?

At first, threads count (Thrift) start increasing.
After 2 or 3 minutes they consume all CPU cores.
After that, simultaneously: message drops occur, read latency increases, active 
read tasks are noticed.

пт, 28 июн. 2019 г. в 01:40, Avinash Mandava 
mailto:avin...@vorstella.com>>:
Yeah i skimmed too fast, don't add more work if CPU is pegged, and if using 
thrift protocol NTR would not have values.

Is there an order in which the events you described happened, or is the order 
with which you presented them the order you notice things going wrong?

On Thu, Jun 27, 2019 at 1:29 PM Dmitry Simonov 
mailto:dimmobor...@gmail.com>> wrote:
Thanks for your reply!

> Have you tried increasing concurrent reads until you see more activity in 
> disk?
When problem occurs, freshly created 1.2k - 2k Thrift threads consume all CPU 
on all cores.
Does increasing concurrent reads may help in this situation?

> org.apache.cassandra.metrics.type=ThreadPools.path=transport.scope=Native-Transport-Requests.name=TotalBlockedTasks.Count
This metric is 0 at all cluster nodes.

пт, 28 июн. 2019 г. в 00:34, Avinash Mandava 
mailto:avin...@vorstella.com>>:
Have you tried increasing concurrent reads until you see more activity in disk? 
If you've always got 32 active reads and high pending reads it could just be 
dropping the reads because the queues are saturated. Could be artificially 
bottlenecking at the C* process level.

Also what does this metric show over time:

org.apache.cassandra.metrics.type=ThreadPools.path=transport.scope=Native-Transport-Requests.name=TotalBlockedTasks.Count



On Thu, Jun 27, 2019 at 1:52 AM Dmitry Simonov 
mailto:dimmobor...@gmail.com>> wrote:
Hello!

We've met several times the following problem.

Cassandra cluster (5 nodes) becomes unresponsive for ~30 minutes:
- all CPUs have 100% load (normally we have LA 5 on 16-cores machine)
- cassandra's threads count raises from 300 to 1300 - 2000,most of them are 
Thrift threads in java.net.SocketInputStream.socketRead0(Native Method) method, 
count of other threads doesn't increase
- some Read messages are dropped
- read latency (p99.9) increases to 20-30 seconds
- there are up to 32 active Read Tasks, up to 3k - 6k pending Read Tasks

Problem starts synchronously on all nodes of cluster.
I cannot tie this problem with increased load from clients ("read rate" does't 
increase during the problem).
Also looks like there is no problem with disks (I/O latencies are OK).

Could anybody please give some advice in further troubleshooting?

--
Best Regards,
Dmitry Simonov


--
www.vorstella.com
408 691 8402


--
Best Regards,
Dmitry Simonov


--
www.vorstella.com
408 691 8402


--
Best Regards,
Dmitry Simonov



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or 

Re: [EXTERNAL] Re: Cassandra migration from 1.25 to 3.x

[Anurag Sharma]

Thanks Michael and Sean. That was very useful.

Regards
Anurag

On Mon, Jun 17, 2019 at 7:07 PM Durity, Sean R 
wrote:

> The advice so far is exactly correct for an in-place kind of upgrade. The
> blog post you mentioned is different. They decided to jump versions in
> Cassandra by standing up a new cluster and using a dual-write/dual-read
> process for their app. They also wrote code to read and interpret sstables
> in order to migrate existing data. Getting that right with compaction
> running, data consistency, etc. it not easy. That is what Cassandra does,
> of course. They had to reverse engineer that process.
>
> I would not personally take that path as it seems a more difficult way to
> go -- for the DBA/admin. It is a nice path for the development team,
> though. They only had to look at their reads and writes (already
> encapsulated in a DAO) for the dual clusters. In a multi-upgrade scenario,
> drivers and statements probably have to get upgraded at several steps along
> the way (including a move from Thrift to CQL, probably). More app testing
> is required each upgrade. So, the decision has to be based on which
> resources you have and trust (app dev and testing + Cassandra upgrades or
> data migration and testing). Once you have automated/semi-automated
> Cassandra upgrades in place, that is an easier path, but that company
> obviously hadn't invested there.
>
> Sean Durity
>
> -Original Message-
> From: Michael Shuler  On Behalf Of Michael Shuler
> Sent: Monday, June 17, 2019 8:26 AM
> To: user@cassandra.apache.org
> Subject: [EXTERNAL] Re: Cassandra migration from 1.25 to 3.x
>
> First and foremost, read NEWS.txt from your current version to the version
> you wish to upgrade to. There are too may details that you many need to be
> aware of. For instance, in the 2.0.0 Upgrading notes:
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_cassandra-2D3.11_NEWS.txt-23L1169-2DL1178=DwIDaQ=MtgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=CPWy2XBaDFHzVDapNO4E7kLIFFfbkRd8KqftSrypjSU=D3Y18E9gxewpushCMETjHt9cS8lKvLMrhUdhPriF4Dk=
>
> I assume you meant 1.2.5, so you're first step is to upgrade to at least
> 1.2.9 (I would suggest using latest 1.2.x, which is 1.2.19). Then you can
> to to 2.0.x and up.
>
> Practicing on a scratch cluster is valuable experience. Reading the
> upgrade notes in NEWS.txt is a must.
>
> --
> Kind regards,
> Michael
>
> On 6/17/19 3:34 AM, Anurag Sharma wrote:
> > Thanks Alex,
> >
> > I came across some interesting and efficient ways of upgrading from
> > 1.x to 3.x as described in the blog here
> >  > eezilkha_database-2Dmigration-2Dat-2Dscale-2Dae85c14c3621=DwIDaQ=M
> > tgQEAMQGqekjTjiAhkudQ=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ=
> > CPWy2XBaDFHzVDapNO4E7kLIFFfbkRd8KqftSrypjSU=7ekpEXHT1Qm_xL9l6_1Kty32
> > fDDerlB_PgO1-4K1-VQ= > and others. Was curious if someone has
> > open-sourced their custom utility.  :D
> >
> > Regards
> > Anurag
> >
> > On Mon, Jun 17, 2019 at 1:27 PM Oleksandr Shulgin
> > mailto:oleksandr.shul...@zalando.de>>
> wrote:
> >
> > On Mon, Jun 17, 2019 at 9:30 AM Anurag Sharma
> > mailto:anurag.rp.sha...@gmail.com>>
> wrote:
> >
> >
> > We are upgrading Cassandra from 1.25 to 3.X. Just curious if
> > there is any recommended open source utility for the same.
> >
> >
> > Hi,
> >
> > The "recommended  open source utility" is the Apache Cassandra
> > itself. ;-)
> >
> > Given the huge difference between the major versions, though, you
> > will need a decent amount of planning and preparation to
> > successfully complete such a migration.  Most likely you will want
> > to do it in small steps, first upgrading to the latest minor version
> > in the 1.x series, then making a jump to 2.x, then to 3.0, and only
> > then to 3.x if you really mean to.  On each upgrade step, be sure to
> > examine the release notes carefully to understand if there is any
> > impact for your cluster and/or client applications.  Do have a test
> > system with preferably identical setup and configuration and execute
> > the upgrade steps there first to verify your expectations.
> >
> > Good luck!
> > --
> > Alex
> >
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
> 
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions