Re: failing bootstraps with OOM

[Carlos Alonso]

If what you need is a replacement node, to increase the hardware specs I'd
recommend an 'immediate node replacement' like described here:
http://mrcalonso.com/cassandra-instantaneous-in-place-node-replacement/

Basically the process just rsyncs the relevant data (data + configuration)
from one node to another and stop the old node and start the new one. As
the configuration is all the same (just the ip will change) it joins the
ring as if it was the old one. And there's no need for any bootstrapping.

BTW, are you using vnodes?

Regards

Carlos Alonso | Software Engineer | @calonso 

On 3 November 2016 at 15:46, Oleksandr Shulgin  wrote:

> On Thu, Nov 3, 2016 at 2:32 PM, Mike Torra  wrote:
>
>> Hi Alex - I do monitor sstable counts and pending compactions, but
>> probably not closely enough. In 3/4 regions the cluster is running in, both
>> counts are very high - ~30-40k sstables for one particular CF, and on many
>> nodes >1k pending compactions.
>>
>
> It is generally a good idea to try to keep the number of pending
> compactions minimal.  We usually see it is close to zero on every node
> during normal operations and less than some tens during maintenance such as
> repair.
>
> I had noticed this before, but I didn't have a good sense of what a "high"
>> number for these values was.
>>
>
> I would say anything higher than 20 probably requires someone to have a
> look and over 1k is very troublesome.
>
> It makes sense to me why this would cause the issues I've seen. After
>> increasing concurrent_compactors and compaction_throughput_mb_per_sec
>> (to 8 and 64mb, respectively), I'm starting to see those counts go down
>> steadily. Hopefully that will resolve the OOM issues, but it looks like it
>> will take a while for compactions to catch up.
>>
>> Thanks for the suggestions, Alex
>>
>
> Welcome. :-)
>
> --
> Alex
>
>

Re: failing bootstraps with OOM

[Oleksandr Shulgin]

On Thu, Nov 3, 2016 at 2:32 PM, Mike Torra  wrote:

> Hi Alex - I do monitor sstable counts and pending compactions, but
> probably not closely enough. In 3/4 regions the cluster is running in, both
> counts are very high - ~30-40k sstables for one particular CF, and on many
> nodes >1k pending compactions.
>

It is generally a good idea to try to keep the number of pending
compactions minimal.  We usually see it is close to zero on every node
during normal operations and less than some tens during maintenance such as
repair.

I had noticed this before, but I didn't have a good sense of what a "high"
> number for these values was.
>

I would say anything higher than 20 probably requires someone to have a
look and over 1k is very troublesome.

It makes sense to me why this would cause the issues I've seen. After
> increasing concurrent_compactors and compaction_throughput_mb_per_sec (to
> 8 and 64mb, respectively), I'm starting to see those counts go down
> steadily. Hopefully that will resolve the OOM issues, but it looks like it
> will take a while for compactions to catch up.
>
> Thanks for the suggestions, Alex
>

Welcome. :-)

--
Alex

Re: failing bootstraps with OOM

[Mike Torra]

Hi Alex - I do monitor sstable counts and pending compactions, but probably not 
closely enough. In 3/4 regions the cluster is running in, both counts are very 
high - ~30-40k sstables for one particular CF, and on many nodes >1k pending 
compactions. I had noticed this before, but I didn't have a good sense of what 
a "high" number for these values was.

It makes sense to me why this would cause the issues I've seen. After 
increasing concurrent_compactors and compaction_throughput_mb_per_sec (to 8 and 
64mb, respectively), I'm starting to see those counts go down steadily. 
Hopefully that will resolve the OOM issues, but it looks like it will take a 
while for compactions to catch up.

Thanks for the suggestions, Alex

From: Oleksandr Shulgin 
<oleksandr.shul...@zalando.de<mailto:oleksandr.shul...@zalando.de>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Wednesday, November 2, 2016 at 1:07 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: failing bootstraps with OOM

On Wed, Nov 2, 2016 at 3:35 PM, Mike Torra 
<mto...@demandware.com<mailto:mto...@demandware.com>> wrote:
>
> Hi All -
>
> I am trying to bootstrap a replacement node in a cluster, but it consistently 
> fails to bootstrap because of OOM exceptions. For almost a week I've been 
> going through cycles of bootstrapping, finding errors, then restarting / 
> resuming bootstrap, and I am struggling to move forward. Sometimes the 
> bootstrapping node itself fails, which usually manifests first as very high 
> GC times (sometimes 30s+!), then nodetool commands start to fail with 
> timeouts, then the node will crash with an OOM exception. Other times, a node 
> streaming data to this bootstrapping node will have a similar failure. In 
> either case, when it happens I need to restart the crashed node, then resume 
> the bootstrap.
>
> On top of these issues, when I do need to restart a node it takes a lng 
> time 
> (http://stackoverflow.com/questions/40141739/why-does-cassandra-sometimes-take-a-hours-to-start).
>  This exasperates the problem because it takes so long to find out if a 
> change to the cluster helps or if it still fails. I am in the process of 
> upgrading all nodes in the cluster from m4.xlarge to c4.4xlarge, and I am 
> running Cassandra DDC 3.5 on all nodes. The cluster has 26 nodes spread 
> across 4 regions in EC2. Here is some other relevant cluster info (also in 
> stack overflow post):
>
> Cluster Info
>
> Cassandra DDC 3.5
> EC2MultiRegionSnitch
> m4.xlarge, moving to c4.4xlarge
>
> Schema Info
>
> 3 CF's, all 'write once' (ie no updates), 1 week ttl, STCS (default)
> no secondary indexes
>
> I am unsure what to try next. The node that is currently having this 
> bootstrap problem is a pretty beefy box, with 16 cores, 30G of ram, and a 
> 3.2T EBS volume. The slow startup time might be because of the issues with a 
> high number of SSTables that Jeff Jirsa mentioned in a comment on the SO 
> post, but I am at a loss for the OOM issues. I've tried:
>
> Changing from CMS to G1 GC, which seemed to have helped a bit
> Upgrading from 3.5 to 3.9, which did not seem to help
> Upgrading instance types from m4.xlarge to c4.4xlarge, which seems to help, 
> but I'm still having issues
>
> I'd appreciate any suggestions on what else I can try to track down the cause 
> of these OOM exceptions.

Hi,

Do you monitor pending compactions and actual number of SSTable files?

On startup Cassandra needs to touch most of the data files and also seems to 
keep some metadata about every relevant file in memory.  We once went into 
situation where we ended up with hundreds of thousands of files per node which 
resulted in OOMs on every other node of the ring, and startup time was of over 
half an hour (this was on version 2.1).

If you have much more files than you expect, then you should check and adjust 
your concurrent_compactors and compaction_throughput_mb_per_sec settings.  
Increase concurrent_compactors if you're behind (pending compactions metric is 
a hint) and consider un-throttling compaction before your situation is back to 
normal.

Cheers,
--
Alex

Re: failing bootstraps with OOM

[Vladimir Yudovin]

Hi,



probably you can try to start new node with auto_bootstrap: false and then 
repair keypaces or even tables one by one with nodetool repair 



Best regards, Vladimir Yudovin, 

Winguzone - Hosted Cloud Cassandra
Launch your cluster in minutes.





 On Wed, 02 Nov 2016 10:35:45 -0400Mike Torra mto...@demandware.com 
wrote 




Hi All -



I am trying to bootstrap a replacement node in a cluster, but it consistently 
fails to bootstrap because of OOM exceptions. For almost a week I've been going 
through cycles of bootstrapping, finding errors, then restarting / resuming 
bootstrap, and I am struggling to move forward. Sometimes the bootstrapping 
node itself fails, which usually manifests first as very high GC times 
(sometimes 30s+!), then nodetool commands start to fail with timeouts, then the 
node will crash with an OOM exception. Other times, a node streaming data to 
this bootstrapping node will have a similar failure. In either case, when it 
happens I need to restart the crashed node, then resume the bootstrap.



On top of these issues, when I do need to restart a node it takes a lng 
time 
(http://stackoverflow.com/questions/40141739/why-does-cassandra-sometimes-take-a-hours-to-start).
 This exasperates the problem because it takes so long to find out if a change 
to the cluster helps or if it still fails. I am in the process of upgrading all 
nodes in the cluster from m4.xlarge to c4.4xlarge, and I am running Cassandra 
DDC 3.5 on all nodes. The cluster has 26 nodes spread across 4 regions in EC2. 
Here is some other relevant cluster info (also in stack overflow post):



Cluster Info

Cassandra DDC 3.5

EC2MultiRegionSnitch

m4.xlarge, moving to c4.4xlarge

Schema Info

3 CF's, all 'write once' (ie no updates), 1 week ttl, STCS (default)

no secondary indexes

I am unsure what to try next. The node that is currently having this bootstrap 
problem is a pretty beefy box, with 16 cores, 30G of ram, and a 3.2T EBS 
volume. The slow startup time might be because of the issues with a high number 
of SSTables that Jeff Jirsa mentioned in a comment on the SO post, but I am at 
a loss for the OOM issues. I've tried:


Changing from CMS to G1 GC, which seemed to have helped a bit

Upgrading from 3.5 to 3.9, which did not seem to help

Upgrading instance types from m4.xlarge to c4.4xlarge, which seems to help, but 
I'm still having issues

I'd appreciate any suggestions on what else I can try to track down the cause 
of these OOM exceptions.



- Mike

Re: failing bootstraps with OOM

[Oleksandr Shulgin]

On Wed, Nov 2, 2016 at 3:35 PM, Mike Torra  wrote:
>
> Hi All -
>
> I am trying to bootstrap a replacement node in a cluster, but it
consistently fails to bootstrap because of OOM exceptions. For almost a
week I've been going through cycles of bootstrapping, finding errors, then
restarting / resuming bootstrap, and I am struggling to move forward.
Sometimes the bootstrapping node itself fails, which usually manifests
first as very high GC times (sometimes 30s+!), then nodetool commands start
to fail with timeouts, then the node will crash with an OOM exception.
Other times, a node streaming data to this bootstrapping node will have a
similar failure. In either case, when it happens I need to restart the
crashed node, then resume the bootstrap.
>
> On top of these issues, when I do need to restart a node it takes a
lng time (
http://stackoverflow.com/questions/40141739/why-does-cassandra-sometimes-take-a-hours-to-start).
This exasperates the problem because it takes so long to find out if a
change to the cluster helps or if it still fails. I am in the process of
upgrading all nodes in the cluster from m4.xlarge to c4.4xlarge, and I am
running Cassandra DDC 3.5 on all nodes. The cluster has 26 nodes spread
across 4 regions in EC2. Here is some other relevant cluster info (also in
stack overflow post):
>
> Cluster Info
>
> Cassandra DDC 3.5
> EC2MultiRegionSnitch
> m4.xlarge, moving to c4.4xlarge
>
> Schema Info
>
> 3 CF's, all 'write once' (ie no updates), 1 week ttl, STCS (default)
> no secondary indexes
>
> I am unsure what to try next. The node that is currently having this
bootstrap problem is a pretty beefy box, with 16 cores, 30G of ram, and a
3.2T EBS volume. The slow startup time might be because of the issues with
a high number of SSTables that Jeff Jirsa mentioned in a comment on the SO
post, but I am at a loss for the OOM issues. I've tried:
>
> Changing from CMS to G1 GC, which seemed to have helped a bit
> Upgrading from 3.5 to 3.9, which did not seem to help
> Upgrading instance types from m4.xlarge to c4.4xlarge, which seems to
help, but I'm still having issues
>
> I'd appreciate any suggestions on what else I can try to track down the
cause of these OOM exceptions.

Hi,

Do you monitor pending compactions and actual number of SSTable files?

On startup Cassandra needs to touch most of the data files and also seems
to keep some metadata about every relevant file in memory.  We once went
into situation where we ended up with hundreds of thousands of files per
node which resulted in OOMs on every other node of the ring, and startup
time was of over half an hour (this was on version 2.1).

If you have much more files than you expect, then you should check and
adjust your concurrent_compactors and compaction_throughput_mb_per_sec
settings.  Increase concurrent_compactors if you're behind (pending
compactions metric is a hint) and consider un-throttling compaction before
your situation is back to normal.

Cheers,
--
Alex

failing bootstraps with OOM

[Mike Torra]

Hi All -

I am trying to bootstrap a replacement node in a cluster, but it consistently 
fails to bootstrap because of OOM exceptions. For almost a week I've been going 
through cycles of bootstrapping, finding errors, then restarting / resuming 
bootstrap, and I am struggling to move forward. Sometimes the bootstrapping 
node itself fails, which usually manifests first as very high GC times 
(sometimes 30s+!), then nodetool commands start to fail with timeouts, then the 
node will crash with an OOM exception. Other times, a node streaming data to 
this bootstrapping node will have a similar failure. In either case, when it 
happens I need to restart the crashed node, then resume the bootstrap.

On top of these issues, when I do need to restart a node it takes a lng 
time 
(http://stackoverflow.com/questions/40141739/why-does-cassandra-sometimes-take-a-hours-to-start).
 This exasperates the problem because it takes so long to find out if a change 
to the cluster helps or if it still fails. I am in the process of upgrading all 
nodes in the cluster from m4.xlarge to c4.4xlarge, and I am running Cassandra 
DDC 3.5 on all nodes. The cluster has 26 nodes spread across 4 regions in EC2. 
Here is some other relevant cluster info (also in stack overflow post):

Cluster Info

  *   Cassandra DDC 3.5
  *   EC2MultiRegionSnitch
  *   m4.xlarge, moving to c4.4xlarge

Schema Info

  *   3 CF's, all 'write once' (ie no updates), 1 week ttl, STCS (default)
  *   no secondary indexes

I am unsure what to try next. The node that is currently having this bootstrap 
problem is a pretty beefy box, with 16 cores, 30G of ram, and a 3.2T EBS 
volume. The slow startup time might be because of the issues with a high number 
of SSTables that Jeff Jirsa mentioned in a comment on the SO post, but I am at 
a loss for the OOM issues. I've tried:

  *   Changing from CMS to G1 GC, which seemed to have helped a bit
  *   Upgrading from 3.5 to 3.9, which did not seem to help
  *   Upgrading instance types from m4.xlarge to c4.4xlarge, which seems to 
help, but I'm still having issues

I'd appreciate any suggestions on what else I can try to track down the cause 
of these OOM exceptions.

- Mike