A quick and dirty way is to run jstack a few times and see if you can spot
some common methods where code is spending time.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Sep 10, 2015 at 1:05 AM, Roman Tk
again I haven’t read this thread from the beginning so I don’t know which node
is which, but if nodes pause for longish GC, then other nodes will likely be
saving hints (assuming you are writing at the time), then they will be
delivered once the machines become responsive again. I’m just guessin
I have another datapoint from our monitoring system that shows huge
outbound network traffic increase for the affected boxes during these
spikes:
[image: Inline image 1]
Looking at inbound traffic, it is increased on nodes other than these
(purple, yellow and blue) so it does look like some kind
Haven’t been following this thread, but we run beefy machines with 8gig new
gen, 12 gig old gen (down from 16g since moving memtables off heap, we can
probably go lower)…
Apart from making sure you have all the latest -XX: flags from cassandra-env.sh
(and MALLOC_ARENA_MAX), I personally would r
rows.
From: Roman Tkachenko
Reply-To: "user@cassandra.apache.org"
Date: Thursday, September 10, 2015 at 10:54 AM
To: "user@cassandra.apache.org"
Subject: Re: High CPU usage on some of nodes
Thanks for the responses guys.
I also suspected GC and I guess it could be
On Thu, Sep 10, 2015 at 10:54 AM, Roman Tkachenko
wrote:
>
> [5 second CMS GC] Is my best shot to play with JVM settings trying to tune
> garbage collection then?
>
Yep. As a minor note, if the machines are that beefy, they probably have a
lot of RAM, you might wish to consider trying G1 GC and a
uot;user@cassandra.apache.org" ,
> > Date : 10/09/2015 15:30
> > Objet : Re: High CPU usage on some of nodes
> >
> > It sounds like its probably GC. Grep for GC in system.log to verify.
> > If it is GC, there are a myriad of issues that could cause i
/2015 15:30
> Objet : Re: High CPU usage on some of nodes
>
> It sounds like its probably GC. Grep for GC in system.log to verify.
> If it is GC, there are a myriad of issues that could cause it, but
> at least you?ve narrowed it down.
>
> On Sep 9, 2015, at 11:05 PM, Roman
It sounds like its probably GC. Grep for GC in system.log to verify. If it is
GC, there are a myriad of issues that could cause it, but at least you’ve
narrowed it down.
On Sep 9, 2015, at 11:05 PM, Roman Tkachenko wrote:
> Hey guys,
>
> We've been having issues in the past couple of days wit
Hey guys,
We've been having issues in the past couple of days with CPU usage / load
average suddenly skyrocketing on some nodes of the cluster, affecting
performance significantly so majority of requests start timing out. It can
go on for several hours, with CPU spiking through the roof then comin
10 matches
Mail list logo
