I would do a thread dump and work out the threads with the highest CPU
consumers from it. But in my experience, 90% of the time it's GC from high
app traffic unless you've hit an edge case bug. Which means the cluster
doesn't have enough capacity and you need to review the cluster size.
Cheers!
or pending compactions?
>>>
>>> Get Outlook for Android <https://aka.ms/ghei36>
>>> --
>>> *From:* Patrick McFadin
>>> *Sent:* Tuesday, January 28, 2020 11:25:49 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:*
ck McFadin
>> *Sent:* Tuesday, January 28, 2020 11:25:49 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: How to read content of hints file and apply them manually?
>>
>> Just to add in here. Any time I see any hints on a cluster, that's like
>> see
//aka.ms/ghei36>
> --
> *From:* Patrick McFadin
> *Sent:* Tuesday, January 28, 2020 11:25:49 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: How to read content of hints file and apply them manually?
>
> Just to add in here. Any time I
: user@cassandra.apache.org
Subject: Re: How to read content of hints file and apply them manually?
Just to add in here. Any time I see any hints on a cluster, that's like seeing
smoke. If you can't explain it, you have a fire somewhere and it's not going to
get any better.
By the few messag
Just to add in here. Any time I see any hints on a cluster, that's like
seeing smoke. If you can't explain it, you have a fire somewhere and it's
not going to get any better.
By the few messages I've seen, I would start by looking at your IO
subsystem on your nodes. Do you have enough throughput t
We tried to tune sethintedhandoffthrottlekb to 100 , 1024 , 10240 but
nothing helped .
Our hints related parameters are as below, if you don't find any parameter
below then it is not set in our environment and should be of the default
value.
max_hint_window_in_ms: 1080 # 3 hours
hinted_handof
The high cpu is probably the hints getting replayed slamming the write path
Slowing it down with the hint throttle may help
It’s not instant.
> On Jan 27, 2020, at 6:05 PM, Erick Ramirez wrote:
>
>
>> Increase the max_hint_window_in_ms setting in cassandra.yaml to more than 3
>> hours, pe
>
> Increase the max_hint_window_in_ms setting in cassandra.yaml to more than
> 3 hours, perhaps 6 hours. If the issue still persists networking may need
> to be tested for bandwidth issues.
>
Just a note of warning about bumping up the hint window without
understanding the pros and cons. Be aware
Surbhi,
The hints could be getting accumulated for one or both of the following reasons:
- Some node is becoming unavailable very routinely, which is unlikely- The
hints are getting replayed very slowly due to network bandwidth issues, which
is more likely
Increase the max_hint_window_in_ms set
Why we think it might be related to hints is , because if we truncate the
hints then load goes normal on the nodes.
FYI , We had to run repair after truncating hints.
Any thoughts ?
On Mon, 27 Jan 2020 at 15:27, Deepak Vohra
wrote:
>
> Hints are a stopgap measure and not a fix to the underlying
Hints are a stopgap measure and not a fix to the underlying issue. Run a full
repair.On Monday, January 27, 2020, 10:17:01 p.m. UTC, Surbhi Gupta
wrote:
Hi,
We are on Open source 3.11 .We have a issue in one of the cluster where lots of
hints gets piled up and they don't get applied
There isn't a tool that I'm aware of that's readily available to do that.
Your best bet is to run a regular repair.
But really, hints are just a side-issue of a much wider problem and that is
the nodes are overloaded. Is your application getting hit with a much
higher than expected traffic? The sc
Hi,
We are on Open source 3.11 .
We have a issue in one of the cluster where lots of hints gets piled up and
they don't get applied within hinted handoff period ( 3 hour in our case) .
And load and CPU of the server goes very high.
We see lot of messages in system.log and debug.log . Our read re
14 matches
Mail list logo
