Re: How to read content of hints file and apply them manually?

I would do a thread dump and work out the threads with the highest CPU consumers from it. But in my experience, 90% of the time it's GC from high app traffic unless you've hit an edge case bug. Which means the cluster doesn't have enough capacity and you need to review the cluster size. Cheers!

Re: How to read content of hints file and apply them manually?

So this problem we face is , every time a node goes down or a node is under high load or CPU. We see lots of hints piles up and doesn’t apply on the other nodes. Last time when this happened we noticed, high pending mutations but when I have gone back and checked the history of events , not every

Re: How to read content of hints file and apply them manually?

I would definitely check the IO stats then, If you see latency going over 20ms, you need to solve that problem. Patrick On Tue, Jan 28, 2020 at 12:01 PM Surbhi Gupta wrote: > We have also noticed a lot of MutationStage pending . > > > On Tue, 28 Jan 2020 at 11:06, Richard Andersen > wrote: >

Re: How to read content of hints file and apply them manually?

We have also noticed a lot of MutationStage pending . On Tue, 28 Jan 2020 at 11:06, Richard Andersen wrote: > I am in agreement with Patrick, this is a typical symptom of saturated IO. > Are there a high of drops and/or pending compactions? > > Get Outlook for Android >

Re: How to read content of hints file and apply them manually?

I am in agreement with Patrick, this is a typical symptom of saturated IO. Are there a high of drops and/or pending compactions? Get Outlook for Android From: Patrick McFadin Sent: Tuesday, January 28, 2020 11:25:49 AM To:

Re: How to read content of hints file and apply them manually?

Just to add in here. Any time I see any hints on a cluster, that's like seeing smoke. If you can't explain it, you have a fire somewhere and it's not going to get any better. By the few messages I've seen, I would start by looking at your IO subsystem on your nodes. Do you have enough throughput

Re: How to read content of hints file and apply them manually?

We tried to tune sethintedhandoffthrottlekb to 100 , 1024 , 10240 but nothing helped . Our hints related parameters are as below, if you don't find any parameter below then it is not set in our environment and should be of the default value. max_hint_window_in_ms: 1080 # 3 hours

Re: How to read content of hints file and apply them manually?

The high cpu is probably the hints getting replayed slamming the write path Slowing it down with the hint throttle may help It’s not instant. > On Jan 27, 2020, at 6:05 PM, Erick Ramirez wrote: > >  >> Increase the max_hint_window_in_ms setting in cassandra.yaml to more than 3 >> hours,

Re: How to read content of hints file and apply them manually?

> > Increase the max_hint_window_in_ms setting in cassandra.yaml to more than > 3 hours, perhaps 6 hours. If the issue still persists networking may need > to be tested for bandwidth issues. > Just a note of warning about bumping up the hint window without understanding the pros and cons. Be

Re: How to read content of hints file and apply them manually?

Surbhi, The hints could be getting accumulated for one or both of the following reasons: - Some node is becoming unavailable very routinely, which is unlikely- The hints are getting replayed very slowly due to network bandwidth issues, which is more likely Increase the max_hint_window_in_ms

Re: How to read content of hints file and apply them manually?

Why we think it might be related to hints is , because if we truncate the hints then load goes normal on the nodes. FYI , We had to run repair after truncating hints. Any thoughts ? On Mon, 27 Jan 2020 at 15:27, Deepak Vohra wrote: > > Hints are a stopgap measure and not a fix to the

Re: How to read content of hints file and apply them manually?

Hints are a stopgap measure and not a fix to the underlying issue. Run a full repair.On Monday, January 27, 2020, 10:17:01 p.m. UTC, Surbhi Gupta wrote: Hi, We are on Open source 3.11 .We have a issue in one of the cluster where lots of hints gets piled up and they don't get applied

Re: How to read content of hints file and apply them manually?

There isn't a tool that I'm aware of that's readily available to do that. Your best bet is to run a regular repair. But really, hints are just a side-issue of a much wider problem and that is the nodes are overloaded. Is your application getting hit with a much higher than expected traffic? The

How to read content of hints file and apply them manually?

Hi, We are on Open source 3.11 . We have a issue in one of the cluster where lots of hints gets piled up and they don't get applied within hinted handoff period ( 3 hour in our case) . And load and CPU of the server goes very high. We see lot of messages in system.log and debug.log . Our read