The number of Completed HH tasks is interesting. AFAIK a task is started when
the node detects another in the cluster has returned. Were you doing some
other restarts around the cluster ?
Not at all. The restarts seem to happen as normal operation.
I don't want to divert from the GC
Greetings,
Just out of curiosity is this on the receiver or sender side?
Looks like sender side, although the 2 nodes were replicating to each
other so it's hard to tell.
I have been wondering a bit if the hint playback could need some
adjustment.
There is potentially quite big
An if you have 10 nodes, do all of them happen to send hints to the two
with GC?
The 2 nodes are adjacent in token range. They are replicating to each other.
Other nodes have no data to replicate so there's no proof one way or
another.
Best,
Gabriel
Greetings,
Doesn't really look abnormal to me for a heavy write load situation
which is what receiving hints is.
I would agree with you but this raises some questions about write
performance. Plus I've only seen this kind of behavior recently and only
on 2 adjacent nodes. So I have good reason
I'm assuming the two nodes are the ones receiving the HH after they were
down.
Adjacent, so yes.
Are there a lot of hints collected while they are down ? you can check the
HintedHandOffManager MBean in JConsole
There wasn't any downtime - that's something else that's weird.
What
What does the TPStats look like on the nodes under pressure ? And how many
nodes are delivering hints to the nodes when they restart?
$nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed
ReadStage 1 11992475
Greetings,
I'm experiencing some issues with 2 nodes (out of more than 10). Right
after startup (Listening for thrift clients...) the nodes will create
objects at high rate using all available CPU cores:
INFO 18:13:15,350 GC for PS Scavenge: 292 ms, 494902976 reclaimed
leaving 2024909864 used;
