We’re seeing a strange issue on our Cassandra cluster wherein 3 nodes out
of 21 appear to have a significant amount of hints piling up. We’re not
seeing a lot in the system log showing that the node is having issues with
hints and nodetool status is not showing any issues with the other nodes in
the cluster.
In attempting to help the node catch up with hints we’ve tried to increase
the hinted handoff KB throttle (we ran "nodetool sethintedhandoffthrottlekb
20480" on the 3 nodes getting backed up) but that does not appear to have
made a difference in the hints processing.
We’re looking for guidance on how we can debug the cluster to determine why
the node may be falling behind on hints and how to resolve the situation.
We’re currently looking at at the JMX Storage.TotalHints.count metric as
well as the hints directory itself.
[image: Inline image 1]
sudo du -hs /mnt/cassandra/data/hints
39G /mnt/cassandra/data/hints
Nodetool tpstats is showing 1 active HintsDispatcher.
nodetool tpstats | grep Hints
Pool Name Active Pending Completed Blocked All
time blocked
HintsDispatcher 1 6 66 0
0
