Hi,
Is there any way to monitor hints delivery throughout/performance/issue
delivering hints?
We have 2 DC c* cluster with 2.0.17 with RF=3 setup. Due to inter DC
connectivity issues/some other issues hints shoot upto GBs/node.
So I would like to monitor hints throughput/pin point the reason for
Hi,
You should first figure out why you have so many hints and then think about
throughput of hints delivery.
Hints are generated for dead nodes and in a healthy cluster are not present.
Are all your nodes alive and running? What is the issue of inter DC
connectivity?
Matija
--
*Matija Gobec*
Hi Matija,
All nodes are UP & running and even GC patterns are all well. But I see lot
of "Timed out replaying hints" in HintedHandOff Manger, I suspect this
might be the reason why GBs of hints getting piled up instead of proper
delivery.
So this clearly indicates some network related issues , so
timeouts indicate network or equivalent throughput delays, from the
physical box's network card out and to the other dc's card. If you are
using VM's add that layer. Your network team needs to be looking for ANY
timeouts, retries, packets delivered in retry window > 0, etc. ANY value
other than zer
Hey Nicolas,
There are no dead nodes. 'nodetool status' and 'nodetool describecluster'
both show 4 healthy nodes. In the past we have had some nodes we eliminated
by using 'nodetool assassinate'. However, I checked system.peers table on
all 4 of our nodes and they each show 3 peers as expected. So
