timeouts indicate network or equivalent throughput delays, from the physical box's network card out and to the other dc's card. If you are using VM's add that layer. Your network team needs to be looking for ANY timeouts, retries, packets delivered in retry window > 0, etc. ANY value other than zero forever is your problem
*.......* *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Sat, Sep 17, 2016 at 12:02 PM, laxmikanth sadula <laxmikanth...@gmail.com > wrote: > Hi Matija, > > All nodes are UP & running and even GC patterns are all well. But I see > lot of "Timed out replaying hints" in HintedHandOff Manger, I suspect this > might be the reason why GBs of hints getting piled up instead of proper > delivery. > So this clearly indicates some network related issues , so just wanted to > know the way to monitor hints delivery throughput and also tcp > throughput(packets sent, received ,dropped etc., on an interface). > > If anyone monitoring such stats, please let me know. > > On Sat, Sep 17, 2016 at 11:26 PM, Matija Gobec <matija0...@gmail.com> > wrote: > >> Hi, >> >> You should first figure out why you have so many hints and then think >> about throughput of hints delivery. >> Hints are generated for dead nodes and in a healthy cluster are not >> present. >> Are all your nodes alive and running? What is the issue of inter DC >> connectivity? >> >> Matija >> >> -- >> >> *Matija Gobec* >> *Co-Founder & Senior Consultant* >> www.smartcat.io >> >> >> >> *Data <https://www.smartcat.io/expertise/> --> Knowledge >> <https://www.smartcat.io/blog/> --> Power <https://www.smartcat.io/news/> * >> >> On Sat, Sep 17, 2016 at 3:16 PM, laxmikanth sadula < >> laxmikanth...@gmail.com> wrote: >> >>> Hi, >>> >>> Is there any way to monitor hints delivery throughout/performance/issue >>> delivering hints? >>> >>> We have 2 DC c* cluster with 2.0.17 with RF=3 setup. Due to inter DC >>> connectivity issues/some other issues hints shoot upto GBs/node. >>> >>> So I would like to monitor hints throughput/pin point the reason for >>> hints growth on nodes. >>> >>> Kindly let us know if any one of you have such thing to monitor hints. >>> >>> >>> Thanks >>> Laxmikanth >>> >>> >>> -- >>> Regards, >>> Laxmikanth >>> 99621 38051 >>> >>> >>> >> > > > -- > Regards, > Laxmikanth > 99621 38051 > >
