Re: Hints are not getting partially processed
I tried nodetool sethintedhandoffthrottlekb 0;nodetool pausehandoff;sleep 15;nodetool resumehandoff; But still the same On Sun, 26 Sept 2021 at 08:25, Surbhi Gupta wrote: > All nodes are up and running . > Checked system.log and debug.log but nothing useful i got it. > > > On Sun, 26 Sept 2021 at 08:03, Surbhi Gupta > wrote: > >> I tried truncatehints and some of the hints file did not delete but some >> hints file got deleted. >> >> >> On Sun, 26 Sept 2021 at 04:00, Surbhi Gupta >> wrote: >> >>> Hi, >>> >>> We are on 3.11.5 and we have 2 DC cassandra cluster. >>> Suddenly we started seeing hints issue . >>> system.logs says that hints are getting partially replayed and we are >>> seeing hints dropped in target nodes after hinted_handoff_period is over. >>> We checked debug.log as well but nothing we found there. >>> >>> INFO [HintsDispatcher:680] 2021-09-26 03:54:26,399 >>> HintsDispatchExecutor.java:289 - Finished hinted handoff of file >>> aa-1632638838462-1.hints to endpoint /10.xx.yyy.zz: >>> aa, partially >>> >>> This is happening only on one node. I stopped the cassandra and deleted >>> the hints files and started cassandra but it started happening again after >>> few hours. >>> >>> Any idea ? >>> >>> Thanks >>> Surbhi >>> >>> >>>
Re: Hints are not getting partially processed
All nodes are up and running . Checked system.log and debug.log but nothing useful i got it. On Sun, 26 Sept 2021 at 08:03, Surbhi Gupta wrote: > I tried truncatehints and some of the hints file did not delete but some > hints file got deleted. > > > On Sun, 26 Sept 2021 at 04:00, Surbhi Gupta > wrote: > >> Hi, >> >> We are on 3.11.5 and we have 2 DC cassandra cluster. >> Suddenly we started seeing hints issue . >> system.logs says that hints are getting partially replayed and we are >> seeing hints dropped in target nodes after hinted_handoff_period is over. >> We checked debug.log as well but nothing we found there. >> >> INFO [HintsDispatcher:680] 2021-09-26 03:54:26,399 >> HintsDispatchExecutor.java:289 - Finished hinted handoff of file >> aa-1632638838462-1.hints to endpoint /10.xx.yyy.zz: >> aa, partially >> >> This is happening only on one node. I stopped the cassandra and deleted >> the hints files and started cassandra but it started happening again after >> few hours. >> >> Any idea ? >> >> Thanks >> Surbhi >> >> >>
Re: Hints are not getting partially processed
I tried truncatehints and some of the hints file did not delete but some hints file got deleted. On Sun, 26 Sept 2021 at 04:00, Surbhi Gupta wrote: > Hi, > > We are on 3.11.5 and we have 2 DC cassandra cluster. > Suddenly we started seeing hints issue . > system.logs says that hints are getting partially replayed and we are > seeing hints dropped in target nodes after hinted_handoff_period is over. > We checked debug.log as well but nothing we found there. > > INFO [HintsDispatcher:680] 2021-09-26 03:54:26,399 > HintsDispatchExecutor.java:289 - Finished hinted handoff of file > aa-1632638838462-1.hints to endpoint /10.xx.yyy.zz: > aa, partially > > This is happening only on one node. I stopped the cassandra and deleted > the hints files and started cassandra but it started happening again after > few hours. > > Any idea ? > > Thanks > Surbhi > > >
Hints are not getting partially processed
Hi, We are on 3.11.5 and we have 2 DC cassandra cluster. Suddenly we started seeing hints issue . system.logs says that hints are getting partially replayed and we are seeing hints dropped in target nodes after hinted_handoff_period is over. We checked debug.log as well but nothing we found there. INFO [HintsDispatcher:680] 2021-09-26 03:54:26,399 HintsDispatchExecutor.java:289 - Finished hinted handoff of file aa-1632638838462-1.hints to endpoint /10.xx.yyy.zz: aa, partially This is happening only on one node. I stopped the cassandra and deleted the hints files and started cassandra but it started happening again after few hours. Any idea ? Thanks Surbhi
