ShaunakDas88 opened a new pull request #1123:
URL: https://github.com/apache/cassandra/pull/1123


   This seems to behave as expected. We are injecting a writes to all nodes in 
the cluster when we kill one node:
   ```
   INFO  [GossipStage:1] 2021-07-30 00:49:25,579 Gossiper.java:1290 - 
InetAddress /<DOWNED NODE IP> is now DOWN
   ```
   
   We see hints are getting stored on other nodes, and when we run `nodetool 
transferhints -d`:
   - no more hints on disk
   - the command blocks for a bit
   - hinted handoff is disabled
   ```
   $ ls /mnt/cass_data_disks/data1/hints/
   a7c7f39d-bcaf-4b51-811a-5b00e708b7f5-1627606160115-2.crc32  
a7c7f39d-bcaf-4b51-811a-5b00e708b7f5-1627606160115-2.hints  
a7c7f39d-bcaf-4b51-811a-5b00e708b7f5-1627606170111-2.hints
   $
   $ nodetool statushandoff
   Hinted handoff is running
   $
   $ date -u; nodetool transferhints -d; date -u
   Fri Jul 30 00:59:10 UTC 2021
   Fri Jul 30 00:59:18 UTC 2021
   $
   $ nodetool statushandoff
   Hinted handoff is not running
   ```
   
   The above timestamps for the hint transfer window are confirmed both in the 
system logs:
   ```
   INFO  [HintsDispatcher:19] 2021-07-30 00:59:12,396 
HintsDispatchExecutor.java:158 - Transferring all hints to /<RECEIVING NODE 
IP>:7000: 15737a33-a0a6-4331-97bf-32f8af2534c1
   DEBUG [RMI TCP Connection(10)-<SENDING NODE IP>] 2021-07-30 00:59:18,372 
StorageService.java:4536 - Hint transfer complete
   ```
   and the metric for total hints flat-lining:
   
![nodetool-transferhints](https://user-images.githubusercontent.com/11130580/127586500-c909f2c1-e26b-4cdd-9782-7c68e37b779d.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to