I started up a new node and did a "-Dcassandra.replace_address" which I terminated before it could join. This has resulted in one more hibernating node similar to the other one that isn't disappearing after 72 hours.
I've added trace level logging to a working node to produce the following (I'm not sure what I'm looking for here, so this is just everything for a few seconds): TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:889 - local > heartbeat version 1227 greater than 1224 for /172.31.132.4 > TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:904 - Adding > state SEVERITY: 0.5076141953468323 > TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:889 - local > heartbeat version 1326 greater than 1325 for /172.31.132.5 > TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:889 - local > heartbeat version 1903 greater than 1900 for /172.31.150.151 > TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:05,065 > GossipDigestAck2VerbHandler.java:38 - Received a GossipDigestAck2Message > from /172.31.126.217 > TRACE [GossipStage:1] 2016-10-17 11:17:05,065 FailureDetector.java:272 - > Average for /172.31.126.217 is 9.973875527435898E8 > TRACE [GossipStage:1] 2016-10-17 11:17:05,065 Gossiper.java:1140 - / > 172.31.126.217 local generation 1476702656, remote generation 1476702656 > TRACE [GossipStage:1] 2016-10-17 11:17:05,065 Gossiper.java:1193 - > Updating heartbeat state version to 1136 from 1133 for /172.31.126.217 ... > TRACE [GossipStage:1] 2016-10-17 11:17:05,482 Gossiper.java:889 - local > heartbeat version 1040 greater than 1037 for /172.31.126.216 > TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:889 - local > heartbeat version 1136 greater than 1133 for /172.31.126.217 > TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:904 - Adding > state SEVERITY: 0.5102040767669678 > TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:889 - local > heartbeat version 1446 greater than 1443 for /172.31.150.150 > TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:889 - local > heartbeat version 1903 greater than 1902 for /172.31.150.151 > TRACE [GossipStage:1] 2016-10-17 11:17:05,484 > GossipDigestAck2VerbHandler.java:38 - Received a GossipDigestAck2Message > from /172.31.132.4 > TRACE [GossipStage:1] 2016-10-17 11:17:05,484 FailureDetector.java:272 - > Average for /172.31.132.4 is 1.0427801260213675E9 > TRACE [GossipStage:1] 2016-10-17 11:17:05,484 Gossiper.java:1140 - / > 172.31.132.4 local generation 1476702626, remote generation 1476702626 > TRACE [GossipStage:1] 2016-10-17 11:17:05,484 Gossiper.java:1193 - > Updating heartbeat state version to 1230 from 1227 for /172.31.132.4 ... > TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 Gossiper.java:144 - My > heartbeat is now 1043 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 Gossiper.java:500 - Gossip > Digests are : /172.31.150.151:1476702405:1903 /172.31.132.5:1476702594:1326 > /172.31.126.217:1476702656:1136 /172.31.126.216:1476702687:1043 > /172.31.142.111:1476347719:238 /172.31.137.65:1433571405:232 > /172.31.132.4:1476702626:1230 /172.31.150.150:1476702555:1446 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 Gossiper.java:646 - Sending > a GossipDigestSyn to /172.31.150.150 ... > TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 Gossiper.java:757 - > Performing status check ... > TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 FailureDetector.java:298 - > PHI for /172.31.142.111 : 167.537185959 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:303 - > Node /172.31.142.111 phi 72.76047537559265 > 12.0; intervals: > [2000000000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0] mean: 2.0E9 > DEBUG [GossipTasks:1] 2016-10-17 11:17:05,594 Gossiper.java:337 - > Convicting /172.31.142.111 with status hibernate - alive false > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 - > PHI for /172.31.126.217 : 0.5296914569935668 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 - > PHI for /172.31.126.217 : 0.5296914569935668 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 - > mean for /172.31.126.217 : 9.973875527435898E8 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 - > PHI for /172.31.150.151 : 0.9312981815294517 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 - > PHI for /172.31.150.151 : 0.9312981815294517 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 - > mean for /172.31.150.151 : 9.8568609518E8 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 - > PHI for /172.31.150.150 : 0.9710661780597467 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 - > PHI for /172.31.150.150 : 0.9710661780597467 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 - > mean for /172.31.150.150 : 1.0299954375905173E9 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 - > PHI for /172.31.132.5 : 0.6849131720784052 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 - > PHI for /172.31.132.5 : 0.6849131720784052 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 - > mean for /172.31.132.5 : 1.0157781925682657E9 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 - > PHI for /172.31.132.4 : 0.10601682583058232 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 - > PHI for /172.31.132.4 : 0.10601682583058232 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 - > mean for /172.31.132.4 : 1.0427801260213675E9 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 - > PHI for /172.31.137.65 : 167.5363108695 > TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:303 - > Node /172.31.137.65 phi 72.76009532905162 > 12.0; intervals: [2000000000, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] > mean: 2.0E9 > DEBUG [GossipTasks:1] 2016-10-17 11:17:05,595 Gossiper.java:337 - > Convicting /172.31.137.65 with status hibernate - alive false > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 > GossipDigestAckVerbHandler.java:41 - Received a GossipDigestAckMessage from > /172.31.150.150 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 > GossipDigestAckVerbHandler.java:52 - Received ack with 4 digests and 2 > states > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 FailureDetector.java:272 - > Average for /172.31.150.150 is 1.0298713880042919E9 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:1140 - / > 172.31.150.151 local generation 1476702405, remote generation 1476702405 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:1193 - > Updating heartbeat state version to 1903 from 1903 for /172.31.150.151 ... > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:1140 - / > 172.31.150.150 local generation 1476702555, remote generation 1476702555 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:1193 - > Updating heartbeat state version to 1449 from 1446 for /172.31.150.150 ... > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:889 - local > heartbeat version 1043 greater than 1040 for /172.31.126.216 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:889 - local > heartbeat version 1136 greater than 1133 for /172.31.126.217 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:904 - Adding > state SEVERITY: 0.5102040767669678 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:889 - local > heartbeat version 1230 greater than 1227 for /172.31.132.4 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:904 - Adding > state SEVERITY: 0.5128205418586731 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:889 - local > heartbeat version 1326 greater than 1325 for /172.31.132.5 > TRACE [GossipStage:1] 2016-10-17 11:17:05,595 > GossipDigestAckVerbHandler.java:84 - Sending a GossipDigestAck2Message to / > 172.31.150.150 > TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:889 - local > heartbeat version 1043 greater than 1042 for /172.31.126.216 > TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:889 - local > heartbeat version 1136 greater than 1133 for /172.31.126.217 > TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding > state SEVERITY: 0.5102040767669678 > TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:889 - local > heartbeat version 1230 greater than 1227 for /172.31.132.4 > TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding > state SEVERITY: 0.5128205418586731 > TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:889 - local > heartbeat version 1449 greater than 1446 for /172.31.150.150 > TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding > state SEVERITY: 0.5050504803657532 > TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:05,899 > GossipDigestAck2VerbHandler.java:38 - Received a GossipDigestAck2Message > from /172.31.132.5 > TRACE [GossipStage:1] 2016-10-17 11:17:05,899 FailureDetector.java:272 - > Average for /172.31.132.5 is 1.0157222197536764E9 > TRACE [GossipStage:1] 2016-10-17 11:17:05,899 Gossiper.java:1140 - / > 172.31.132.5 local generation 1476702594, remote generation 1476702594 > TRACE [GossipStage:1] 2016-10-17 11:17:05,899 Gossiper.java:1193 - > Updating heartbeat state version to 1329 from 1326 for /172.31.132.5 ... > TRACE [GossipStage:1] 2016-10-17 11:17:06,064 Gossiper.java:889 - local > heartbeat version 1043 greater than 1042 for /172.31.126.216 > TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:889 - local > heartbeat version 1230 greater than 1227 for /172.31.132.4 > TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:904 - Adding > state SEVERITY: 0.5128205418586731 > TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:889 - local > heartbeat version 1329 greater than 1326 for /172.31.132.5 > TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:904 - Adding > state SEVERITY: 0.5102040767669678 > TRACE [GossipStage:1] 2016-10-17 11:17:06,066 > GossipDigestAck2VerbHandler.java:38 - Received a GossipDigestAck2Message > from /172.31.126.217 > TRACE [GossipStage:1] 2016-10-17 11:17:06,066 FailureDetector.java:272 - > Average for /172.31.150.151 is 9.870286732358804E8 > TRACE [GossipStage:1] 2016-10-17 11:17:06,066 FailureDetector.java:272 - > Average for /172.31.126.217 is 9.973982727956204E8 > TRACE [GossipStage:1] 2016-10-17 11:17:06,066 Gossiper.java:1140 - / > 172.31.150.151 local generation 1476702405, remote generation 1476702405 > TRACE [GossipStage:1] 2016-10-17 11:17:06,066 Gossiper.java:1193 - > Updating heartbeat state version to 1906 from 1903 for /172.31.150.151 ... > TRACE [GossipStage:1] 2016-10-17 11:17:06,066 Gossiper.java:1140 - / > 172.31.126.217 local generation 1476702656, remote generation 1476702656 > TRACE [GossipStage:1] 2016-10-17 11:17:06,066 Gossiper.java:1193 - > Updating heartbeat state version to 1139 from 1136 for /172.31.126.217 ... > TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:144 - My > heartbeat is now 1046 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:500 - Gossip > Digests are : /172.31.132.4:1476702626:1230 /172.31.126.216:1476702687:1046 > /172.31.126.217:1476702656:1139 /172.31.132.5:1476702594:1329 > /172.31.150.151:1476702405:1906 /172.31.137.65:1433571405:232 > /172.31.142.111:1476347719:238 /172.31.150.150:1476702555:1449 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:646 - Sending > a GossipDigestSyn to /172.31.132.4 ... > TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:646 - Sending > a GossipDigestSyn to /172.31.126.217 ... > TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:757 - > Performing status check ... > TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 FailureDetector.java:298 - > PHI for /172.31.142.111 : 168.038111136 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 FailureDetector.java:303 - > Node /172.31.142.111 phi 72.97802441581015 > 12.0; intervals: > [2000000000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0] mean: 2.0E9 > DEBUG [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:337 - > Convicting /172.31.142.111 with status hibernate - alive false > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 - > PHI for /172.31.126.217 : 0.5312224569177403 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 - > PHI for /172.31.126.217 : 0.5312224569177403 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 - > mean for /172.31.126.217 : 9.973982727956204E8 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 - > PHI for /172.31.150.151 : 0.5370039365344064 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 - > PHI for /172.31.150.151 : 0.5370039365344064 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 - > mean for /172.31.150.151 : 9.870286732358804E8 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 - > PHI for /172.31.150.150 : 0.9719434636782321 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 - > PHI for /172.31.150.150 : 0.9719434636782321 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 - > mean for /172.31.150.150 : 1.0298713880042919E9 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 - > PHI for /172.31.132.5 : 0.6862971927197292 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 - > PHI for /172.31.132.5 : 0.6862971927197292 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 - > mean for /172.31.132.5 : 1.0157222197536764E9 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 - > PHI for /172.31.132.4 : 1.066833293270114 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 - > PHI for /172.31.132.4 : 1.066833293270114 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 - > mean for /172.31.132.4 : 1.0427801260213675E9 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 - > PHI for /172.31.137.65 : 168.037260214 > TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:303 - > Node /172.31.137.65 phi 72.97765486508104 > 12.0; intervals: [2000000000, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] > mean: 2.0E9 > DEBUG [GossipTasks:1] 2016-10-17 11:17:06,596 Gossiper.java:337 - > Convicting /172.31.137.65 with status hibernate - alive false > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 > GossipDigestAckVerbHandler.java:41 - Received a GossipDigestAckMessage from > /172.31.132.4 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 > GossipDigestAckVerbHandler.java:52 - Received ack with 4 digests and 2 > states > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 FailureDetector.java:272 - > Average for /172.31.150.150 is 1.029751850534188E9 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 FailureDetector.java:272 - > Average for /172.31.132.4 is 1.0430794573659575E9 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:1140 - / > 172.31.150.150 local generation 1476702555, remote generation 1476702555 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:1193 - > Updating heartbeat state version to 1452 from 1449 for /172.31.150.150 ... > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:1140 - / > 172.31.132.4 local generation 1476702626, remote generation 1476702626 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:1193 - > Updating heartbeat state version to 1233 from 1230 for /172.31.132.4 ... > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:889 - local > heartbeat version 1046 greater than 1043 for /172.31.126.216 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:889 - local > heartbeat version 1139 greater than 1136 for /172.31.126.217 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:904 - Adding > state SEVERITY: 0.0 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:889 - local > heartbeat version 1329 greater than 1326 for /172.31.132.5 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:904 - Adding > state SEVERITY: 0.5102040767669678 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:889 - local > heartbeat version 1906 greater than 1905 for /172.31.150.151 > TRACE [GossipStage:1] 2016-10-17 11:17:06,597 > GossipDigestAckVerbHandler.java:84 - Sending a GossipDigestAck2Message to / > 172.31.132.4 > TRACE [GossipStage:1] 2016-10-17 11:17:06,598 > GossipDigestAckVerbHandler.java:41 - Received a GossipDigestAckMessage from > /172.31.126.217 > TRACE [GossipStage:1] 2016-10-17 11:17:06,598 > GossipDigestAckVerbHandler.java:52 - Received ack with 1 digests and 0 > states > TRACE [GossipStage:1] 2016-10-17 11:17:06,598 Gossiper.java:889 - local > heartbeat version 1046 greater than 1045 for /172.31.126.216 > TRACE [GossipStage:1] 2016-10-17 11:17:06,598 > GossipDigestAckVerbHandler.java:84 - Sending a GossipDigestAck2Message to / > 172.31.126.217 On Thu, Oct 13, 2016 at 4:36 PM, Joel Knighton <joel.knigh...@datastax.com> wrote: > In the normal case, absolutely. That should happen quickly. I think it is > likely the case that you've hit another race condition where this phantom > node is not correctly marked as dead. In this case, even if it is removed > from some nodes in the cluster, it will get re-added by the node that > doesn't have the phantom node marked down. I can't say definitively without > some more information though. > > On Thu, Oct 13, 2016 at 7:11 AM, Kasper Petersen <kas...@sybogames.com> > wrote: > >> Thanks for the details. >> >> I don't know what happened on that node. It's a long time ago I think. I >> wasn't aware of it earlier. >> >> Will a node in hibernating state that failed joining and subsequently was >> discarded get removed from gossip at some point? >> >> On Wed, Oct 12, 2016 at 5:23 PM, Joel Knighton < >> joel.knigh...@datastax.com> wrote: >> >>> 1. A hibernating node is participating in gossip but intentionally >>> hasn't yet joined the ring. The two cases where a node would set a >>> hibernating status are when the node was started with >>> "-Dcassandra.join_ring=False" and has tokens or when the node was started >>> to replace another node (using "-Dcassandra.replace_address" or >>> "-Dcassandra.replace_address_first_boot"). >>> >>> 2. A rolling restart is probably your best bet. You may have more luck >>> with an assassinate in the case that you connect to a node that is not >>> continuously removing/adding the state. I suspect that this node will have >>> an alive status for this endpoint state. As usual, you should wield >>> assassinate with lots of caution. >>> >>> This issue sounds most similar to CASSANDRA-10371. If you provide >>> debugging information similar to that requested on the above ticket as well >>> as what operation you were performing on the node (was it a failed attempt >>> at replacing? etc) on a JIRA ticket, someone might have a chance to look >>> into this further. >>> >>> On Wed, Oct 12, 2016 at 9:48 AM, Kasper Petersen <kas...@sybogames.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I've recently upgraded our Cassandra cluster from 2.1 to 3.9. By >>>> default(?) 3.9 creates a debug.log file containing a ton of lines (a new >>>> one every second) with: >>>> >>>> DEBUG [GossipTasks:1] 2016-10-12 14:43:38,761 Gossiper.java:337 - >>>>> Convicting /172.31.137.65 with status hibernate - alive false >>>> >>>> >>>> That node has not been around for a very long time now. >>>> >>>> It does not show up in nodetool status and nodetool gossipinfo returns >>>> the following output about that node: >>>> >>>> /172.31.137.65 >>>>> generation:1433571405 >>>>> heartbeat:232 >>>>> STATUS:3:hibernate,true >>>>> LOAD:225:96445.0 >>>>> SCHEMA:53:e2d1a288-581c-3f35-b492-1b9d5a803631 >>>>> DC:9:us-east >>>>> RACK:11:1b >>>>> RELEASE_VERSION:7:2.1.5 >>>>> RPC_ADDRESS:6:172.31.137.65 >>>>> SEVERITY:231:0.2512562870979309 >>>>> NET_VERSION:4:8 >>>>> HOST_ID:5:7988d3c9-dec8-4b71-b5a9-0b962aad0680 >>>>> TOKENS:2:<hidden> >>>> >>>> >>>> nodetool removenode 7988d3c9-dec8-4b71-b5a9-0b962aad0680 resulted in: >>>> >>>> error: Host ID not found. >>>>> >>>> >>>> Now my questions are: >>>> >>>> 1. What does it mean for a node to be "hibernating"? How does it >>>> end up in that state? >>>> 2. How do I get rid of it? Its not coming back. >>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> Kasper Middelboe Petersen >>>> >>>> *Lead Backend Developer* >>>> >>>> *SYBO Games ApS* >>>> Jorcks Passage 1A, 4th. >>>> 1162 Copenhagen K >>>> >>> >>> >> >> >> -- >> Best regards, >> Kasper Middelboe Petersen >> >> *Lead Backend Developer* >> >> *SYBO Games ApS* >> Jorcks Passage 1A, 4th. >> 1162 Copenhagen K >> > > -- Best regards, Kasper Middelboe Petersen *Lead Backend Developer* *SYBO Games ApS* Jorcks Passage 1A, 4th. 1162 Copenhagen K