Re: Gossip status: hibernate

Kasper Petersen Mon, 17 Oct 2016 04:28:05 -0700

I started up a new node and did a "-Dcassandra.replace_address" which I
terminated before it could join. This has resulted in one more hibernating
node similar to the other one that isn't disappearing after 72 hours.


I've added trace level logging to a working node to produce the following
(I'm not sure what I'm looking for here, so this is just everything for a
few seconds):

TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:889 - local
> heartbeat version 1227 greater than 1224 for /172.31.132.4
> TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:904 - Adding
> state SEVERITY: 0.5076141953468323
> TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:889 - local
> heartbeat version 1326 greater than 1325 for /172.31.132.5
> TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:889 - local
> heartbeat version 1903 greater than 1900 for /172.31.150.151
> TRACE [GossipStage:1] 2016-10-17 11:17:05,064 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:05,065
> GossipDigestAck2VerbHandler.java:38 - Received a GossipDigestAck2Message
> from /172.31.126.217
> TRACE [GossipStage:1] 2016-10-17 11:17:05,065 FailureDetector.java:272 -
> Average for /172.31.126.217 is 9.973875527435898E8
> TRACE [GossipStage:1] 2016-10-17 11:17:05,065 Gossiper.java:1140 - /
> 172.31.126.217 local generation 1476702656, remote generation 1476702656
> TRACE [GossipStage:1] 2016-10-17 11:17:05,065 Gossiper.java:1193 -
> Updating heartbeat state version to 1136 from 1133 for /172.31.126.217 ...
> TRACE [GossipStage:1] 2016-10-17 11:17:05,482 Gossiper.java:889 - local
> heartbeat version 1040 greater than 1037 for /172.31.126.216
> TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:889 - local
> heartbeat version 1136 greater than 1133 for /172.31.126.217
> TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:904 - Adding
> state SEVERITY: 0.5102040767669678
> TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:889 - local
> heartbeat version 1446 greater than 1443 for /172.31.150.150
> TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:05,483 Gossiper.java:889 - local
> heartbeat version 1903 greater than 1902 for /172.31.150.151
> TRACE [GossipStage:1] 2016-10-17 11:17:05,484
> GossipDigestAck2VerbHandler.java:38 - Received a GossipDigestAck2Message
> from /172.31.132.4
> TRACE [GossipStage:1] 2016-10-17 11:17:05,484 FailureDetector.java:272 -
> Average for /172.31.132.4 is 1.0427801260213675E9
> TRACE [GossipStage:1] 2016-10-17 11:17:05,484 Gossiper.java:1140 - /
> 172.31.132.4 local generation 1476702626, remote generation 1476702626
> TRACE [GossipStage:1] 2016-10-17 11:17:05,484 Gossiper.java:1193 -
> Updating heartbeat state version to 1230 from 1227 for /172.31.132.4 ...
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 Gossiper.java:144 - My
> heartbeat is now 1043
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 Gossiper.java:500 - Gossip
> Digests are : /172.31.150.151:1476702405:1903 /172.31.132.5:1476702594:1326
> /172.31.126.217:1476702656:1136 /172.31.126.216:1476702687:1043
> /172.31.142.111:1476347719:238 /172.31.137.65:1433571405:232
> /172.31.132.4:1476702626:1230 /172.31.150.150:1476702555:1446
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 Gossiper.java:646 - Sending
> a GossipDigestSyn to /172.31.150.150 ...
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 Gossiper.java:757 -
> Performing status check ...
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,593 FailureDetector.java:298 -
> PHI for /172.31.142.111 : 167.537185959
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:303 -
> Node /172.31.142.111 phi 72.76047537559265 > 12.0; intervals:
> [2000000000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0] mean: 2.0E9
> DEBUG [GossipTasks:1] 2016-10-17 11:17:05,594 Gossiper.java:337 -
> Convicting /172.31.142.111 with status hibernate - alive false
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 -
> PHI for /172.31.126.217 : 0.5296914569935668
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 -
> PHI for /172.31.126.217 : 0.5296914569935668
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 -
> mean for /172.31.126.217 : 9.973875527435898E8
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 -
> PHI for /172.31.150.151 : 0.9312981815294517
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 -
> PHI for /172.31.150.151 : 0.9312981815294517
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 -
> mean for /172.31.150.151 : 9.8568609518E8
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 -
> PHI for /172.31.150.150 : 0.9710661780597467
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 -
> PHI for /172.31.150.150 : 0.9710661780597467
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 -
> mean for /172.31.150.150 : 1.0299954375905173E9
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 -
> PHI for /172.31.132.5 : 0.6849131720784052
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 -
> PHI for /172.31.132.5 : 0.6849131720784052
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 -
> mean for /172.31.132.5 : 1.0157781925682657E9
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 -
> PHI for /172.31.132.4 : 0.10601682583058232
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:315 -
> PHI for /172.31.132.4 : 0.10601682583058232
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:316 -
> mean for /172.31.132.4 : 1.0427801260213675E9
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:298 -
> PHI for /172.31.137.65 : 167.5363108695
> TRACE [GossipTasks:1] 2016-10-17 11:17:05,594 FailureDetector.java:303 -
> Node /172.31.137.65 phi 72.76009532905162 > 12.0; intervals: [2000000000,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
> mean: 2.0E9
> DEBUG [GossipTasks:1] 2016-10-17 11:17:05,595 Gossiper.java:337 -
> Convicting /172.31.137.65 with status hibernate - alive false
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595
> GossipDigestAckVerbHandler.java:41 - Received a GossipDigestAckMessage from
> /172.31.150.150
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595
> GossipDigestAckVerbHandler.java:52 - Received ack with 4 digests and 2
> states
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 FailureDetector.java:272 -
> Average for /172.31.150.150 is 1.0298713880042919E9
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:1140 - /
> 172.31.150.151 local generation 1476702405, remote generation 1476702405
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:1193 -
> Updating heartbeat state version to 1903 from 1903 for /172.31.150.151 ...
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:1140 - /
> 172.31.150.150 local generation 1476702555, remote generation 1476702555
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:1193 -
> Updating heartbeat state version to 1449 from 1446 for /172.31.150.150 ...
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:889 - local
> heartbeat version 1043 greater than 1040 for /172.31.126.216
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:889 - local
> heartbeat version 1136 greater than 1133 for /172.31.126.217
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:904 - Adding
> state SEVERITY: 0.5102040767669678
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:889 - local
> heartbeat version 1230 greater than 1227 for /172.31.132.4
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:904 - Adding
> state SEVERITY: 0.5128205418586731
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595 Gossiper.java:889 - local
> heartbeat version 1326 greater than 1325 for /172.31.132.5
> TRACE [GossipStage:1] 2016-10-17 11:17:05,595
> GossipDigestAckVerbHandler.java:84 - Sending a GossipDigestAck2Message to /
> 172.31.150.150
> TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:889 - local
> heartbeat version 1043 greater than 1042 for /172.31.126.216
> TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:889 - local
> heartbeat version 1136 greater than 1133 for /172.31.126.217
> TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding
> state SEVERITY: 0.5102040767669678
> TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:889 - local
> heartbeat version 1230 greater than 1227 for /172.31.132.4
> TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding
> state SEVERITY: 0.5128205418586731
> TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:889 - local
> heartbeat version 1449 greater than 1446 for /172.31.150.150
> TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding
> state SEVERITY: 0.5050504803657532
> TRACE [GossipStage:1] 2016-10-17 11:17:05,898 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:05,899
> GossipDigestAck2VerbHandler.java:38 - Received a GossipDigestAck2Message
> from /172.31.132.5
> TRACE [GossipStage:1] 2016-10-17 11:17:05,899 FailureDetector.java:272 -
> Average for /172.31.132.5 is 1.0157222197536764E9
> TRACE [GossipStage:1] 2016-10-17 11:17:05,899 Gossiper.java:1140 - /
> 172.31.132.5 local generation 1476702594, remote generation 1476702594
> TRACE [GossipStage:1] 2016-10-17 11:17:05,899 Gossiper.java:1193 -
> Updating heartbeat state version to 1329 from 1326 for /172.31.132.5 ...
> TRACE [GossipStage:1] 2016-10-17 11:17:06,064 Gossiper.java:889 - local
> heartbeat version 1043 greater than 1042 for /172.31.126.216
> TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:889 - local
> heartbeat version 1230 greater than 1227 for /172.31.132.4
> TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:904 - Adding
> state SEVERITY: 0.5128205418586731
> TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:889 - local
> heartbeat version 1329 greater than 1326 for /172.31.132.5
> TRACE [GossipStage:1] 2016-10-17 11:17:06,065 Gossiper.java:904 - Adding
> state SEVERITY: 0.5102040767669678
> TRACE [GossipStage:1] 2016-10-17 11:17:06,066
> GossipDigestAck2VerbHandler.java:38 - Received a GossipDigestAck2Message
> from /172.31.126.217
> TRACE [GossipStage:1] 2016-10-17 11:17:06,066 FailureDetector.java:272 -
> Average for /172.31.150.151 is 9.870286732358804E8
> TRACE [GossipStage:1] 2016-10-17 11:17:06,066 FailureDetector.java:272 -
> Average for /172.31.126.217 is 9.973982727956204E8
> TRACE [GossipStage:1] 2016-10-17 11:17:06,066 Gossiper.java:1140 - /
> 172.31.150.151 local generation 1476702405, remote generation 1476702405
> TRACE [GossipStage:1] 2016-10-17 11:17:06,066 Gossiper.java:1193 -
> Updating heartbeat state version to 1906 from 1903 for /172.31.150.151 ...
> TRACE [GossipStage:1] 2016-10-17 11:17:06,066 Gossiper.java:1140 - /
> 172.31.126.217 local generation 1476702656, remote generation 1476702656
> TRACE [GossipStage:1] 2016-10-17 11:17:06,066 Gossiper.java:1193 -
> Updating heartbeat state version to 1139 from 1136 for /172.31.126.217 ...
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:144 - My
> heartbeat is now 1046
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:500 - Gossip
> Digests are : /172.31.132.4:1476702626:1230 /172.31.126.216:1476702687:1046
> /172.31.126.217:1476702656:1139 /172.31.132.5:1476702594:1329
> /172.31.150.151:1476702405:1906 /172.31.137.65:1433571405:232
> /172.31.142.111:1476347719:238 /172.31.150.150:1476702555:1449
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:646 - Sending
> a GossipDigestSyn to /172.31.132.4 ...
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:646 - Sending
> a GossipDigestSyn to /172.31.126.217 ...
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:757 -
> Performing status check ...
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 FailureDetector.java:298 -
> PHI for /172.31.142.111 : 168.038111136
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,595 FailureDetector.java:303 -
> Node /172.31.142.111 phi 72.97802441581015 > 12.0; intervals:
> [2000000000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0] mean: 2.0E9
> DEBUG [GossipTasks:1] 2016-10-17 11:17:06,595 Gossiper.java:337 -
> Convicting /172.31.142.111 with status hibernate - alive false
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 -
> PHI for /172.31.126.217 : 0.5312224569177403
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 -
> PHI for /172.31.126.217 : 0.5312224569177403
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 -
> mean for /172.31.126.217 : 9.973982727956204E8
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 -
> PHI for /172.31.150.151 : 0.5370039365344064
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 -
> PHI for /172.31.150.151 : 0.5370039365344064
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 -
> mean for /172.31.150.151 : 9.870286732358804E8
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 -
> PHI for /172.31.150.150 : 0.9719434636782321
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 -
> PHI for /172.31.150.150 : 0.9719434636782321
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 -
> mean for /172.31.150.150 : 1.0298713880042919E9
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 -
> PHI for /172.31.132.5 : 0.6862971927197292
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 -
> PHI for /172.31.132.5 : 0.6862971927197292
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 -
> mean for /172.31.132.5 : 1.0157222197536764E9
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 -
> PHI for /172.31.132.4 : 1.066833293270114
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:315 -
> PHI for /172.31.132.4 : 1.066833293270114
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:316 -
> mean for /172.31.132.4 : 1.0427801260213675E9
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:298 -
> PHI for /172.31.137.65 : 168.037260214
> TRACE [GossipTasks:1] 2016-10-17 11:17:06,596 FailureDetector.java:303 -
> Node /172.31.137.65 phi 72.97765486508104 > 12.0; intervals: [2000000000,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
> mean: 2.0E9
> DEBUG [GossipTasks:1] 2016-10-17 11:17:06,596 Gossiper.java:337 -
> Convicting /172.31.137.65 with status hibernate - alive false
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597
> GossipDigestAckVerbHandler.java:41 - Received a GossipDigestAckMessage from
> /172.31.132.4
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597
> GossipDigestAckVerbHandler.java:52 - Received ack with 4 digests and 2
> states
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 FailureDetector.java:272 -
> Average for /172.31.150.150 is 1.029751850534188E9
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 FailureDetector.java:272 -
> Average for /172.31.132.4 is 1.0430794573659575E9
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:1140 - /
> 172.31.150.150 local generation 1476702555, remote generation 1476702555
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:1193 -
> Updating heartbeat state version to 1452 from 1449 for /172.31.150.150 ...
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:1140 - /
> 172.31.132.4 local generation 1476702626, remote generation 1476702626
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:1193 -
> Updating heartbeat state version to 1233 from 1230 for /172.31.132.4 ...
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:889 - local
> heartbeat version 1046 greater than 1043 for /172.31.126.216
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:889 - local
> heartbeat version 1139 greater than 1136 for /172.31.126.217
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:904 - Adding
> state SEVERITY: 0.0
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:889 - local
> heartbeat version 1329 greater than 1326 for /172.31.132.5
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:904 - Adding
> state SEVERITY: 0.5102040767669678
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597 Gossiper.java:889 - local
> heartbeat version 1906 greater than 1905 for /172.31.150.151
> TRACE [GossipStage:1] 2016-10-17 11:17:06,597
> GossipDigestAckVerbHandler.java:84 - Sending a GossipDigestAck2Message to /
> 172.31.132.4
> TRACE [GossipStage:1] 2016-10-17 11:17:06,598
> GossipDigestAckVerbHandler.java:41 - Received a GossipDigestAckMessage from
> /172.31.126.217
> TRACE [GossipStage:1] 2016-10-17 11:17:06,598
> GossipDigestAckVerbHandler.java:52 - Received ack with 1 digests and 0
> states
> TRACE [GossipStage:1] 2016-10-17 11:17:06,598 Gossiper.java:889 - local
> heartbeat version 1046 greater than 1045 for /172.31.126.216
> TRACE [GossipStage:1] 2016-10-17 11:17:06,598
> GossipDigestAckVerbHandler.java:84 - Sending a GossipDigestAck2Message to /
> 172.31.126.217



On Thu, Oct 13, 2016 at 4:36 PM, Joel Knighton <joel.knigh...@datastax.com>
wrote:

> In the normal case, absolutely. That should happen quickly. I think it is
> likely the case that you've hit another race condition where this phantom
> node is not correctly marked as dead. In this case, even if it is removed
> from some nodes in the cluster, it will get re-added by the node that
> doesn't have the phantom node marked down. I can't say definitively without
> some more information though.
>
> On Thu, Oct 13, 2016 at 7:11 AM, Kasper Petersen <kas...@sybogames.com>
> wrote:
>
>> Thanks for the details.
>>
>> I don't know what happened on that node. It's a long time ago I think. I
>> wasn't aware of it earlier.
>>
>> Will a node in hibernating state that failed joining and subsequently was
>> discarded get removed from gossip at some point?
>>
>> On Wed, Oct 12, 2016 at 5:23 PM, Joel Knighton <
>> joel.knigh...@datastax.com> wrote:
>>
>>> 1. A hibernating node is participating in gossip but intentionally
>>> hasn't yet joined the ring. The two cases where a node would set a
>>> hibernating status are when the node was started with
>>> "-Dcassandra.join_ring=False" and has tokens or when the node was started
>>> to replace another node (using "-Dcassandra.replace_address" or
>>> "-Dcassandra.replace_address_first_boot").
>>>
>>> 2. A rolling restart is probably your best bet. You may have more luck
>>> with an assassinate in the case that you connect to a node that is not
>>> continuously removing/adding the state. I suspect that this node will have
>>> an alive status for this endpoint state. As usual, you should wield
>>> assassinate with lots of caution.
>>>
>>> This issue sounds most similar to CASSANDRA-10371. If you provide
>>> debugging information similar to that requested on the above ticket as well
>>> as what operation you were performing on the node (was it a failed attempt
>>> at replacing? etc) on a JIRA ticket, someone might have a chance to look
>>> into this further.
>>>
>>> On Wed, Oct 12, 2016 at 9:48 AM, Kasper Petersen <kas...@sybogames.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I've recently upgraded our Cassandra cluster from 2.1 to 3.9. By
>>>> default(?) 3.9 creates a debug.log file containing a ton of lines (a new
>>>> one every second) with:
>>>>
>>>> DEBUG [GossipTasks:1] 2016-10-12 14:43:38,761 Gossiper.java:337 -
>>>>> Convicting /172.31.137.65 with status hibernate - alive false
>>>>
>>>>
>>>> That node has not been around for a very long time now.
>>>>
>>>> It does not show up in nodetool status and nodetool gossipinfo returns
>>>> the following output about that node:
>>>>
>>>> /172.31.137.65
>>>>>   generation:1433571405
>>>>>   heartbeat:232
>>>>>   STATUS:3:hibernate,true
>>>>>   LOAD:225:96445.0
>>>>>   SCHEMA:53:e2d1a288-581c-3f35-b492-1b9d5a803631
>>>>>   DC:9:us-east
>>>>>   RACK:11:1b
>>>>>   RELEASE_VERSION:7:2.1.5
>>>>>   RPC_ADDRESS:6:172.31.137.65
>>>>>   SEVERITY:231:0.2512562870979309
>>>>>   NET_VERSION:4:8
>>>>>   HOST_ID:5:7988d3c9-dec8-4b71-b5a9-0b962aad0680
>>>>>   TOKENS:2:<hidden>
>>>>
>>>>
>>>> nodetool removenode 7988d3c9-dec8-4b71-b5a9-0b962aad0680 resulted in:
>>>>
>>>> error: Host ID not found.
>>>>>
>>>>
>>>> Now my questions are:
>>>>
>>>>    1. What does it mean for a node to be "hibernating"? How does it
>>>>    end up in that state?
>>>>    2. How do I get rid of it? Its not coming back.
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Kasper Middelboe Petersen
>>>>
>>>> *Lead Backend Developer*
>>>>
>>>> *SYBO Games ApS*
>>>> Jorcks Passage 1A, 4th.
>>>> 1162 Copenhagen K
>>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>> Kasper Middelboe Petersen
>>
>> *Lead Backend Developer*
>>
>> *SYBO Games ApS*
>> Jorcks Passage 1A, 4th.
>> 1162 Copenhagen K
>>
>
>


-- 
Best regards,
Kasper Middelboe Petersen

*Lead Backend Developer*

*SYBO Games ApS*
Jorcks Passage 1A, 4th.
1162 Copenhagen K

Re: Gossip status: hibernate

Reply via email to