*Env: * apache cassandra 2.1.8, 6-nodes
*Problem: *one node had kernel panic and crashed twice this morning. seems
gossip generation was messed up. all nodes are flooded with "received an
invalid gossip generation for peer" warning messages. multiple rolling
restarts only fixed "nodetool status". the warning messages are still
happening. and gossipinfo is showing "shutdown" for some nodes.
*Question: *Does anyone know how to get rid of this warning messages?
take the whole cluster down is not desired as this is production.
*$ nodetool gossipinfo|grep STATUS*
STATUS:NORMAL,3074457345618258600
STATUS:NORMAL,-3074457345618258604
STATUS:NORMAL,6148914691236517202
STATUS:shutdown,true
STATUS:shutdown,true
STATUS:NORMAL,-9223372036854775808
*system.log:*
WARN [GossipStage:1] 2016-10-12 09:58:02,913 Gossiper.java:1078 - received
an invalid gossip generation for peer /10.150.12.118; local generation =
1442622223, received generation = 1476286723
cqlsh> select gossip_generation from system.local;
*1476286662 (I don't see where 1442622223 is from...)*
