I resolved this by doing more rolling restarts on the nodes that had this
WARN.... it's just more restarts than I thought I would have to do.
On Wed, Oct 12, 2016 at 1:08 PM, Yucheng Liu <y...@rhapsody.com> wrote:
> *Env: * apache cassandra 2.1.8, 6-nodes
> *Problem: *one node had kernel panic and crashed twice this morning.
> seems gossip generation was messed up. all nodes are flooded with
> "received an invalid gossip generation for peer" warning messages.
> multiple rolling restarts only fixed "nodetool status". the warning
> messages are still happening. and gossipinfo is showing "shutdown" for some
> *Question: *Does anyone know how to get rid of this warning messages?
> take the whole cluster down is not desired as this is production.
> *$ nodetool gossipinfo|grep STATUS*
> WARN [GossipStage:1] 2016-10-12 09:58:02,913 Gossiper.java:1078 -
> received an invalid gossip generation for peer /10.150.12.118; local
> generation = 1442622223, received generation = 1476286723
> cqlsh> select gossip_generation from system.local;
> *1476286662 (I don't see where 1442622223 is from...)*