I resolved this by doing more rolling restarts on the nodes that had this
WARN.... it's just more restarts than I thought I would have to do.
 annoying!

On Wed, Oct 12, 2016 at 1:08 PM, Yucheng Liu <y...@rhapsody.com> wrote:

> *Env: * apache cassandra 2.1.8, 6-nodes
>
> *Problem: *one node had kernel panic and crashed twice this morning.
>  seems gossip generation was messed up.  all nodes are flooded with
> "received an invalid gossip generation for peer" warning messages.
>  multiple rolling restarts only fixed "nodetool status".  the warning
> messages are still happening. and gossipinfo is showing "shutdown" for some
> nodes.
>
> *Question: *Does anyone know how to get rid of this warning messages?
> take the whole cluster down is not desired as this is production.
>
> *$ nodetool gossipinfo|grep STATUS*
>   STATUS:NORMAL,3074457345618258600
>   STATUS:NORMAL,-3074457345618258604
>   STATUS:NORMAL,6148914691236517202
>   STATUS:shutdown,true
>   STATUS:shutdown,true
>   STATUS:NORMAL,-9223372036854775808
>
> *system.log:*
>
> WARN  [GossipStage:1] 2016-10-12 09:58:02,913 Gossiper.java:1078 -
> received an invalid gossip generation for peer /10.150.12.118; local
> generation = 1442622223, received generation = 1476286723
>
> cqlsh> select gossip_generation from system.local;
>
>         *1476286662   (I don't see where 1442622223 is from...)*
>
>
>

Reply via email to