Re: corrupted gossip generation

[Yucheng Liu]

I resolved this by doing more rolling restarts on the nodes that had this
WARN it's just more restarts than I thought I would have to do.
 annoying!

On Wed, Oct 12, 2016 at 1:08 PM, Yucheng Liu  wrote:

> *Env: * apache cassandra 2.1.8, 6-nodes
>
> *Problem: *one node had kernel panic and crashed twice this morning.
>  seems gossip generation was messed up.  all nodes are flooded with
> "received an invalid gossip generation for peer" warning messages.
>  multiple rolling restarts only fixed "nodetool status".  the warning
> messages are still happening. and gossipinfo is showing "shutdown" for some
> nodes.
>
> *Question: *Does anyone know how to get rid of this warning messages?
> take the whole cluster down is not desired as this is production.
>
> *$ nodetool gossipinfo|grep STATUS*
>   STATUS:NORMAL,3074457345618258600
>   STATUS:NORMAL,-3074457345618258604
>   STATUS:NORMAL,6148914691236517202
>   STATUS:shutdown,true
>   STATUS:shutdown,true
>   STATUS:NORMAL,-9223372036854775808
>
> *system.log:*
>
> WARN  [GossipStage:1] 2016-10-12 09:58:02,913 Gossiper.java:1078 -
> received an invalid gossip generation for peer /10.150.12.118; local
> generation = 144263, received generation = 1476286723
>
> cqlsh> select gossip_generation from system.local;
>
> *1476286662   (I don't see where 144263 is from...)*
>
>
>

corrupted gossip generation

[Yucheng Liu]

*Env: * apache cassandra 2.1.8, 6-nodes

*Problem: *one node had kernel panic and crashed twice this morning.  seems
gossip generation was messed up.  all nodes are flooded with "received an
invalid gossip generation for peer" warning messages.  multiple rolling
restarts only fixed "nodetool status".  the warning messages are still
happening. and gossipinfo is showing "shutdown" for some nodes.

*Question: *Does anyone know how to get rid of this warning messages?
take the whole cluster down is not desired as this is production.

*$ nodetool gossipinfo|grep STATUS*
  STATUS:NORMAL,3074457345618258600
  STATUS:NORMAL,-3074457345618258604
  STATUS:NORMAL,6148914691236517202
  STATUS:shutdown,true
  STATUS:shutdown,true
  STATUS:NORMAL,-9223372036854775808

*system.log:*

WARN  [GossipStage:1] 2016-10-12 09:58:02,913 Gossiper.java:1078 - received
an invalid gossip generation for peer /10.150.12.118; local generation =
144263, received generation = 1476286723

cqlsh> select gossip_generation from system.local;

*1476286662   (I don't see where 144263 is from...)*