Hi, We are running a production Cassandra version (3.0.14) with 256 tokens v-node configuration. Occasionally, we see that different nodes show different ownership for the same key. Only a node restart corrects; otherwise, it continues to behave in a split-brain.
Say, for example, *NodeA* nodetool getendpoints ks1 table1 10 - n1 - n2 - n3 *NodeB* nodetool getendpoints ks1 table1 10 - n1 - n2 *- n5* If I restart NodeB, then it shows the correct ownership {n1,n2,n3}. The majority of the nodes in the ring show correct ownership {n1,n2,n3}, only a few show this issue, and restarting them solves the problem. To me, it seems I think Cassandra's Gossip cache and StorageService cache (TokenMetadata) are having some sort of cache coherence. Anyone has observed this behavior? Any help would be highly appreciated. Jaydeep