RE: nodes moving spontaneously
Thanks for the tip. Checked nodetool ring on all nodes and they all have a consistent view of the ring. We have had other problems like nodes crashing etc so anything could have happened, but we're sure we didnt issue a nodetool move command. From: Tyler Hobbs [mailto:ty...@datastax.com] OpsCenter just periodically calls describe_ring() on different nodes in the cluster, so that's how it's getting that information. Maybe try running nodetool ring on each node in your cluster to make sure they all have the same view of the ring?
Re: nodes moving spontaneously
Nodes don't move themselves, you likely have some kind of 'bouncing gossip' issue where a node was removed/replaced and is hanging around, but only periodically held in statee between nodes. Unfortunately node removal is very prone to this before 0.8.3 and even after that, you can't fix it without https://issues.apache.org/jira/browse/CASSANDRA-3337 so upgrading is really your best bet here. On Fri, Jun 1, 2012 at 4:01 PM, Curt Allred c...@mediosystems.com wrote: We have a 10 node cluster (v0.7.9) split into 2 datacenters. Three times we have seen nodes move themselves to different locations in the ring. In each case, the move unbalanced the ring. In one case a node moved to the opposite side of the ring. Sometime after the first spontaneous move we started using Datastax OpsCenter. The next 2 moves showed up in its event log like: 5/20/2012 11:23am - Info - Host 12.34.56.78 moved from '12345' to '54321' where '12345' and '54321' are the old and new tokens. Anyone know whats causing this?
nodes moving spontaneously
We have a 10 node cluster (v0.7.9) split into 2 datacenters. Three times we have seen nodes move themselves to different locations in the ring. In each case, the move unbalanced the ring. In one case a node moved to the opposite side of the ring. Sometime after the first spontaneous move we started using Datastax OpsCenter. The next 2 moves showed up in its event log like: 5/20/2012 11:23am - Info - Host 12.34.56.78 moved from '12345' to '54321' where '12345' and '54321' are the old and new tokens. Anyone know whats causing this?
Re: nodes moving spontaneously
OpsCenter just periodically calls describe_ring() on different nodes in the cluster, so that's how it's getting that information. Maybe try running nodetool ring on each node in your cluster to make sure they all have the same view of the ring? On Fri, Jun 1, 2012 at 4:01 PM, Curt Allred c...@mediosystems.com wrote: We have a 10 node cluster (v0.7.9) split into 2 datacenters. Three times we have seen nodes move themselves to different locations in the ring. In each case, the move unbalanced the ring. In one case a node moved to the opposite side of the ring. ** ** Sometime after the first spontaneous move we started using Datastax OpsCenter. The next 2 moves showed up in its event log like: 5/20/2012 11:23am - Info - Host 12.34.56.78 moved from '12345' to '54321' ** ** where '12345' and '54321' are the old and new tokens. ** ** Anyone know whats causing this? ** ** -- Tyler Hobbs DataStax http://datastax.com/
