I don't think that's what I'm after here since the unwanted nodes were originally assimilated into the cluster with the same initial_token values as other nodes that were already in the cluster (that have, and still do have, useful data). I know this is an awkward situation so I'll try to depict it in a simpler way:
Let's say I have a simplified version of our production cluster that looks like this - cass-1 token = A cass-2 token = B cass-3 token = C Then I tried to create a second cluster that looks like this - cass-analysis-1 token = A (and contains same data as cass-1) cass-analysis-2 token = B (and contains same data as cass-2) cass-analysis-3 token = C (and contains same data as cass-3) But after starting the second cluster, things got crossed up between the clusters and here's what the original cluster now looks like - cass-1 token = A (has data and schema) cass-2 token = B (has data and schema) cass-3 token = C (had data and schema) cass-analysis-1 token = A (has *no* data and is not part of the ring, but is trying to be included in cluster schema) A simplified version of "describe cluster" for the original cluster now shows: Cluster Information: Schema versions: SCHEMA-UUID-1: [cass-1, cass-2, cass-3] SCHEMA-UUID-2: [cass-analysis-1] But the simplified ring looks like this (has only 3 nodes instead of 4): Host Owns Token cass-1 33% A cass-2 33% B cass-3 33% C The original cluster is still working correctly but all live schema updates are failing because of the inconsistent schema versions introduced by the unwanted node. >From my perspective, a simple fix seems to be for cassandra to exclude nodes that aren't part of the ring from the schema consistency requirements. Any reason that wouldn't work? And aside from a possible code patch, any recommendations as to how I can best fix this given the current 8.4 release? On Thu, Oct 13, 2011 at 12:14 AM, Jonathan Ellis <jbel...@gmail.com> wrote: > Does nodetool removetoken not work? > > On Thu, Oct 13, 2011 at 12:59 AM, Eric Czech <e...@nextbigsound.com> > wrote: > > Not sure if anyone has seen this before but it's really killing me right > > now. Perhaps that was too long of a description of the issue so here's a > > more succinct question -- How do I remove nodes associated with a cluster > > that contain no data and have no reason to be associated with the cluster > > whatsoever? > > My last resort here is to stop cassandra (after recording all tokens for > > each node), set the initial token for each node in the cluster in > > cassandra.yaml, manually delete the LocationInfo* sstables in the system > > keyspace, and then restart. I'm hoping there's a simpler, less seemingly > > risky way to do this so please, please let me know if that's true! > > Thanks again. > > - Eric > > On Tue, Oct 11, 2011 at 11:55 AM, Eric Czech <e...@nextbigsound.com> > wrote: > >> > >> Hi, I'm having what I think is a fairly uncommon schema issue -- > >> My situation is that I had a cluster with 10 nodes and a consistent > >> schema. Then, in an experiment to setup a second cluster with the same > >> information (by copying the raw sstables), I left the LocationInfo* > sstables > >> in the system keyspace in the new cluster and after starting the second > >> cluster, I realized that the two clusters were discovering each other > when > >> they shouldn't have been. Since then, I changed the cluster name for > the > >> second cluster and made sure to delete the LocationInfo* sstables before > >> starting it and the two clusters are now operating independent of one > >> another for the most part. The only remaining connection between the > two > >> seems to be that the first cluster is still maintaining references to > nodes > >> in the second cluster in the schema versions despite those nodes not > >> actually being part of the ring. > >> Here's what my "describe cluster" looks like on the original cluster: > >> Cluster Information: > >> Snitch: org.apache.cassandra.locator.SimpleSnitch > >> Partitioner: org.apache.cassandra.dht.RandomPartitioner > >> Schema versions: > >> 48971cb0-e9ff-11e0-0000-eb9eab7d90bf: [<INTENTIONAL_IP1>, > >> <INTENTIONAL_IP2>, ..., <INTENTIONAL_IP10>] > >> 848bcfc0-eddf-11e0-0000-8a3bb58f08ff: [<NOT_INTENTIONAL_IP1>, > >> <NOT_INTENTIONAL_IP2>] > >> The second cluster, however, contains no schema versions involving nodes > >> from the first cluster. > >> My question then is, how can I remove those schema versions from the > >> original cluster that are associated with the unwanted nodes from the > second > >> cluster? Is there any way to remove or evict an IP from a cluster > instead > >> of just a token? > >> Thanks in advance! > >> - Eric > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
