Re:Re: Re: Re: stream failed when bootstrap
Only do rolling restart can't solve problem. But I thought I found a way. There are two same name folder with different suffix in data dictionary. e.g. dayu_123 and dayu_234, the dayu_123 folder is empty and dayu_234 folder is not. then I use cql to query system_schema.tables，the id of table named 'dayu' is 123。 because the data in table 'dayu' can be re-generate. I simplly remove dayu_234 folder when rolling restart. Now when running nodetool resetlocalschema, there is no error. and I am bootstrap new node again. I will see is there any other problem left. I hope my experience can help others. Thanks for all your help! Dayu At 2018-06-28 13:53:16, "kurt greaves" wrote: Yeah, but you only really need to drain, restart Cassandra one by one. Not that the others will hurt, but they aren't strictly necessary. On 28 June 2018 at 05:38, dayu wrote: Hi kurt, a rolling restart means run disablebinary, disablethrift, disablegossip, drain, stop cassandra and start cassandra command one by one, right? Only one node is executed at a time Dayu At 2018-06-28 11:37:43, "kurt greaves" wrote: Best off trying a rolling restart. On 28 June 2018 at 03:18, dayu wrote: the output of nodetool describecluster Cluster Information: Name: online-xxx Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Schema versions: c3f00d61-1ad7-3702-8703-af2a29e401c1: [10.136.71.43] 0568e8c1-48ba-3fb0-bb3c-462438978d7b: [10.136.71.33, ] after I run nodetool resetlocalschema, a error log outcome ERROR [InternalResponseStage:209417] 2018-06-28 11:14:12,904 MigrationTask.java:96 - Configuration exception merging remote schema org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 5552bba0-2 dc6-11e8-9b5c-254242d97235; expected 53f6d520-2dc6-11e8-948d-ab7caa3c8c36) at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:790) ~[apac he-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:750) ~[apache-cassandra-3.0 .10.jar:3.0.10] at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[apache-cassandra-3.0.1 0.jar:3.0.10] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1348) ~[ap ache-cassandra-3.0.10.jar:3.0.10] At 2018-06-28 10:01:52, "Jeff Jirsa" wrote: You can sometimes bounce your way through it (or use nodetool resetlocalschema if it’s a single node that’s wrong), but there are some edge cases from which it’s very hard to recover What’s the output of nodetool describecluster? If you select from the schema tables, do you see that CFID on any real tables? -- Jeff Jirsa On Jun 27, 2018, at 7:58 PM, dayu wrote: That sound reasonable, I have seen schema mismatch error before. So any advise to deal with schema mismatches? Dayu At 2018-06-28 09:50:37, "Jeff Jirsa" wrote: >That log message says you did: > > CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming > >If you’re absolutely sure you didn’t, you should look for schema mismatches in >your cluster > > >-- >Jeff Jirsa > > >> On Jun 27, 2018, at 7:49 PM, dayu wrote: >> >> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming > >- >To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Re: Re: stream failed when bootstrap
Yeah, but you only really need to drain, restart Cassandra one by one. Not that the others will hurt, but they aren't strictly necessary. On 28 June 2018 at 05:38, dayu wrote: > Hi kurt, a rolling restart means run disablebinary, disablethrift, > disablegossip, drain, > stop cassandra and start cassandra command one by one, right? > Only one node is executed at a time > > Dayu > > > > At 2018-06-28 11:37:43, "kurt greaves" wrote: > > Best off trying a rolling restart. > > On 28 June 2018 at 03:18, dayu wrote: > >> the output of nodetool describecluster >> Cluster Information: >> Name: online-xxx >> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch >> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner >> Schema versions: >> c3f00d61-1ad7-3702-8703-af2a29e401c1: [10.136.71.43] >> >> 0568e8c1-48ba-3fb0-bb3c-462438978d7b: [10.136.71.33, ] >> >> after I run nodetool resetlocalschema, a error log outcome >> >> ERROR [InternalResponseStage:209417] 2018-06-28 11:14:12,904 >> MigrationTask.java:96 - Configuration >> exception merging remote schema >> org.apache.cassandra.exceptions.ConfigurationException: Column family ID >> mismatch (found 5552bba0-2 >> dc6-11e8-9b5c-254242d97235; expected 53f6d520-2dc6-11e8-948d-ab7caa >> 3c8c36) >> at >> org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:790) >> ~[apac >> he-cassandra-3.0.10.jar:3.0.10] >> at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:750) >> ~[apache-cassandra-3.0 >> .10.jar:3.0.10] >> at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) >> ~[apache-cassandra-3.0.1 >> 0.jar:3.0.10] >> at >> org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1348) >> ~[ap >> ache-cassandra-3.0.10.jar:3.0.10] >> >> >> >> >> >> At 2018-06-28 10:01:52, "Jeff Jirsa" wrote: >> >> You can sometimes bounce your way through it (or use nodetool >> resetlocalschema if it’s a single node that’s wrong), but there are some >> edge cases from which it’s very hard to recover >> >> What’s the output of nodetool describecluster? >> >> If you select from the schema tables, do you see that CFID on any real >> tables? >> >> -- >> Jeff Jirsa >> >> >> On Jun 27, 2018, at 7:58 PM, dayu wrote: >> >> That sound reasonable, I have seen schema mismatch error before. >> So any advise to deal with schema mismatches? >> >> Dayu >> >> At 2018-06-28 09:50:37, "Jeff Jirsa" wrote: >> >That log message says you did: >> > >> > CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming >> > >> >If you’re absolutely sure you didn’t, you should look for schema mismatches >> >in your cluster >> > >> > >> >-- >> >Jeff Jirsa >> > >> > >> >> On Jun 27, 2018, at 7:49 PM, dayu wrote: >> >> >> >> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming >> > >> >- >> >To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> >For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >> >> >> >> >> >> >> > > > > >
Re: Re: stream failed when bootstrap
Best off trying a rolling restart. On 28 June 2018 at 03:18, dayu wrote: > the output of nodetool describecluster > Cluster Information: > Name: online-xxx > Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch > Partitioner: org.apache.cassandra.dht.Murmur3Partitioner > Schema versions: > c3f00d61-1ad7-3702-8703-af2a29e401c1: [10.136.71.43] > > 0568e8c1-48ba-3fb0-bb3c-462438978d7b: [10.136.71.33, ] > > after I run nodetool resetlocalschema, a error log outcome > > ERROR [InternalResponseStage:209417] 2018-06-28 11:14:12,904 > MigrationTask.java:96 - Configuration > exception merging remote schema > org.apache.cassandra.exceptions.ConfigurationException: Column family ID > mismatch (found 5552bba0-2 > dc6-11e8-9b5c-254242d97235; expected 53f6d520-2dc6-11e8-948d-ab7caa3c8c36) > at > org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:790) > ~[apac > he-cassandra-3.0.10.jar:3.0.10] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:750) > ~[apache-cassandra-3.0 > .10.jar:3.0.10] > at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) > ~[apache-cassandra-3.0.1 > 0.jar:3.0.10] > at > org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1348) > ~[ap > ache-cassandra-3.0.10.jar:3.0.10] > > > > > > At 2018-06-28 10:01:52, "Jeff Jirsa" wrote: > > You can sometimes bounce your way through it (or use nodetool > resetlocalschema if it’s a single node that’s wrong), but there are some > edge cases from which it’s very hard to recover > > What’s the output of nodetool describecluster? > > If you select from the schema tables, do you see that CFID on any real > tables? > > -- > Jeff Jirsa > > > On Jun 27, 2018, at 7:58 PM, dayu wrote: > > That sound reasonable, I have seen schema mismatch error before. > So any advise to deal with schema mismatches? > > Dayu > > At 2018-06-28 09:50:37, "Jeff Jirsa" wrote: > >That log message says you did: > > > > CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming > > > >If you’re absolutely sure you didn’t, you should look for schema mismatches > >in your cluster > > > > > >-- > >Jeff Jirsa > > > > > >> On Jun 27, 2018, at 7:49 PM, dayu wrote: > >> > >> CF 53f6d520-2dc6-11e8-948d-ab7caa3c8c36 was dropped during streaming > > > >- > >To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > >For additional commands, e-mail: user-h...@cassandra.apache.org > > > > > > > > >
