Sometimes even waiting hours does not change. I have a cluster where I did like you, synchronization of create tables statement, then even I tried waiting for schema agreement, in loop until success, but sometimes the success never happens, i got that error in loop in the logs of a node, it seems we must restart nodes really often :(
Sébastien ERROR [InternalResponseStage:1117] 2021-06-03 17:32:34,937 MigrationCoordinator.java:408 - Unable to merge schema from /135.181.222.100 org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found a991bb50-c475-11eb-83cb-df35fc5a9bea; expected 994bee02-c475-11eb-beff-6d70d473832f) at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:984) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:938) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.config.Schema.updateTable(Schema.java:687) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1478) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1434) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1403) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1380) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.MigrationCoordinator.mergeSchemaFrom(MigrationCoordinator.java:367) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:404) [apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:393) [apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) [apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) [apache-cassandra-3.11.10.jar:3.11.10] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_292] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.10.jar:3.11.10] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292] Le jeu. 3 juin 2021 à 17:23, Joe Obernberger <joseph.obernber...@gmail.com> a écrit : > How does this work? I have a program that runs a series of alter table > statements, and then does inserts. In some cases, the insert happens > immediately after the alter table statement and the insert fails because > the schema (apparently) has not had time to propagate. I get an Undefined > column name error. > > The alter statements run single threaded, but the inserts run in multiple > threads. The alter statement is run in a synchronized block (Java). > Should I put an artificial delay after the alter statement? > > -Joe > On 6/1/2021 2:59 PM, Max C. wrote: > > We use ZooKeeper + kazoo’s lock implementation. Kazoo is a Python client > library for ZooKeeper. > > - Max > > Yes this is quite annoying. How did you implement that "external lock"? I > also thought of doing an external service that would be dedicated to that. > Cassandra client apps would send create instruction to that service, that > would receive them and do the creates 1 by 1, and the client app would wait > the response from it before starting to insert. > > Best, > > Sébastien. > > Le mar. 1 juin 2021 à 05:21, Max C. <mc_cassand...@core43.com> a écrit : > >> In our case we have a shared dev cluster with (for example) a key space >> for each developer, a key space for each CI runner, etc. As part of >> initializing our test suite we setup the schema to match the code that is >> about to be tested. This can mean multiple CI runners each adding/dropping >> tables at the same time but for different key spaces. >> >> Our experience is even though the schema changes do not conflict, we >> still run into schema mismatch problems. Our solution to this was to have >> a lock (external to Cassandra) that ensures only a single schema change >> operation is being issued at a time. >> >> People assume schema changes in Cassandra work the same way as MySQL or >> multiple users editing files on disk — i.e. as long as you’re not editing >> the same file (or same MySQL table), then there’s no problem. *This is >> NOT the case.* Cassandra schema changes are more like “git push”ing a >> commit to the same branch — i.e. at most one change can be outstanding at a >> time (across all tables, all key spaces)…otherwise you will run into >> trouble. >> >> Hope that helps. Best of luck. >> >> - Max >> >> Hello, >>> >>> I have a more general question about that, I cannot find clear answer. >>> >>> In my use case I have many tables (around 10k new tables created per >>> months) and they are created from many clients and only dynamically, with >>> several clients creating same tables simulteanously. >>> >>> What is the recommended way of creating tables dynamically? If I am >>> doing "if not exists" queries + wait for schema aggreement before and after >>> each create statement, will it work correctly for Cassandra? >>> >>> Sébastien. >>> >> >> > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > Virus-free. > www.avg.com > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > <#m_-7057239499363649981_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > >
