When you say schema to settle, do you means we provide proper consistency level? I don't think there is a provision to do that in tool. Or I can change the SYSTEM KEYSPACE definition of replication factor equal to number of nodes?
I mean the nodes have to all be on the same schema version. In the steps described below for correcting this problem - when you say move data from old directory to new, do you mean move the .db file? It will override the current file right? Yes copy the db files. No they will not override existing data. Do we have to rename the directory name to remove CFID i.e. just column family name without CFID? After that, update the System table as well? No, follow my instructions precisely, do not rename the directories, just copy to the new one and rm the old one. On Jul 13, 2015 2:52 PM, "Saladi Naidu" <[email protected]> wrote: > Sebastian, > Thank you so much for providing detailed explanation. I still have some > questions and I need to provide some clarifications > > 1. We do not have code that is creating the tables dynamically. All DDL > operations are done through Datastax DevCenter tool. When you say schema to > settle, do you means we provide proper consistency level? I don't think > there is a provision to do that in tool. Or I can change the SYSTEM > KEYSPACE definition of replication factor equal to number of nodes? > > 2. In the steps described below for correcting this problem - when you say > move data from old directory to new, do you mean move the .db file? It > will override the current file right? > > 3. Do we have to rename the directory name to remove CFID i.e. just > column family name without CFID? After that, update the System table as > well? > > > Naidu Saladi > > ------------------------------ > *From:* Sebastian Estevez <[email protected]> > *To:* [email protected]; Saladi Naidu <[email protected]> > *Sent:* Friday, July 10, 2015 5:25 PM > *Subject:* Re: DROP Table > > #1 The cause of this problem is a CREATE TABLE statement collision. Do > *not* generate tables dynamically from multiple clients, even with IF NOT > EXISTS. First thing you need to do is fix your code so that this does not > happen. Just create your tables manually from cqlsh allowing time for the > schema to settle. > > #2 Here's the fix: > > 1) *Change your code to not automatically re-create tables (even with IF > NOT EXISTS).* > > 2) Run a rolling restart to ensure schema matches across nodes. Run > nodetool describecluster around your cluster. Check that there is only one > schema version. > > ON EACH NODE: > 3) Check your filesystem and see if you have two directories for the table > in question in the data directory. > > If THERE ARE TWO OR MORE DIRECTORIES: > 4)Identify from schema_column_families which cf ID is the "new" one > (currently in use). > > cqlsh -e "select * from system.schema_column_families"|grep <table name> > > > 5) Move the data from the "old" one to the "new" one and remove the old > directory. > > 6) If there are multiple "old" ones repeat 5 for every "old" directory. > > 7) run nodetool refresh > > IF THERE IS ONLY ONE DIRECTORY: > > No further action is needed. > > All the best, > > [image: datastax_logo.png] <http://www.datastax.com/> > Sebastián Estévez > Solutions Architect | 954 905 8615 | [email protected] > [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: > facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] > <https://twitter.com/datastax> [image: g+.png] > <https://plus.google.com/+Datastax/about> > <http://feeds.feedburner.com/datastax> > > <http://cassandrasummit-datastax.com/> > > DataStax is the fastest, most scalable distributed database technology, > delivering Apache Cassandra to the world’s most innovative enterprises. > Datastax is built to be agile, always-on, and predictably scalable to any > size. With more than 500 customers in 45 countries, DataStax is the > database technology and transactional backbone of choice for the worlds > most innovative companies such as Netflix, Adobe, Intuit, and eBay. > > > > On Fri, Jul 10, 2015 at 12:15 PM, Saladi Naidu <[email protected]> > wrote: > > My understanding is that Cassandra File Structure follows below naming > convention > > /cassandra/*data/ <key-spaces> <table>* > > > > Whereas our file structure is as below, each table has multiple names and > when we drop tables and recreate these directories remain. Also when we > dropped the table one node was down, when it came back, we tried to do > Nodetool repair and repair kept failing referring to CFID error listed > below > > > drwxr-xr-x. 16 cass cass 4096 May 24 06:49 ../ > drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 > application_by_user-e0eec95019a211e58b954ffc8e9bfaa6/ > drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15 application_info- > 4dba2bf0054f11e58b954ffc8e9bfaa6/ > drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 > application_info-a0ee65d019a311e58b954ffc8e9bfaa6/ > drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 > configproperties-228ea2e0c13811e4aa1d4ffc8e9bfaa6/ > drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 > user_activation-95d005f019a311e58b954ffc8e9bfaa6/ > drwxr-xr-x. 3 cass cass 4096 Jun 25 10:16 > user_app_permission-9fddcd62ffbe11e4a25a45259f96ec68/ > drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 > user_credential-86cfff1019a311e58b954ffc8e9bfaa6/ > drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 > user_info-2fa076221b1011e58b954ffc8e9bfaa6/ > drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15 > user_info-36028c00054f11e58b954ffc8e9bfaa6/ > drwxr-xr-x. 3 cass cass 4096 Jun 25 10:15 > user_info-fe1d7b101a5711e58b954ffc8e9bfaa6/ > drwxr-xr-x. 4 cass cass 4096 Jun 25 10:16 > user_role-9ed0ca30ffbe11e4b71d09335ad2d5a9/ > > > WARN [Thread-2579] 2015-07-02 16:02:27,523 IncomingTcpConnection.java:91 > - UnknownColumnFamilyException reading from socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find > cfId=218e3c90-1b0e-11e5-a34b-d7c17b3e318a > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > ~[apache-cassandra-2.1.2.jar:2.1.2] > > > Naidu Saladi > > > > >
