thanks Bowen for answering Actually, I checked the server log and the only warning was that a node went offline. No, I have no backups or snapshots.
In the meantime I found that probably Cassandra moved all files from a directory to the snapshot directory. I am pretty sure of that because I have recently deleted all the snapshots I made because it was going out of disk space and I found this very directory full of files where the modification timestamp was the same as the first error I got in the log. Il giorno lun 1 mar 2021 alle ore 12:13 Bowen Song <bo...@bso.ng.invalid> ha scritto: > The first thing I'd check is the server log. The log may contain vital > information about the cause of it, and that there may be different ways to > recover from it depending on the cause. > > Also, please allow me to ask a seemingly obvious question, do you have a > backup? > > > On 01/03/2021 09:34, Marco Gasparini wrote: > > hello everybody, > > This morning, Monday!!!, I was checking on Cassandra cluster and I noticed > that all data was missing. I noticed the following error on each node (9 > nodes in the cluster): > > > > > > > > > > > *2021-03-01 09:05:52,984 WARN [MessagingService-Incoming-/x.x.x.x] > IncomingTcpConnection.java:103 run UnknownColumnFamilyException reading > from socket; closing org.apache.cassandra.db.UnknownColumnFamilyException: > Couldn't find table for cfId cba90a70-5c46-11e9-9e36-f54fe3235e69. If a > table was just created, this is likely due to the schema not being fully > propagated. Please wait for schema agreement on table creation. at > org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1533) > at > org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:758) > at > org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:697) > at > org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50) > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:183) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)* > > I tried to query the keyspace and got this: > > node1# cqlsh > Connected to Cassandra Cluster at x.x.x.x:9042. > [cqlsh 5.0.1 | Cassandra 3.11.5.1 | CQL spec 3.4.4 | Native protocol v4] > Use HELP for help. > cqlsh> select * from mykeyspace.mytable where id = 123935; > *InvalidRequest: Error from server: code=2200 [Invalid query] > message="Keyspace * *mykeyspace does not exist"* > > Investigating on each node I found that all the *SStables exist*, so I > think data is still there but the keyspace vanished, "magically". > > Other facts I can tell you are: > > - I have been getting Anticompaction errors from 2 nodes due to the > fact the disk was almost full. > - the cluster was online friday > - this morning, Monday, the whole cluster was offline and I noticed > the problem of "missing keyspace" > - During the weekend the cluster has been subject to inserts and > deletes > - I have a 9 node (HDD) Cassandra 3.11 cluster. > > I really need help on this, how can I restore the cluster? > > Thank you very much > Marco > > > > > > > > >
