The first thing I'd check is the server log. The log may contain vital
information about the cause of it, and that there may be different ways
to recover from it depending on the cause.
Also, please allow me to ask a seemingly obvious question, do you have a
backup?
On 01/03/2021 09:34, Marco Gasparini wrote:
hello everybody,
This morning, Monday!!!, I was checking on Cassandra cluster and I
noticed that all data was missing. I noticed the following error on
each node (9 nodes in the cluster):
*2021-03-01 09:05:52,984 WARN [MessagingService-Incoming-/x.x.x.x]
IncomingTcpConnection.java:103 run UnknownColumnFamilyException
reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
table for cfId cba90a70-5c46-11e9-9e36-f54fe3235e69. If a table was
just created, this is likely due to the schema not being fully
propagated. Please wait for schema agreement on table creation.
at
org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1533)
at
org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:758)
at
org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:697)
at
org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50)
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:183)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)*
*
*
I tried to query the keyspace and got this:
node1# cqlsh
Connected to Cassandra Cluster at x.x.x.x:9042.
[cqlsh 5.0.1 | Cassandra 3.11.5.1 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> select * from mykeyspace.mytable where id = 123935;
*InvalidRequest: Error from server: code=2200 [Invalid query]
message="Keyspace * *mykeyspace does not exist"*
*
*
Investigating on each node I found that all the *SStables exist*, so I
think data is still there but the keyspace vanished, "magically".
Other facts I can tell you are:
* I have been getting Anticompaction errors from 2 nodes due to the
fact the disk was almost full.
* the cluster was online friday
* this morning, Monday, the whole cluster was offline and I noticed
the problem of "missing keyspace"
* During the weekend the cluster has been subject to inserts and deletes
* I have a 9 node (HDD) Cassandra 3.11 cluster.
I really need help on this, how can I restore the cluster?
Thank you very much
Marco
*
*
*
*
*
*
*
*
*
*
