Michal Lefler created CASSANDRA-12575:
-----------------------------------------
Summary: A Cassandra node gets corrupted and contains only "key"
and "column1" columns
Key: CASSANDRA-12575
URL: https://issues.apache.org/jira/browse/CASSANDRA-12575
Project: Cassandra
Issue Type: Bug
Environment: We use a 3 data center environment, each data center
contains two Cassandra nodes.
Reporter: Michal Lefler
Attachments: 122310b.jpg, Site2_Tables.JPG, system.log,
system.log.1.zip
We executed our application for a few hours, and all the nodes were functioning
correctly.
Then, after a few hours, we suddenly started getting the following exceptions:
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') [main_24]
[TopologyManager.<init>()]
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
siteId in where clause ('siteId = 'Site2-sdclab008-11-12'')
at
com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
at
com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)
After some investigation, we found out that we got this kind of an exception
whenever we tried to query EACH ONE OF THE TABLES of one of our Cassandra nodes.
When we tried to query the tables directly using CQL, we got the following
result:
All the tables only had one column named "key" for the partition key, and a
"columnx" column for each one of the clustering keys (column1, column2 etc.)
For example:
This query was executed on a healthy node: (siteId is the partition key, and
flowType is the clustering key)
cqlsh> SELECT * from statusflow.flow ;
siteId | flowType | currentVersion | flowName | flowOwner
| flowState | lockUpdatedTime | previousVersion | rev
-----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
EMS-roger-01-02 | statusApi | 5.1-787 | installation |
| idle | 1472618536062180 | 5.1-787 | 1
EMS-roger-01-02 | topology | null | topology_inserted |
10.240.32.130 | idle | 1472618335876404 | null | 1
The same query was executed on the problematic node:
cqlsh> SELECT * from statusflow.flow ;
key | column1
-----------------+-----------
EMS-roger-01-02 | statusApi
EMS-roger-01-02 | topology
We tried to restart the node, and also to performed a full repair, but nothing
helped.
Finally we had to delete the node's data entirely, and resync it from the other
nodes.
We saw these lines on the Cassandra log (may or may not be related to the
problem):
ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception
encountered during startup
java.lang.AssertionError:
org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table
roles
at
org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
[apache-cassandra-2.2.4.jar:2.2.4]
Caused by: org.apache.cassandra.exceptions.InvalidRequestException:
unconfigured table roles
at
org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
~[apache-cassandra-2.2.4.jar:2.2.4]
... 8 common frames omitted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
