[
https://issues.apache.org/jira/browse/CASSANDRA-12575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454911#comment-15454911
]
Michal Lefler commented on CASSANDRA-12575:
-------------------------------------------
Yes. I understand. I will let you know if we will reproduce.
Currently we have no idea how to reproduce either. It's a mystery... :-)
> A Cassandra node gets corrupted and contains only "key" and "column1" columns
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-12575
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12575
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra version: cassandra-2.2.7-6
> We use a 3 data center environment, each data center contains two Cassandra
> nodes.
> Reporter: Michal Lefler
> Attachments: 122310b.jpg, Site2_Tables.JPG, system.log,
> system.log.1.zip
>
>
> We executed our application for a few hours, and all the nodes were
> functioning correctly.
> Then, after a few hours, we suddenly started getting the following exceptions:
> {code}
> com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
> siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') [main_24]
> [TopologyManager.<init>()]
> com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
> siteId in where clause ('siteId = 'Site2-sdclab008-11-12'')
> at
> com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
>
> at
> com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
>
> at
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
>
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
> at
> com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)
>
> {code}
> After some investigation, we found out that we got this kind of an exception
> whenever we tried to query EACH ONE OF THE TABLES of one of our Cassandra
> nodes.
> When we tried to query the tables directly using CQL, we got the following
> result:
> All the tables only had one column named "key" for the partition key, and a
> "columnx" column for each one of the clustering keys (column1, column2 etc.)
> For example:
> This query was executed on a healthy node: (siteId is the partition key, and
> flowType is the clustering key)
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> siteId | flowType | currentVersion | flowName | flowOwner
> | flowState | lockUpdatedTime | previousVersion | rev
> -----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
> EMS-roger-01-02 | statusApi | 5.1-787 | installation |
> | idle | 1472618536062180 | 5.1-787 | 1
> EMS-roger-01-02 | topology | null | topology_inserted |
> 10.240.32.130 | idle | 1472618335876404 | null | 1
> {code}
> The same query was executed on the problematic node:
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> key | column1
> -----------------+-----------
> EMS-roger-01-02 | statusApi
> EMS-roger-01-02 | topology
> {code}
> We tried to restart the node, and also to performed a full repair, but
> nothing helped.
> Finally we had to delete the node's data entirely, and resync it from the
> other nodes.
> We saw these lines on the Cassandra log (may or may not be related to the
> problem):
> {code}
> ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception
> encountered during startup
> java.lang.AssertionError:
> org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table
> roles
> at
> org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
> [apache-cassandra-2.2.4.jar:2.2.4]
> Caused by: org.apache.cassandra.exceptions.InvalidRequestException:
> unconfigured table roles
> at
> org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 8 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
