[
https://issues.apache.org/jira/browse/CASSANDRA-12575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452324#comment-15452324
]
Sylvain Lebresne commented on CASSANDRA-12575:
----------------------------------------------
Given the symptoms, it would suggest the unhealthy node somehow doesn't have
the columns definitions from the schema (but has the rest of the table
definition, which is why it's using the "default" name 'key' and 'column1' for
the primary key columns but doesn't have anything else). Which doesn't really
say why that happens.
bq. "describe table" on the problematic node shows the correct table
description.
That's interesting. "describe" actually rebuild it's definition directly from
the system schema tables, so it would mean those are correct, which maybe you
could confirm by calling {{SELECT * FROM system.schema_columns WHERE
keyspace_name='statusflow' AND columnfamily_name='flow'}} on the broken node?
If the schema tables are indeed correct, that suggest the in-memory schema
representation isn't, which would be pretty weird.
Have you tried restarting the problematic node btw?
> A Cassandra node gets corrupted and contains only "key" and "column1" columns
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-12575
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12575
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra version: cassandra-2.2.7-6
> We use a 3 data center environment, each data center contains two Cassandra
> nodes.
> Reporter: Michal Lefler
> Attachments: 122310b.jpg, Site2_Tables.JPG, system.log,
> system.log.1.zip
>
>
> We executed our application for a few hours, and all the nodes were
> functioning correctly.
> Then, after a few hours, we suddenly started getting the following exceptions:
> {code}
> com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
> siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') [main_24]
> [TopologyManager.<init>()]
> com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
> siteId in where clause ('siteId = 'Site2-sdclab008-11-12'')
> at
> com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
>
> at
> com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
>
> at
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
>
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
> at
> com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)
>
> {code}
> After some investigation, we found out that we got this kind of an exception
> whenever we tried to query EACH ONE OF THE TABLES of one of our Cassandra
> nodes.
> When we tried to query the tables directly using CQL, we got the following
> result:
> All the tables only had one column named "key" for the partition key, and a
> "columnx" column for each one of the clustering keys (column1, column2 etc.)
> For example:
> This query was executed on a healthy node: (siteId is the partition key, and
> flowType is the clustering key)
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> siteId | flowType | currentVersion | flowName | flowOwner
> | flowState | lockUpdatedTime | previousVersion | rev
> -----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
> EMS-roger-01-02 | statusApi | 5.1-787 | installation |
> | idle | 1472618536062180 | 5.1-787 | 1
> EMS-roger-01-02 | topology | null | topology_inserted |
> 10.240.32.130 | idle | 1472618335876404 | null | 1
> {code}
> The same query was executed on the problematic node:
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> key | column1
> -----------------+-----------
> EMS-roger-01-02 | statusApi
> EMS-roger-01-02 | topology
> {code}
> We tried to restart the node, and also to performed a full repair, but
> nothing helped.
> Finally we had to delete the node's data entirely, and resync it from the
> other nodes.
> We saw these lines on the Cassandra log (may or may not be related to the
> problem):
> {code}
> ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception
> encountered during startup
> java.lang.AssertionError:
> org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table
> roles
> at
> org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
> [apache-cassandra-2.2.4.jar:2.2.4]
> Caused by: org.apache.cassandra.exceptions.InvalidRequestException:
> unconfigured table roles
> at
> org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 8 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
