[
https://issues.apache.org/jira/browse/CASSANDRA-12575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-12575:
----------------------------------------
Description:
We executed our application for a few hours, and all the nodes were functioning
correctly.
Then, after a few hours, we suddenly started getting the following exceptions:
{code}
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') [main_24]
[TopologyManager.<init>()]
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
siteId in where clause ('siteId = 'Site2-sdclab008-11-12'')
at
com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
at
com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)
{code}
After some investigation, we found out that we got this kind of an exception
whenever we tried to query EACH ONE OF THE TABLES of one of our Cassandra nodes.
When we tried to query the tables directly using CQL, we got the following
result:
All the tables only had one column named "key" for the partition key, and a
"columnx" column for each one of the clustering keys (column1, column2 etc.)
For example:
This query was executed on a healthy node: (siteId is the partition key, and
flowType is the clustering key)
{code}
cqlsh> SELECT * from statusflow.flow ;
siteId | flowType | currentVersion | flowName | flowOwner
| flowState | lockUpdatedTime | previousVersion | rev
-----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
EMS-roger-01-02 | statusApi | 5.1-787 | installation |
| idle | 1472618536062180 | 5.1-787 | 1
EMS-roger-01-02 | topology | null | topology_inserted |
10.240.32.130 | idle | 1472618335876404 | null | 1
{code}
The same query was executed on the problematic node:
{code}
cqlsh> SELECT * from statusflow.flow ;
key | column1
-----------------+-----------
EMS-roger-01-02 | statusApi
EMS-roger-01-02 | topology
{code}
We tried to restart the node, and also to performed a full repair, but nothing
helped.
Finally we had to delete the node's data entirely, and resync it from the other
nodes.
We saw these lines on the Cassandra log (may or may not be related to the
problem):
{code}
ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception
encountered during startup
java.lang.AssertionError:
org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table
roles
at
org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
[apache-cassandra-2.2.4.jar:2.2.4]
Caused by: org.apache.cassandra.exceptions.InvalidRequestException:
unconfigured table roles
at
org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
~[apache-cassandra-2.2.4.jar:2.2.4]
... 8 common frames omitted
{code}
was:
We executed our application for a few hours, and all the nodes were functioning
correctly.
Then, after a few hours, we suddenly started getting the following exceptions:
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') [main_24]
[TopologyManager.<init>()]
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
siteId in where clause ('siteId = 'Site2-sdclab008-11-12'')
at
com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
at
com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)
After some investigation, we found out that we got this kind of an exception
whenever we tried to query EACH ONE OF THE TABLES of one of our Cassandra nodes.
When we tried to query the tables directly using CQL, we got the following
result:
All the tables only had one column named "key" for the partition key, and a
"columnx" column for each one of the clustering keys (column1, column2 etc.)
For example:
This query was executed on a healthy node: (siteId is the partition key, and
flowType is the clustering key)
cqlsh> SELECT * from statusflow.flow ;
siteId | flowType | currentVersion | flowName | flowOwner
| flowState | lockUpdatedTime | previousVersion | rev
-----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
EMS-roger-01-02 | statusApi | 5.1-787 | installation |
| idle | 1472618536062180 | 5.1-787 | 1
EMS-roger-01-02 | topology | null | topology_inserted |
10.240.32.130 | idle | 1472618335876404 | null | 1
The same query was executed on the problematic node:
cqlsh> SELECT * from statusflow.flow ;
key | column1
-----------------+-----------
EMS-roger-01-02 | statusApi
EMS-roger-01-02 | topology
We tried to restart the node, and also to performed a full repair, but nothing
helped.
Finally we had to delete the node's data entirely, and resync it from the other
nodes.
We saw these lines on the Cassandra log (may or may not be related to the
problem):
ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception
encountered during startup
java.lang.AssertionError:
org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table
roles
at
org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
[apache-cassandra-2.2.4.jar:2.2.4]
Caused by: org.apache.cassandra.exceptions.InvalidRequestException:
unconfigured table roles
at
org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
~[apache-cassandra-2.2.4.jar:2.2.4]
at
org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
~[apache-cassandra-2.2.4.jar:2.2.4]
... 8 common frames omitted
> A Cassandra node gets corrupted and contains only "key" and "column1" columns
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-12575
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12575
> Project: Cassandra
> Issue Type: Bug
> Environment: We use a 3 data center environment, each data center
> contains two Cassandra nodes.
> Reporter: Michal Lefler
> Attachments: 122310b.jpg, Site2_Tables.JPG, system.log,
> system.log.1.zip
>
>
> We executed our application for a few hours, and all the nodes were
> functioning correctly.
> Then, after a few hours, we suddenly started getting the following exceptions:
> {code}
> com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
> siteId in where clause ('siteId = 'Site2-sdclab008-11-12'') [main_24]
> [TopologyManager.<init>()]
> com.datastax.driver.core.exceptions.InvalidQueryException: Undefined name
> siteId in where clause ('siteId = 'Site2-sdclab008-11-12'')
> at
> com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
>
> at
> com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
>
> at
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
>
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
> at
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
> at
> com.traffix.openblox.core.utils.clustering.infrastructure.topology.TopologyDao.getEmsServers(TopologyDao.java:94)
>
> {code}
> After some investigation, we found out that we got this kind of an exception
> whenever we tried to query EACH ONE OF THE TABLES of one of our Cassandra
> nodes.
> When we tried to query the tables directly using CQL, we got the following
> result:
> All the tables only had one column named "key" for the partition key, and a
> "columnx" column for each one of the clustering keys (column1, column2 etc.)
> For example:
> This query was executed on a healthy node: (siteId is the partition key, and
> flowType is the clustering key)
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> siteId | flowType | currentVersion | flowName | flowOwner
> | flowState | lockUpdatedTime | previousVersion | rev
> -----------------+-----------+----------------+-------------------+---------------+-----------+------------------+-----------------+-----
> EMS-roger-01-02 | statusApi | 5.1-787 | installation |
> | idle | 1472618536062180 | 5.1-787 | 1
> EMS-roger-01-02 | topology | null | topology_inserted |
> 10.240.32.130 | idle | 1472618335876404 | null | 1
> {code}
> The same query was executed on the problematic node:
> {code}
> cqlsh> SELECT * from statusflow.flow ;
> key | column1
> -----------------+-----------
> EMS-roger-01-02 | statusApi
> EMS-roger-01-02 | topology
> {code}
> We tried to restart the node, and also to performed a full repair, but
> nothing helped.
> Finally we had to delete the node's data entirely, and resync it from the
> other nodes.
> We saw these lines on the Cassandra log (may or may not be related to the
> problem):
> {code}
> ERROR [main] 2016-08-28 14:20:01.272 CassandraDaemon.java:651 - Exception
> encountered during startup
> java.lang.AssertionError:
> org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table
> roles
> at
> org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:458)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.setup(CassandraRoleManager.java:144)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1023)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
> [apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638)
> [apache-cassandra-2.2.4.jar:2.2.4]
> Caused by: org.apache.cassandra.exceptions.InvalidRequestException:
> unconfigured table roles
> at
> org.apache.cassandra.thrift.ThriftValidation.validateColumnFamily(ThriftValidation.java:116)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:798)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> at
> org.apache.cassandra.auth.CassandraRoleManager.prepare(CassandraRoleManager.java:454)
> ~[apache-cassandra-2.2.4.jar:2.2.4]
> ... 8 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
