[
https://issues.apache.org/jira/browse/CASSANDRA-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853891#comment-13853891
]
Lajos Bodrogi commented on CASSANDRA-6224:
------------------------------------------
Maybe i have a related problem. (Cassandra 2.0.3, Ubuntu 12.04, Cluster of 3
nodes)
I got the following Perl error:
ARGH! NEW PRIMARY FAILED: OPCODE_ERROR: unconfigured columnfamily source
8704 Protocol::CassandraCQL::Frame=SCALAR(0x10f79a0)
And one node stopped responding (the java process was still running) while the
others in the cluster kept working.
Here are the server logs:
INFO 12:23:26,299 flushing high-traffic column family CFS(Keyspace='vst8',
ColumnFamily='series') (estimated 38461316 bytes)
INFO 12:23:26,299 Enqueuing flush of
Memtable-series@1926422961(11944355/38461316 serialized/live bytes, 230603 ops)
INFO 12:23:26,300 Writing Memtable-series@1926422961(11944355/38461316
serialized/live bytes, 230603 ops)
ERROR 12:23:26,511 Exception in thread Thread[FlushWriter:46,5,main]
FSWriteError in
/var/lib/cassandra/data/vst8/series/vst8-series-tmp-jb-760-Filter.db
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:462)
at
org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:210)
at
org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:288)
at
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:387)
at
org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:320)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.write(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:282)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
at
org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)
at
org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)
at
org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:455)
... 9 more
ERROR 12:23:26,513 Stopping gossiper
WARN 12:23:26,513 Stopping gossip by operator request
INFO 12:23:26,513 Announcing shutdown
ERROR 12:23:28,513 Stopping RPC server
INFO 12:23:28,513 Stop listening to thrift clients
ERROR 12:23:28,517 Stopping native transport
INFO 12:23:28,532 Stop listening for CQL clients
As you can see, the disk went full.
> CQL3 Column family / tables "disappear", get unconfigured columnfamily errors
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-6224
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6224
> Project: Cassandra
> Issue Type: Bug
> Components: API, Core
> Environment: Cassandra 2.0.1 Amazon AWS Ubuntu Single-node
> Reporter: Constance Eustace
>
> We're seeing CQL3 tables seemingly arbitrarily disappear. Need to repair for
> prod meant we reconstructed the affected schema before nodetool repairs or
> similar attempts were done.
> It seems to take a few days to appear. Volumes are not tremendously high
> yet...
> Caused by: java.sql.SQLSyntaxErrorException:
> InvalidRequestException(why:unconfigured columnfamily entity_hierarchydef)
> at
> org.apache.cassandra.cql.jdbc.CassandraPreparedStatement.<init>(CassandraPreparedStatement.java:103)
> ~[cassandra-jdbc-1.2.5.jar:na]
> at
> org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:388)
> ~[cassandra-jdbc-1.2.5.jar:na]
> at
> org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:372)
> ~[cassandra-jdbc-1.2.5.jar:na]
> at
> org.apache.cassandra.cql.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:50)
> ~[cassandra-jdbc-1.2.5.jar:na]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.3.jar:1.3]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.3.jar:1.3]
> at
> com.bestbuy.contentsystems.cupcake.storage.cassandra.cqlentity.CQL.tool.CassPSC.createPreparedStatement(CassPSC.java:61)
> ~[ingest-storage-QA-SNAPSHOT.jar:QA-SNAPSHOT]
> at
> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:583)
> ~[spring-jdbc-3.2.4.RELEASE.jar:3.2.4.RELEASE]
> ... 148 common frames omitted
> Caused by: org.apache.cassandra.thrift.InvalidRequestException: null
> at
> org.apache.cassandra.thrift.Cassandra$prepare_cql3_query_result.read(Cassandra.java:39567)
> ~[cassandra-thrift-1.2.8.jar:1.2.8]
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> ~[libthrift-0.7.0.jar:0.7.0]
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_prepare_cql3_query(Cassandra.java:1625)
> ~[cassandra-thrift-1.2.8.jar:1.2.8]
> at
> org.apache.cassandra.thrift.Cassandra$Client.prepare_cql3_query(Cassandra.java:1611)
> ~[cassandra-thrift-1.2.8.jar:1.2.8]
> at
> org.apache.cassandra.cql.jdbc.CassandraConnection.prepare(CassandraConnection.java:517)
> ~[cassandra-jdbc-1.2.5.jar:na]
> at
> org.apache.cassandra.cql.jdbc.CassandraConnection.prepare(CassandraConnection.java:532)
> ~[cassandra-jdbc-1.2.5.jar:na]
> at
> org.apache.cassandra.cql.jdbc.CassandraPreparedStatement.<init>(CassandraPreparedStatement.java:96)
> ~[cassandra-jdbc-1.2.5.jar:na]
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
