Hi there, Sorry if this is not the right place to report bugs. I am using 2.0.7 and I have a 10 boxes clusters with about 200TB capacity. I just found I had 3 boxes with error exceptions. With datastax opscenter I can see these three nodes lost connections (no reponse), but after I sshed to these server, cassandara were still running, and the 'system.log' still had logs.
I think this might be a bug so any one would kindly help to investigate into it? Thanks~ ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249 CassandraDaemon.java > (line 198) Exception in thread Thread[CompactionExecutor:1,1,main] > FSWriteError in /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db > at > org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475) > at > org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212) > at > org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.io.IOException: No space left on device > at java.io.FileOutputStream.write(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:295) > at java.io.DataOutputStream.writeInt(DataOutputStream.java:197) > at > org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34) > at > org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44) > at > org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41) > at > org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468) > ... 13 more > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406 StorageService.java > (line 367) Stopping gossiper > WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406 StorageService.java > (line 281) Stopping gossip by operator request > INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java (line > 1271) Announcing shutdown > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406 StorageService.java > (line 372) Stopping RPC server > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java > (line 141) Stop listening to thrift clients > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417 StorageService.java > (line 377) Stopping native transport > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line > 181) Stop listening for CQL clients >