Hi all,
We have recently run in production a new cluster C* 2.0.0 with 3 nodes RF 3.
After 7 days, we seen on log this error on one node :
ERROR [FlushWriter:216] 2013-10-07 07:11:46,538 CassandraDaemon.java (line
186) Exception in thread Thread[FlushWriter:216,5,main]
java.lang.AssertionError
at
org.apache.cassandra.io.sstable.SSTableWriter.rawAppend(SSTableWriter.java:198)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:186)
at
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:417)
at
org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:376)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
$ nodetool tpstats
Pool Name Active Pending Completed Blocked All
time blocked
ReadStage 0 0 20102712
0 0
RequestResponseStage 0 0 12186090
0 0
MutationStage 0 0 17999146
0 0
ReadRepairStage 0 0 1013914
0 0
ReplicateOnWriteStage 0 0 77415
0 0
GossipStage 0 0 234894
0 0
AntiEntropyStage 0 0 0
0 0
MigrationStage 0 0 4
0 0
MemoryMeter 0 0 475
0 0
MemtablePostFlusher 1 59 210
0 0
FlushWriter 0 0 248
0 155
MiscStage 0 0 0
0 0
commitlog_archiver 0 0 0
0 0
InternalResponseStage 0 0 3
0 0
HintedHandoff 0 0 5
0 0
$ nodetool compactionstats
pending tasks: 72
Active compaction remaining time : n/a
A few days later, the other two nodes had this error, since it seems that
the active thread on MemtablePostFlusher blocks any compaction, I can not
make snapshot, or drain, I tried rebooted servers one by one, it works
again for
a moment, but after a few days in the log I have this error again. I also
had loss of data that are occurring after the reboot. Now I do not know how
to fix the problem, I think the solution would be to install another
cluster and
copy the data, except if one of you had this problem and could guide me.
Thanks a lot,
