Hi All,
We got lots of Exception in the log, and later the server crashed. any idea
what is happening and how to fix it?
ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[RequestResponseStage:4,5,main]
java.io.IOError: java.io.EOFException
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
at
org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
... 6 more
ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[RequestResponseStage:2,5,main]
java.io.IOError:
org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
cfId=-387130991
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
at
org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException:
Couldn't find cfId=-387130991
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
The is the exception before server crashes.
ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[ReadRepairStage:299,5,main]
java.lang.RuntimeException:
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.util.concurrent.RejectedExecutionException:
ThreadPoolExecutor has shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
at
org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
at
org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
at
org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
Thank you in advance,
Daning
