Thanks Aaron. when I do describe cluster, always there are "UNREACHABLE",
but nodetool ring is fine. it is pretty busy cluster, read 3K/sec
$ cassandra-cli -h localhost -u root -pw cassy
Connected to: "Production Cluster" on localhost/9160
Welcome to the Cassandra CLI.
Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[root@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
UNREACHABLE: [10.218.17.208, 10.123.83.114, 10.120.205.48,
10.240.113.171]
e331e720-4844-11e1-0000-d808570c0dfd: [10.28.78.123, 10.28.20.200,
10.6.115.239]
[root@unknown]
$ nodetool -h localhost ring
Address DC Rack Status State Load
Owns Token
141784319550391026443072753096570088105
10.28.78.123 datacenter1 rack1 Up Normal 5.46 GB
16.67% 0
10.120.205.48 datacenter1 rack1 Up Normal 5.49 GB
16.67% 28356863910078205288614550619314017621
10.6.115.239 datacenter1 rack1 Up Normal 5.53 GB
16.67% 56713727820156410577229101238628035242
10.28.20.200 datacenter1 rack1 Up Normal 5.51 GB
16.67% 85070591730234615865843651857942052863
10.123.83.114 datacenter1 rack1 Up Normal 5.49 GB
16.67% 113427455640312821154458202477256070484
10.240.113.171 datacenter1 rack1 Up Normal 5.43 GB
16.67% 141784319550391026443072753096570088105
Daning
On Thu, Mar 22, 2012 at 1:47 PM, aaron morton <aa...@thelastpickle.com>wrote:
> java.io.IOError:
>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>> cfId=-387130991
>>
> Schema may have diverged between nodes.
>
> use cassandra-cli and run describe cluster; to see how many schema
> versions you have.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/03/2012, at 6:27 AM, Daning Wang wrote:
>
> and we are on 0.8.6.
>
>
>
> On Wed, Mar 21, 2012 at 10:24 AM, Daning Wang <dan...@netseer.com> wrote:
>
>> Hi All,
>>
>>
>> We got lots of Exception in the log, and later the server crashed. any
>> idea what is happening and how to fix it?
>>
>> ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[RequestResponseStage:4,5,main]
>> java.io.IOError: java.io.EOFException
>> at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>> at
>> org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
>> at
>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:197)
>> at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
>> at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>> at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
>> ... 6 more
>> ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[RequestResponseStage:2,5,main]
>> java.io.IOError:
>> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
>> cfId=-387130991
>> at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
>> at
>> org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
>> at
>> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException:
>> Couldn't find cfId=-387130991
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
>> at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
>> at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
>> at
>> org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
>> at
>> org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64
>>
>>
>>
>> The is the exception before server crashes.
>>
>>
>> ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808
>> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
>> Thread[ReadRepairStage:299,5,main]
>> java.lang.RuntimeException:
>> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
>> shut down
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.util.concurrent.RejectedExecutionException:
>> ThreadPoolExecutor has shut down
>> at
>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>> at
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
>> at
>> org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
>> at
>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
>> at
>> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
>> at
>> org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
>> at
>> org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
>> at
>> org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> ... 3 more
>>
>> Thank you in advance,
>>
>> Daning
>>
>
>
>
