The ArrayIndexOutOfBounds in the ReadStage looks like it can happen if a key is not of the expected type. Could the comparator for the CF have changed ?
The error in the RequestResponseStage may be the race condition identified here https://issues.apache.org/jira/browse/CASSANDRA-1959 Aaron On 27 Jan 2011, at 19:22, B. Todd Burruss wrote: > i ran out of file handles on the "repairing node" after doing nodetool repair > - strange as i have never had this issue until using 0.7.0 (but i should say > that i have not truly tested 0.7.0 until now.) up'ed the number of file > handles, removed data, restarted nodes, then restarted my test. waited a > little while. i have two keyspaces on the cluster, so i checked the number > of SSTABLES in one of them before "nodetool repair" and i see 36 "data.db" > files, spread over 11 column families. very reasonable. > > after running nodetool repair i have over 900 "data.db" files, immediately! > now after waiting several hours i have over 1500 data.db files. out of these > i have 95 "compacted" files > > lsof reporting 803 files in use by cassandra for the "Queues" keyspace ... > > [cassandra@kv-app02 ~]$ /usr/sbin/lsof -p 32645|grep Data.db|grep -c Queues > 803 > > .. this doesn't sound right to me. checking the server log i see a lot of > these messages: > > ERROR [RequestResponseStage:14] 2011-01-26 17:00:29,493 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.lang.ArrayIndexOutOfBoundsException: -1 > at java.util.ArrayList.fastRemove(ArrayList.java:441) > at java.util.ArrayList.remove(ArrayList.java:424) > at > com.google.common.collect.AbstractMultimap.remove(AbstractMultimap.java:219) > at > com.google.common.collect.ArrayListMultimap.remove(ArrayListMultimap.java:60) > at > org.apache.cassandra.net.MessagingService.responseReceivedFrom(MessagingService.java:436) > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:40) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > > and a lot of these: > > ERROR [ReadStage:809] 2011-01-26 21:48:01,047 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.lang.ArrayIndexOutOfBoundsException > ERROR [ReadStage:809] 2011-01-26 21:48:01,047 AbstractCassandraDaemon.java > (line 91) Fatal exception in thread Thread[ReadStage:809,5,main] > java.lang.ArrayIndexOutOfBoundsException > > and some more like this: > ERROR [ReadStage:15] 2011-01-26 20:59:14,695 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.lang.ArrayIndexOutOfBoundsException: 6 > at > org.apache.cassandra.db.marshal.TimeUUIDType.compareTimestampBytes(TimeUUIDType.java:56) > at > org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:45) > at > org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:29) > at > org.apache.cassandra.db.filter.QueryFilter$1.compare(QueryFilter.java:98) > at > org.apache.cassandra.db.filter.QueryFilter$1.compare(QueryFilter.java:95) > at > org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:334) > at > org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230) > at > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:118) > at > org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:142) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1230) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1107) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1077) > at org.apache.cassandra.db.Table.getRow(Table.java:384) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63) > at > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:68) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) >
