[ https://issues.apache.org/jira/browse/CASSANDRA-11726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274214#comment-15274214 ]
Aleksey Yeschenko commented on CASSANDRA-11726: ----------------------------------------------- It sounds like corruption to me. Repair would not correct this, but scrub *might*. Can you run scrub on the 'bad' nodes and get back to us with the result? Thanks. > IndexOutOfBoundsException when selecting (distinct) row ids from counter > table. > ------------------------------------------------------------------------------- > > Key: CASSANDRA-11726 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11726 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: C* 3.5, cluster of 4 nodes. > Reporter: Jaroslav Kamenik > > I have simple table containing counters: > CREATE TABLE tablename ( > object_id ascii, > counter_id ascii, > count counter, > PRIMARY KEY (object_id, counter_id) > ) WITH CLUSTERING ORDER BY (counter_id ASC) > AND bloom_filter_fp_chance = 0.01 > AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} > AND comment = '' > AND compaction = {'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32', 'min_threshold': '4'} > AND compression = {'enabled': 'false'} > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = '99PERCENTILE'; > Counters are often inc/decreased, whole rows are queried, deleted sometimes. > After some time I tried to query all object_ids, but it failed with: > cqlsh:woc> consistency quorum; > cqlsh:woc> select object_id from tablename; > ServerError: <ErrorMessage code=0000 [Server error] > message="java.lang.IndexOutOfBoundsException"> > select * from ..., select where .., updates works well.. > With consistency one it works sometimes, so it seems something is broken at > one server, but I tried to repair table there and it did not help. > Whole exception from server log: > java.lang.IndexOutOfBoundsException: null > at java.nio.Buffer.checkIndex(Buffer.java:546) ~[na:1.8.0_73] > at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:314) > ~[na:1.8.0_73] > at > org.apache.cassandra.db.context.CounterContext.headerLength(CounterContext.java:141) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.context.CounterContext.access$100(CounterContext.java:76) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.context.CounterContext$ContextState.<init>(CounterContext.java:758) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.context.CounterContext$ContextState.wrap(CounterContext.java:765) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.context.CounterContext.merge(CounterContext.java:271) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.Conflicts.mergeCounterValues(Conflicts.java:76) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.db.rows.Cells.reconcile(Cells.java:143) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:591) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:549) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:526) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:473) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:437) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.FilteredRows.isEmpty(FilteredRows.java:30) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.Filter.closeIfEmpty(Filter.java:49) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:22) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:6) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.StorageProxy$SingleRangeResponse.computeNext(StorageProxy.java:2000) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.StorageProxy$SingleRangeResponse.computeNext(StorageProxy.java:1972) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.BaseIterator.tryGetMoreContents(BaseIterator.java:113) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.BaseIterator.hasMoreContents(BaseIterator.java:81) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:85) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2041) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2010) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:655) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:382) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:335) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) > [apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) > [apache-cassandra-3.5.jar:3.5] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_73] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > [apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.5.jar:3.5] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73] -- This message was sent by Atlassian JIRA (v6.3.4#6332)