Jeff Evans created CASSANDRA-11812:
--------------------------------------
Summary: IndexOutOfBoundsException in CounterContext.headerLength
Key: CASSANDRA-11812
URL: https://issues.apache.org/jira/browse/CASSANDRA-11812
Project: Cassandra
Issue Type: Bug
Environment: Ubuntu 14.04 TLS
Cassandra 3.4
Reporter: Jeff Evans
My team is using
https://github.com/Contrast-Security-OSS/cassandra-migration for schema
migrations, and it creates a table with a counter to store the schema version.
We're able to create the table fine in 3.4, but when we run the tool on an
existing keyspace, the client reports an error from the server. The server
logs show an IndexOutOfBounds exception related to the counter column.
the library creates a table with name and count:
CREATE TABLE silver.cassandra_migration_version_counts (
name text PRIMARY KEY,
count counter
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
then when the library performs a migration, it counts the rows in this table:
SELECT count(*) FROM silver.cassandra_migration_version_counts;
this query throws the error:
cqlsh:silver> SELECT count(*) FROM silver.cassandra_migration_version_counts;
ServerError: <ErrorMessage code=0000 [Server error]
message="java.lang.IndexOutOfBoundsException">
the client driver debug logs show the query is running with CONSISTENCY ALL
2016-05-16 18:03:05 [cluster2-nio-worker-3] WARN
c.d.driver.core.RequestHandler - /172.24.131.52:9042 replied with server error
(java.lang.IndexOutOfBoundsException), defuncting connection.
2016-05-16 18:03:05 [cluster2-nio-worker-3] DEBUG
com.datastax.driver.core.Host.STATES - Defuncting
Connection[/172.24.131.52:9042-1, inFlight=0, closed=false] because: An
unexpected error occurred server side on /172.24.131.52:9042:
java.lang.IndexOutOfBoundsException
2016-05-16 18:03:05 [cluster2-nio-worker-3] DEBUG
com.datastax.driver.core.Connection - Connection[/172.24.131.52:9042-1,
inFlight=0, closed=true] closing connection
2016-05-16 18:03:05 [cluster2-nio-worker-3] DEBUG
com.datastax.driver.core.Host.STATES - [/172.24.131.52:9042] preventing new
connections for the next 1000 ms
2016-05-16 18:03:05 [cluster2-nio-worker-3] DEBUG
com.datastax.driver.core.Host.STATES - [/172.24.131.52:9042]
Connection[/172.24.131.52:9042-1, inFlight=0, closed=true] failed, remaining = 0
2016-05-16 18:03:05 [cluster2-nio-worker-3] DEBUG
c.d.driver.core.RequestHandler - [937744315-1] Doing retry 1 for query SELECT
count(*) FROM silver.cassandra_migration_version_counts; at consistency ALL
2016-05-16 18:03:05 [cluster2-nio-worker-3] DEBUG
c.d.driver.core.RequestHandler - [937744315-1] Error querying
/172.24.131.52:9042 : com.datastax.driver.core.exceptions.ServerError: An
unexpected error occurred server side on /172.24.131.52:9042:
java.lang.IndexOutOfBoundsException
I can repro the error with this table, but I haven't found a simple repro yet.
here's the call stack from the server log:
ERROR [SharedPool-Worker-2] 2016-05-16 17:00:39,313 ErrorMessage.java:338 -
Unexpected exception during request
java.lang.IndexOutOfBoundsException: null
at java.nio.Buffer.checkIndex(Buffer.java:546) ~[na:1.8.0_72]
at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:314)
~[na:1.8.0_72]
at
org.apache.cassandra.db.context.CounterContext.headerLength(CounterContext.java:141)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.context.CounterContext.access$100(CounterContext.java:76)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.context.CounterContext$ContextState.<init>(CounterContext.java:758)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.context.CounterContext$ContextState.wrap(CounterContext.java:765)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.context.CounterContext.merge(CounterContext.java:271)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.Conflicts.mergeCounterValues(Conflicts.java:76)
~[apache-cassandra-3.4.jar:3.4]
at org.apache.cassandra.db.rows.Cells.reconcile(Cells.java:143)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:591)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:549)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.4.jar:3.4]
at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:526)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:473)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:437)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.FilteredRows.isEmpty(FilteredRows.java:30)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.Filter.closeIfEmpty(Filter.java:49)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:22)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:6)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.service.StorageProxy$SingleRangeResponse.computeNext(StorageProxy.java:2000)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.service.StorageProxy$SingleRangeResponse.computeNext(StorageProxy.java:1972)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.BaseIterator.tryGetMoreContents(BaseIterator.java:113)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.BaseIterator.hasMoreContents(BaseIterator.java:81)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:85)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2041)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2010)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.cql3.statements.SelectStatement.pageAggregateQuery(SelectStatement.java:371)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:331)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:215)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:77)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:237)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
~[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
[apache-cassandra-3.4.jar:3.4]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
[apache-cassandra-3.4.jar:3.4]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_72]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.4.jar:3.4]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.4.jar:3.4]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
