[
https://issues.apache.org/jira/browse/CASSANDRA-13339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15938151#comment-15938151
]
Chris Richards commented on CASSANDRA-13339:
--------------------------------------------
I added some debug to see what was causing this : the size of the serialized
mutation is changing between when it was originally calculated and the point
that the write occurs. This seems to occur at any location within the
CommitLog file (not just near the end).
ERROR [MutationStage-1] 2017-03-23 08:42:15,184 CommitLog.java:301 - Caught
buffer overflow exception
ERROR [MutationStage-1] 2017-03-23 08:42:15,184 CommitLog.java:302 - totalSize
73, size 61
ERROR [MutationStage-1] 2017-03-23 08:42:15,184 CommitLog.java:303 - buffer:
position 22737351, limit 22737351, capacity 33554432
ERROR [MutationStage-1] 2017-03-23 08:42:15,184 CommitLog.java:306 - recomputed
size 106
where recomputed size is the value of
Mutation.serializer.serializedSize(mutation,
MessagingService.current_version);
after the exception has been thrown and caught.
I assume therefore that the mutation is changing between when the serialized
size was calculated and when it serialized - I have a new build that will try
and show more information about the mutation when this occurs to see if this
sheds any light on what is happening.
> java.nio.BufferOverflowException: null
> --------------------------------------
>
> Key: CASSANDRA-13339
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13339
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Richards
>
> I'm seeing the following exception running Cassandra 3.9 (with Netty updated
> to 4.1.8.Final) running on a 2 node cluster. It would have been processing
> around 50 queries/second at the time (mixture of
> inserts/updates/selects/deletes) : there's a collection of tables (some with
> counters some without) and a single materialized view.
> ERROR [MutationStage-4] 2017-03-15 22:50:33,052 StorageProxy.java:1353 -
> Failed to apply mutation locally : {}
> java.nio.BufferOverflowException: null
> at
> org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:393)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:279)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:227)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:241)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
> [apache-cassandra-3.9.jar:3.9]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> [apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> and then again shortly afterwards
> ERROR [MutationStage-3] 2017-03-15 23:27:36,198 StorageProxy.java:1353 -
> Failed to apply mutation locally : {}
> java.nio.BufferOverflowException: null
> at
> org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:393)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:279)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:227)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:241)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
> [apache-cassandra-3.9.jar:3.9]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> [apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
