[
https://issues.apache.org/jira/browse/CASSANDRA-13339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946784#comment-15946784
]
Chris Richards commented on CASSANDRA-13339:
--------------------------------------------
This seems to be very sensitive to timing : adding code to convert the mutation
to text before computing the size of the serialized data (for subsequent
reporting of the mutation that was seen by at the time serializedSize() runs)
seems to reduce the frequency that this occurs significantly (I've not managed
to catch an instance with this in place).
In all cases the serialized mutation that has overflowed the allocated buffer
contains a single modification : I have seen it fire on various tables /
operations : even plain INSERTS with no paxos constraints.
> java.nio.BufferOverflowException: null
> --------------------------------------
>
> Key: CASSANDRA-13339
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13339
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Richards
>
> I'm seeing the following exception running Cassandra 3.9 (with Netty updated
> to 4.1.8.Final) running on a 2 node cluster. It would have been processing
> around 50 queries/second at the time (mixture of
> inserts/updates/selects/deletes) : there's a collection of tables (some with
> counters some without) and a single materialized view.
> ERROR [MutationStage-4] 2017-03-15 22:50:33,052 StorageProxy.java:1353 -
> Failed to apply mutation locally : {}
> java.nio.BufferOverflowException: null
> at
> org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:393)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:279)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:227)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:241)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
> [apache-cassandra-3.9.jar:3.9]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> [apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> and then again shortly afterwards
> ERROR [MutationStage-3] 2017-03-15 23:27:36,198 StorageProxy.java:1353 -
> Failed to apply mutation locally : {}
> java.nio.BufferOverflowException: null
> at
> org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:393)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:279)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:227)
> ~[apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:241)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
> [apache-cassandra-3.9.jar:3.9]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> [apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.9.jar:3.9]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.9.jar:3.9]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
