[
https://issues.apache.org/jira/browse/CASSANDRA-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565389#comment-13565389
]
André Borgqvist commented on CASSANDRA-5191:
--------------------------------------------
I've added a catch and log print on row 265 in CommitLogSegment.
try
{
buffer.putLong(checksum.getValue());
}catch(BufferOverflowException e)
{
logger.error("BUFFEROVERFLOW: mutation="+rowMutation + ",
checksum="+checksum.getValue() + ", bufferLen=" + buffer.remaining());
throw e;
}
Result in log (column names contains colons from our application logic which
looks a bit weird in the printout, for example 'c:res:State' is a column name):
ERROR [COMMIT-LOG-WRITER] 2013-01-29 12:40:15,003 CommitLogSegment.java (line
271) BUFFEROVERFLOW: mutation=RowMutation(keyspace='cake',
key='376366343966643633653264313631626438383736316366316432333133653331653030633633336566653035626333',
modifications=[ColumnFamily(vouchers
[c:res:State:false:7@1359459603537005,c:res:TemporaryState:true:4@1359459603537007,c:res:subscriberId:false:12@1359459603537003,c:res:transactionId:false:5@1359459603537001,])]),
checksum=3781223592, bufferLen=4
Noticed that "marked for deletion" was true on
c:res:TemporaryState:true:4@1359459603537007 and I know we write with a very
short ttl (5s), I suspected this has something to do with it.
So I made a small test program that loops and inserts a column with a ttl of
5s, and reproduced the error with it.
The program is run toward node 1 in the cluster and node 2 crashes after some
minutes (I started three instances on the test program to try to speed it up).
ERROR [COMMIT-LOG-WRITER] 2013-01-29 14:52:43,304 CommitLogSegment.java (line
271) BUFFEROVERFLOW: mutation=RowMutation(keyspace='cake',
key='31383037363038', modifications=[ColumnFamily(vouchers
[testcol:true:4@1359467551818,])]), checksum=2632894560, bufferLen=6
Just noticed NTP is not correctly configured and the system clock on node 2 is
12 seconds ahead.
> BufferOverflowException in CommitLogSegment
> -------------------------------------------
>
> Key: CASSANDRA-5191
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5191
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.9
> Environment: RHEL 2.6.32-220.el6.x86_64, jdk1.6.0_27
> Reporter: André Borgqvist
>
> Running mixed reads, writes and deletes on a single column family in a two
> node cluster. After a few minutes the following appears in the system log:
> ERROR [COMMIT-LOG-WRITER] 2013-01-25 12:49:55,955
> AbstractCassandraDaemon.java (line 135) Exception in thread
> Thread[COMMIT-LOG-WRITER,5,main]
> java.nio.BufferOverflowException
> at java.nio.Buffer.nextPutIndex(Buffer.java:499)
> at java.nio.DirectByteBuffer.putLong(DirectByteBuffer.java:756)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:265)
> at
> org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:382)
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:50)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:662)
> Possibly related to https://issues.apache.org/jira/browse/CASSANDRA-3615
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
