[
https://issues.apache.org/jira/browse/CASSANDRA-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-3957:
----------------------------------------
Attachment: 3957.txt
Here's my theory.
The stacktrace concerning the sblocks is due to a bug in DSE's CFS. This
happened because CFS sometimes communicate with C* in the same JVM without
going trough the wire, so the re-use of ByteBuffer was problematic.
The other stacktrace however can't be the same problem. Whatever Hector does,
it goes over the wire and thus can't change the size of a ByteBuffer in C*.
However, if we eliminate the sblocks trace, all the other trace uses
SuperColumn. And as it turns out, I think we have a race with SC. As Jonathan
pointed out, with the period commit log, if we happen to modify a mutation that
was sent to the commit log at any time, that would be a bug. This means not
modifying the CF object in the RowMutation, but for SuperColumn, this also mean
we shouldn't modify those super columns. However, when we update the row cache
(and if it's not a SerializingCache), we do potentially store references to the
original SCs in the cached row, which could then lead to the stack trace on
this issue. I'm attaching a patch that fixes this.
To make sure this could be the issue here, it would help to know if the
stacktrace comes from column families where row cache was used and was not set
to the serializing cache.
I'll note however that the very small difference in the stacktraces between the
predicted size and the actual serialized side support the theory of having say
just one column of a super column updated, while in the sblocks case, the
difference was much bigger, supporting the theory of the ByteBuffer content
being changed to something completely different.
> Supercolumn serialization assertion failure
> -------------------------------------------
>
> Key: CASSANDRA-3957
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3957
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Labels: datastax_qa
> Fix For: 1.0.9, 1.1.0
>
> Attachments: 3957.txt
>
>
> As reported at
> http://mail-archives.apache.org/mod_mbox/cassandra-user/201202.mbox/%3CCADJL=w5kH5TEQXOwhTn5Jm3cmR4Rj=nfjcqlryxv7plyasi...@mail.gmail.com%3E,
> {noformat}
> ERROR 10:51:44,282 Fatal exception in thread
> Thread[COMMIT-LOG-WRITER,5,main]
> java.lang.AssertionError: Final buffer length 4690 to accomodate data size
> of 2347 (predicted 2344) for RowMutation(keyspace='Player',
> key='36336138643338652d366162302d343334392d383466302d356166643863353133356465',
> modifications=[ColumnFamily(PlayerCity [SuperColumn(owneditem_1019
> []),SuperColumn(owneditem_1024 []),SuperColumn(owneditem_1026
> []),SuperColumn(owneditem_1074 []),SuperColumn(owneditem_1077
> []),SuperColumn(owneditem_1084 []),SuperColumn(owneditem_1094
> []),SuperColumn(owneditem_1130 []),SuperColumn(owneditem_1136
> []),SuperColumn(owneditem_1141 []),SuperColumn(owneditem_1142
> []),SuperColumn(owneditem_1145 []),SuperColumn(owneditem_1218
> [636f6e6e6563746564:false:5@1329648704269002
> ,63757272656e744865616c7468:false:3@1329648704269006
> ,656e64436f6e737472756374696f6e54696d65:false:13@1329648704269007
> ,6964:false:4@1329648704269000,6974656d4964:false:15@1329648704269001
> ,6c61737444657374726f79656454696d65:false:1@1329648704269008
> ,6c61737454696d65436f6c6c6563746564:false:13@1329648704269005
> ,736b696e4964:false:7@1329648704269009,78:false:4@1329648704269003
> ,79:false:3@1329648704269004,]),SuperColumn(owneditem_133
> []),SuperColumn(owneditem_134 []),SuperColumn(owneditem_135
> []),SuperColumn(owneditem_141 []),SuperColumn(owneditem_147
> []),SuperColumn(owneditem_154 []),SuperColumn(owneditem_159
> []),SuperColumn(owneditem_171 []),SuperColumn(owneditem_253
> []),SuperColumn(owneditem_422 []),SuperColumn(owneditem_438
> []),SuperColumn(owneditem_515 []),SuperColumn(owneditem_521
> []),SuperColumn(owneditem_523 []),SuperColumn(owneditem_525
> []),SuperColumn(owneditem_562 []),SuperColumn(owneditem_61
> []),SuperColumn(owneditem_634 []),SuperColumn(owneditem_636
> []),SuperColumn(owneditem_71 []),SuperColumn(owneditem_712
> []),SuperColumn(owneditem_720 []),SuperColumn(owneditem_728
> []),SuperColumn(owneditem_787 []),SuperColumn(owneditem_797
> []),SuperColumn(owneditem_798 []),SuperColumn(owneditem_838
> []),SuperColumn(owneditem_842 []),SuperColumn(owneditem_847
> []),SuperColumn(owneditem_849 []),SuperColumn(owneditem_851
> []),SuperColumn(owneditem_852 []),SuperColumn(owneditem_853
> []),SuperColumn(owneditem_854 []),SuperColumn(owneditem_857
> []),SuperColumn(owneditem_858 []),SuperColumn(owneditem_874
> []),SuperColumn(owneditem_884 []),SuperColumn(owneditem_886
> []),SuperColumn(owneditem_908 []),SuperColumn(owneditem_91
> []),SuperColumn(owneditem_911 []),SuperColumn(owneditem_930
> []),SuperColumn(owneditem_934 []),SuperColumn(owneditem_937
> []),SuperColumn(owneditem_944 []),SuperColumn(owneditem_945
> []),SuperColumn(owneditem_962 []),SuperColumn(owneditem_963
> []),SuperColumn(owneditem_964 []),])])
> at
> org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:682)
> at
> org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:279)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:122)
> at
> org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:599)
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
