[
https://issues.apache.org/jira/browse/CASSANDRA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069243#comment-13069243
]
ivan commented on CASSANDRA-2930:
---------------------------------
Hi Jonathan,
we built our Cassandra package from git://git.apache.org/cassandra.git
cassandra-0.8.1 branch.
As I see it's the same as official 0.8.1 Cassandra code.
grep "Cassandra version" /var/log/cassandra/system.log:
INFO [main] 2011-07-21 14:42:48,553 StorageService.java (line 378) Cassandra
version: 0.8.1
I checked CASSANDRA-2675 report.
Patch 0002-Avoid-modifying-super-column-in-memtable-being-flush-v2.patch is in
our code.
PAtch 0001-Don-t-remove-columns-from-super-columns-in-memtable.patch is not in
our code, but as I see it's not in official package and trunk also.
This error happens rarely. I found it 2 to 3 times in a 128MB commitlog.
I suspect some race condition, but a RowMutation shouldn't change.
(SuperColumn.java:371)
So I have no clue yet. It's welcomed any further test suggestion.
Regards,
ivan
> corrupt commitlog
> -----------------
>
> Key: CASSANDRA-2930
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2930
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.1
> Environment: Linux, amd64.
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Reporter: ivan
> Attachments: CommitLog-1310637513214.log
>
>
> We get "Exception encountered during startup" error while Cassandra starts.
> Error messages:
> INFO 13:56:28,736 Finished reading
> /var/lib/cassandra/commitlog/CommitLog-1310637513214.log
> ERROR 13:56:28,736 Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at
> java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> Exception encountered during startup.
> java.io.IOError: java.io.EOFException
> at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
> at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
> at
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
> at
> java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
> at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:419)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:139)
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:127)
> at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:382)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:278)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:175)
> at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:368)
> at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
> at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:261)
> ... 13 more
> After some debugging I found that in some serialized supercolumns column
> counter is less than the number of serialized columns. Difference was always
> 1 in corrupt commitlogs. This error always appears with supercolumns with
> more than one column, but there are properly serialized supercolumns also in
> commitlog.
> I have no clue yet why this error happens. I suspect it maybe a race
> condition.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
