[
https://issues.apache.org/jira/browse/CASSANDRA-6181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13801772#comment-13801772
]
Fabien Rousseau commented on CASSANDRA-6181:
--------------------------------------------
Don't know if this can help but, unit tests do not set "-Xss" parameter and
uses the default value (1M in general, see
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html )
while default Xss is 180k for Cassandra1.2
Here are some raw numbers using unit tests and setting Xss in the build.xml :
N = 193; // 180k
N = 313; // 228k
N = 394; // 256k
N = 1036; // 512k
N = 2321; // 1024k
This number represents the number of inserted tombstones before the tests
starts failing (ie : incrementing this number by one : tests fails with
SOException)
This should probably explain why it was not reproducible.
By the way, maybe the unit tests should set the Xss parameter in order to be as
close as possible of a running cassandra instance ?
> Replaying a commit led to java.lang.StackOverflowError and node crash
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-6181
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6181
> Project: Cassandra
> Issue Type: Bug
> Environment: 1.2.8 & 1.2.10 - ubuntu 12.04
> Reporter: Jeffrey Damick
> Assignee: Sylvain Lebresne
> Priority: Critical
> Fix For: 1.2.12
>
> Attachments: 6181.txt
>
>
> 2 of our nodes died after attempting to replay a commit. I can attach the
> commit log file if that helps.
> It was occurring on 1.2.8, after several failed attempts to start, we
> attempted startup with 1.2.10. This also yielded the same issue (below).
> The only resolution was to physically move the commit log file out of the way
> and then the nodes were able to start...
> The replication factor was 3 so I'm hoping there was no data loss...
> {code}
> INFO [main] 2013-10-11 14:50:35,891 CommitLogReplayer.java (line 119)
> Replaying /ebs/cassandra/commitlog/CommitLog-2-1377542389560.log
> ERROR [MutationStage:18] 2013-10-11 14:50:37,387 CassandraDaemon.java (line
> 191) Exception in thread Thread[MutationStage:18,5,main]
> java.lang.StackOverflowError
> at
> org.apache.cassandra.db.marshal.TimeUUIDType.compareTimestampBytes(TimeUUIDType.java:68)
> at
> org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:57)
> at
> org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:29)
> at
> org.apache.cassandra.db.marshal.AbstractType.compareCollectionMembers(AbstractType.java:229)
> at
> org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:81)
> at
> org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
> at
> org.apache.cassandra.db.RangeTombstoneList.insertAfter(RangeTombstoneList.java:439)
> at
> org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:405)
> at
> org.apache.cassandra.db.RangeTombstoneList.weakInsertFrom(RangeTombstoneList.java:472)
> at
> org.apache.cassandra.db.RangeTombstoneList.insertAfter(RangeTombstoneList.java:456)
> at
> org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:405)
> at
> org.apache.cassandra.db.RangeTombstoneList.weakInsertFrom(RangeTombstoneList.java:472)
> at
> org.apache.cassandra.db.RangeTombstoneList.insertAfter(RangeTombstoneList.java:456)
> at
> org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:405)
> at
> org.apache.cassandra.db.RangeTombstoneList.weakInsertFrom(RangeTombstoneList.java:472)
> .... etc.... over and over until ....
> at
> org.apache.cassandra.db.RangeTombstoneList.weakInsertFrom(RangeTombstoneList.java:472)
> at
> org.apache.cassandra.db.RangeTombstoneList.insertAfter(RangeTombstoneList.java:456)
> at
> org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:405)
> at
> org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:144)
> at
> org.apache.cassandra.db.RangeTombstoneList.addAll(RangeTombstoneList.java:186)
> at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:180)
> at
> org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:197)
> at
> org.apache.cassandra.db.AbstractColumnContainer.addAllWithSizeDelta(AbstractColumnContainer.java:99)
> at org.apache.cassandra.db.Memtable.resolve(Memtable.java:207)
> at org.apache.cassandra.db.Memtable.put(Memtable.java:170)
> at
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:745)
> at org.apache.cassandra.db.Table.apply(Table.java:388)
> at org.apache.cassandra.db.Table.apply(Table.java:353)
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:258)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
