Re: OOM(Java heap space) on start-up during commit log replaying
Graham, Thanks for the reply. As I stated in mine first mail increasing the heap size fixes the problem but I'm more interesting in figuring out the right properties for commitlog and memtable sizes when we need to keep the heap smaller. Also I think we are not seeing CASSANDRA-7546 as I apply your patch but the problem still persists. What more details do you need? I'll be happy to provide them. On Wednesday, August 13, 2014 1:05 AM, graham sanderson gra...@vast.com wrote: Agreed need more details; and just start by increasing heap because that may wells solve the problem. I have just observed (which makes sense when you think about it) while testing fix for https://issues.apache.org/jira/browse/CASSANDRA-7546, that if you are replaying a commit log which has a high level of updates for the same partition key, you can hit that issue - excess memory allocation under high contention for the same partition key - (this might not cause OOM but will certainly massively tax GC and it sounds like you don’t have a lot/any headroom). On Aug 12, 2014, at 12:31 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Aug 12, 2014 at 9:34 AM, jivko donev jivko_...@yahoo.com wrote: We have a node with commit log director ~4G. During start-up of the node on commit log replaying the used heap space is constantly growing ending with OOM error. The heap size and new heap size properties are - 1G and 256M. We are using the default settings for commitlog_sync, commitlog_sync_period_in_ms and commitlog_segment_size_in_mb. What version of Cassandra? 1G is tiny for cassandra heap. There is a direct relationship between the data in the commitlog and memtables and in the heap. You almost certainly need more heap or less commitlog. =Rob
OOM(Java heap space) on start-up during commit log replaying
Hi all, We have a node with commit log director ~4G. During start-up of the node on commit log replaying the used heap space is constantly growing ending with OOM error. The heap size and new heap size properties are - 1G and 256M. We are using the default settings for commitlog_sync, commitlog_sync_period_in_ms and commitlog_segment_size_in_mb. The log shows that cassandra is stuck on MutationStage: Active Pending Completed Blocked 16 385 196 0 The stack trace is: ERROR [metrics-meter-tick-thread-1] 2014-08-12 19:15:10,181 CassandraDaemon.java (line 198) Exception in thread Thread[metrics-meter-tick-thread-1,5,main] java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.locks.AbstractQueuedSynchronizer.addWaiter(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source) at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.offer(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.add(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.add(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor.reExecutePeriodic(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ERROR [MutationStage:8] 2014-08-12 19:15:10,181 CassandraDaemon.java (line 198) Exception in thread Thread[MutationStage:8,5,main] java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.duplicate(Unknown Source) at org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:62) at org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:72) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:99) at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) at org.apache.cassandra.db.RangeTombstoneList.addAll(RangeTombstoneList.java:188) at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:219) at org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:184) at org.apache.cassandra.db.Memtable.resolve(Memtable.java:226) at org.apache.cassandra.db.Memtable.put(Memtable.java:173) at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:893) at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:368) at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:333) at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:352) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ERROR [MutationStage:8] 2014-08-12 19:15:12,080 CassandraDaemon.java (line 198) Exception in thread Thread[MutationStage:8,5,main] java.lang.IllegalThreadStateException at java.lang.Thread.start(Unknown Source) at org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:204) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:203) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Increasing the heap space to 2G solves the problem but we want to know if the problem could be solved without increasing the heap space. Does anyone have experience similar problem? If so are there any tuning options in cassandra.yaml? Any help will be much appreciated. If you need more information fell free to ask. Thanks, Jivko Donev
Re: OOM(Java heap space) on start-up during commit log replaying
On Tue, Aug 12, 2014 at 9:34 AM, jivko donev jivko_...@yahoo.com wrote: We have a node with commit log director ~4G. During start-up of the node on commit log replaying the used heap space is constantly growing ending with OOM error. The heap size and new heap size properties are - 1G and 256M. We are using the default settings for commitlog_sync, commitlog_sync_period_in_ms and commitlog_segment_size_in_mb. What version of Cassandra? 1G is tiny for cassandra heap. There is a direct relationship between the data in the commitlog and memtables and in the heap. You almost certainly need more heap or less commitlog. =Rob
Re: OOM(Java heap space) on start-up during commit log replaying
Hi Robert, Thanks for your reply. The Cassandra version is 2.07. Is there some commonly used rule for determining the commitlog and memtables size depending on the heap size? What would be the main disadvantage when having smaller commitlog? On Tuesday, August 12, 2014 8:32 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Aug 12, 2014 at 9:34 AM, jivko donev jivko_...@yahoo.com wrote: We have a node with commit log director ~4G. During start-up of the node on commit log replaying the used heap space is constantly growing ending with OOM error. The heap size and new heap size properties are - 1G and 256M. We are using the default settings for commitlog_sync, commitlog_sync_period_in_ms and commitlog_segment_size_in_mb. What version of Cassandra? 1G is tiny for cassandra heap. There is a direct relationship between the data in the commitlog and memtables and in the heap. You almost certainly need more heap or less commitlog. =Rob
Re: OOM(Java heap space) on start-up during commit log replaying
Agreed need more details; and just start by increasing heap because that may wells solve the problem. I have just observed (which makes sense when you think about it) while testing fix for https://issues.apache.org/jira/browse/CASSANDRA-7546, that if you are replaying a commit log which has a high level of updates for the same partition key, you can hit that issue - excess memory allocation under high contention for the same partition key - (this might not cause OOM but will certainly massively tax GC and it sounds like you don’t have a lot/any headroom). On Aug 12, 2014, at 12:31 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Aug 12, 2014 at 9:34 AM, jivko donev jivko_...@yahoo.com wrote: We have a node with commit log director ~4G. During start-up of the node on commit log replaying the used heap space is constantly growing ending with OOM error. The heap size and new heap size properties are - 1G and 256M. We are using the default settings for commitlog_sync, commitlog_sync_period_in_ms and commitlog_segment_size_in_mb. What version of Cassandra? 1G is tiny for cassandra heap. There is a direct relationship between the data in the commitlog and memtables and in the heap. You almost certainly need more heap or less commitlog. =Rob smime.p7s Description: S/MIME cryptographic signature
