right, and remembering that MemtableSizeInMB is just the size of the values, you can estimate around 50% overhead on top of that.
On Fri, Feb 12, 2010 at 6:04 PM, Anthony Molinaro <antho...@alumni.caltech.edu> wrote: > Okay, is it better written as > > NumMemtables = 1 + 2 * AvailableProcessors + NumDataFileDirectory > > Thus estimated maximum memory is > > MemtableMemoryUsage = MemtableSizeInMB * NumMemtables > > So in my case where I have 2 core machines with 3 datafile directories > > MemtableMemoryUsage = 512 * (1 + 2 * 2 + 3) = 4096MB > > which is basically greater than the 4G I've given to Xmx, thus the OOM? > > -Anthony > > On Fri, Feb 12, 2010 at 03:09:10PM -0600, Jonathan Ellis wrote: >> 0.5 allows 1 + 2 * Runtime.getRuntime().availableProcessors() >> Memtables + 1 per DataFileLocation to be waiting for flush before it >> will block writes (or log replay) to give those time to flush out. >> So, it sounds like you just need to lower your Memtable max >> size/object count. >> >> On Fri, Feb 12, 2010 at 2:52 PM, Anthony Molinaro >> <antho...@alumni.caltech.edu> wrote: >> > 0.5.0 final. I was able to get things going again by upping the memory >> > then lowering it after a successful restart, but I would like to know how >> > to minimize the chances of OOM via tuning. >> > >> > -Anthony >> > >> > On Thu, Feb 11, 2010 at 01:29:16PM -0600, Jonathan Ellis wrote: >> >> What version are you on these days? :) >> >> >> >> On Thu, Feb 11, 2010 at 12:11 PM, Anthony Molinaro >> >> <antho...@alumni.caltech.edu> wrote: >> >> > Hi, >> >> > >> >> > I've been having nodes failing recently with OOM exceptions (not sure >> >> > why, but we have had an increase in traffic so that could be a cause). >> >> > Most nodes have restarted fine, one node however, has been having >> >> > problems >> >> > restarting. It was failing with >> >> > >> >> > java.lang.OutOfMemoryError: Java heap space >> >> > at java.util.Arrays.copyOfRange(Arrays.java:3209) >> >> > at java.lang.String.<init>(String.java:216) >> >> > at java.io.DataInputStream.readUTF(DataInputStream.java:644) >> >> > at java.io.DataInputStream.readUTF(DataInputStream.java:547) >> >> > at >> >> > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:104) >> >> > at >> >> > org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:308) >> >> > at >> >> > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:318) >> >> > at >> >> > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:271) >> >> > at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:338) >> >> > at >> >> > org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65) >> >> > at >> >> > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90) >> >> > at >> >> > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:166) >> >> > >> >> > And >> >> > >> >> > java.lang.OutOfMemoryError: Java heap space >> >> > at java.lang.StringCoding.encode(StringCoding.java:266) >> >> > at java.lang.StringCoding.encode(StringCoding.java:284) >> >> > at java.lang.String.getBytes(String.java:987) >> >> > at >> >> > org.apache.cassandra.utils.FBUtilities.hash(FBUtilities.java:178) >> >> > at >> >> > org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:116) >> >> > at >> >> > org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:44) >> >> > at org.apache.cassandra.db.Memtable.resolve(Memtable.java:148) >> >> > at org.apache.cassandra.db.Memtable.put(Memtable.java:143) >> >> > at >> >> > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:478) >> >> > at org.apache.cassandra.db.Table.apply(Table.java:445) >> >> > at org.apache.cassandra.db.CommitLog$3.run(CommitLog.java:365) >> >> > at >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> > at >> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >> > at java.lang.Thread.run(Thread.java:619) >> >> > >> >> > I upped the Xmx value from 4G to 6G and it seems to be doing okay, >> >> > however >> >> > it seems odd that it can run mostly fine with 4G, but fail to restart >> >> > with >> >> > that much memory. Maybe this ticket's issue is back? >> >> > >> >> > https://issues.apache.org/jira/browse/CASSANDRA-609 >> >> > >> >> > Anyway, I'm hoping thing will recover with 6G then I can restart again >> >> > with 4G and things will be good. >> >> > >> >> > I'd also like a better understanding of why cassandra might OOM in >> >> > general. >> >> > Are there settings which minimize the chances of OOM? This instance has >> >> > 2 column families and I have >> >> > >> >> > <MemtableSizeInMB>512</MemtableSizeInMB> >> >> > <MemtableObjectCountInMillions>1.0</MemtableObjectCountInMillions> >> >> > <MemtableFlushAfterMinutes>1440</MemtableFlushAfterMinutes> >> >> > >> >> > So if I understand these settings, memtables can at most be 512MB in >> >> > size >> >> > or consist of 1 million objects before they are flushed to disk. The >> >> > maximum >> >> > time before they will be flushed is 24 hours. So does that mean if I >> >> > fill >> >> > up 8G or 16 memtables in less than 24 hours, I've basically used all the >> >> > memory available to me? I assume there are other things using memory, >> >> > (indexes, etc), how is that limited? Anyway, any information about what >> >> > is used where would be appreciated. >> >> > >> >> > Thanks, >> >> > >> >> > -Anthony >> >> > >> >> > -- >> >> > ------------------------------------------------------------------------ >> >> > Anthony Molinaro <antho...@alumni.caltech.edu> >> >> > >> > >> > -- >> > ------------------------------------------------------------------------ >> > Anthony Molinaro <antho...@alumni.caltech.edu> >> > > > -- > ------------------------------------------------------------------------ > Anthony Molinaro <antho...@alumni.caltech.edu> >
