Okay, is it better written as NumMemtables = 1 + 2 * AvailableProcessors + NumDataFileDirectory
Thus estimated maximum memory is MemtableMemoryUsage = MemtableSizeInMB * NumMemtables So in my case where I have 2 core machines with 3 datafile directories MemtableMemoryUsage = 512 * (1 + 2 * 2 + 3) = 4096MB which is basically greater than the 4G I've given to Xmx, thus the OOM? -Anthony On Fri, Feb 12, 2010 at 03:09:10PM -0600, Jonathan Ellis wrote: > 0.5 allows 1 + 2 * Runtime.getRuntime().availableProcessors() > Memtables + 1 per DataFileLocation to be waiting for flush before it > will block writes (or log replay) to give those time to flush out. > So, it sounds like you just need to lower your Memtable max > size/object count. > > On Fri, Feb 12, 2010 at 2:52 PM, Anthony Molinaro > <antho...@alumni.caltech.edu> wrote: > > 0.5.0 final. I was able to get things going again by upping the memory > > then lowering it after a successful restart, but I would like to know how > > to minimize the chances of OOM via tuning. > > > > -Anthony > > > > On Thu, Feb 11, 2010 at 01:29:16PM -0600, Jonathan Ellis wrote: > >> What version are you on these days? :) > >> > >> On Thu, Feb 11, 2010 at 12:11 PM, Anthony Molinaro > >> <antho...@alumni.caltech.edu> wrote: > >> > Hi, > >> > > >> > I've been having nodes failing recently with OOM exceptions (not sure > >> > why, but we have had an increase in traffic so that could be a cause). > >> > Most nodes have restarted fine, one node however, has been having > >> > problems > >> > restarting. It was failing with > >> > > >> > java.lang.OutOfMemoryError: Java heap space > >> > at java.util.Arrays.copyOfRange(Arrays.java:3209) > >> > at java.lang.String.<init>(String.java:216) > >> > at java.io.DataInputStream.readUTF(DataInputStream.java:644) > >> > at java.io.DataInputStream.readUTF(DataInputStream.java:547) > >> > at > >> > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:104) > >> > at > >> > org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:308) > >> > at > >> > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:318) > >> > at > >> > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:271) > >> > at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:338) > >> > at > >> > org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65) > >> > at > >> > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90) > >> > at > >> > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:166) > >> > > >> > And > >> > > >> > java.lang.OutOfMemoryError: Java heap space > >> > at java.lang.StringCoding.encode(StringCoding.java:266) > >> > at java.lang.StringCoding.encode(StringCoding.java:284) > >> > at java.lang.String.getBytes(String.java:987) > >> > at > >> > org.apache.cassandra.utils.FBUtilities.hash(FBUtilities.java:178) > >> > at > >> > org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:116) > >> > at > >> > org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:44) > >> > at org.apache.cassandra.db.Memtable.resolve(Memtable.java:148) > >> > at org.apache.cassandra.db.Memtable.put(Memtable.java:143) > >> > at > >> > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:478) > >> > at org.apache.cassandra.db.Table.apply(Table.java:445) > >> > at org.apache.cassandra.db.CommitLog$3.run(CommitLog.java:365) > >> > at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> > at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> > at java.lang.Thread.run(Thread.java:619) > >> > > >> > I upped the Xmx value from 4G to 6G and it seems to be doing okay, > >> > however > >> > it seems odd that it can run mostly fine with 4G, but fail to restart > >> > with > >> > that much memory. Maybe this ticket's issue is back? > >> > > >> > https://issues.apache.org/jira/browse/CASSANDRA-609 > >> > > >> > Anyway, I'm hoping thing will recover with 6G then I can restart again > >> > with 4G and things will be good. > >> > > >> > I'd also like a better understanding of why cassandra might OOM in > >> > general. > >> > Are there settings which minimize the chances of OOM? This instance has > >> > 2 column families and I have > >> > > >> > <MemtableSizeInMB>512</MemtableSizeInMB> > >> > <MemtableObjectCountInMillions>1.0</MemtableObjectCountInMillions> > >> > <MemtableFlushAfterMinutes>1440</MemtableFlushAfterMinutes> > >> > > >> > So if I understand these settings, memtables can at most be 512MB in size > >> > or consist of 1 million objects before they are flushed to disk. The > >> > maximum > >> > time before they will be flushed is 24 hours. So does that mean if I > >> > fill > >> > up 8G or 16 memtables in less than 24 hours, I've basically used all the > >> > memory available to me? I assume there are other things using memory, > >> > (indexes, etc), how is that limited? Anyway, any information about what > >> > is used where would be appreciated. > >> > > >> > Thanks, > >> > > >> > -Anthony > >> > > >> > -- > >> > ------------------------------------------------------------------------ > >> > Anthony Molinaro <antho...@alumni.caltech.edu> > >> > > > > > -- > > ------------------------------------------------------------------------ > > Anthony Molinaro <antho...@alumni.caltech.edu> > > -- ------------------------------------------------------------------------ Anthony Molinaro <antho...@alumni.caltech.edu>
