What consistencyLevel are you inserting the elements? If you do ./bin/nodeprobe -host localhost tpstats on each machine do you see one metric that has a lot of pending items?
On Sun, Dec 13, 2009 at 8:14 AM, Brian Burruss <[email protected]> wrote: > another OOM exception. the only thing interesting about my testing is that > there are 2 servers, RF=2, W=1, R=1 ... there is 248G of data on each > server. I have -Xmx3G assigned to each server > > 2009-12-12 22:04:37,436 ERROR [pool-1-thread-309] [Cassandra.java:734] > Internal error processing get > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:523) > at > org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:373) > at > org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:92) > at > org.apache.cassandra.service.CassandraServer.multigetColumns(CassandraServer.java:265) > at > org.apache.cassandra.service.CassandraServer.multigetInternal(CassandraServer.java:320) > at > org.apache.cassandra.service.CassandraServer.get(CassandraServer.java:253) > at > org.apache.cassandra.service.Cassandra$Processor$get.process(Cassandra.java:724) > at > org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:712) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > > ________________________________________ > From: Brian Burruss > Sent: Saturday, December 12, 2009 7:45 AM > To: [email protected] > Subject: OOM Exception > > this happened after cassandra was running for a couple of days. I have > -Xmx3G on JVM. > > is there any other info you need so this makes sense? > > thx! > > > 2009-12-11 21:38:37,216 ERROR [HINTED-HANDOFF-POOL:1] > [DebuggableThreadPoolExecutor.java:157] Error in ThreadPoolExecutor > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.io.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151) > at > org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144) > at > org.apache.cassandra.io.SSTableWriter.<init>(SSTableWriter.java:53) > at > org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:911) > at > org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:855) > at > org.apache.cassandra.db.ColumnFamilyStore.doMajorCompactionInternal(ColumnFamilyStore.java:698) > at > org.apache.cassandra.db.ColumnFamilyStore.doMajorCompaction(ColumnFamilyStore.java:670) > at > org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:190) > at > org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:75) > at > org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:249) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > -- Dan Di Spaltro
