here is my JVM options, by default, I didn't modify them, from
cassandra.in.sh
# Arguments to pass to the JVM
JVM_OPTS=" \
-ea \
-Xms128M \
-Xmx1G \
-XX:TargetSurvivorRatio=90 \
-XX:+AggressiveOpts \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-XX:+CMSParallelRemarkEnabled \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:SurvivorRatio=128 \
-XX:MaxTenuringThreshold=0 \
-Dcom.sun.management.jmxremote.port=8080 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"
and my box is normal pc with 2GB ram, Intel E3200 @ 2.40GHz. By the way, I
am using the latest Sun JDK
On Tue, Apr 20, 2010 at 9:33 AM, Schubert Zhang <zson...@gmail.com> wrote:
> Seems you should configure larger jvm-heap.
>
>
> On Tue, Apr 20, 2010 at 9:32 AM, Schubert Zhang <zson...@gmail.com> wrote:
>
>> Please also post your jvm-heap and GC options, i.e. the seting in
>> cassandra.in.sh
>> And what about you node hardware?
>>
>> On Tue, Apr 20, 2010 at 9:22 AM, Ken Sandney <bluefl...@gmail.com> wrote:
>>
>>> Hi
>>> I am doing a insert test with 9 nodes, the command:
>>>
>>>> stress.py -n 1000000000 -t 1000 -c 10 -o insert -i 5 -d
>>>> 10.0.0.1,10.0.0.2.....
>>>
>>> and 5 of the 9 nodes were cashed, only about 6'500'000 rows were
>>> inserted
>>> I checked out the system.log and seems the reason are 'out of memory'. I
>>> don't if this had something to do with my settings.
>>> Any idea about this?
>>> Thank you, and the following are the errors from system.log
>>>
>>>
>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 20:43:14,013 CassandraDaemon.java
>>>> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main]
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> at
>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
>>>
>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>
>>> at java.util.TimerThread.run(Timer.java:462)
>>>
>>> ERROR [ROW-MUTATION-STAGE:9] 2010-04-19 20:43:27,932 CassandraDaemon.java
>>>> (line 78) Fatal exception in thread Thread[ROW-MUTATION-STAGE:9,5,main]
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> at
>>>> java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:893)
>>>
>>> at
>>>> java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893)
>>>
>>> at
>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:192)
>>>
>>> at
>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:118)
>>>
>>> at
>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:108)
>>>
>>> at
>>>> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:359)
>>>
>>> at
>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:369)
>>>
>>> at
>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:322)
>>>
>>> at
>>>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:45)
>>>
>>> at
>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>
>>> at java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>> and another
>>>
>>> INFO [GC inspection] 2010-04-19 21:13:09,034 GCInspector.java (line 110)
>>>> GC for ConcurrentMarkSweep: 2016 ms, 1239096 reclaimed leaving 1094238944
>>>> used; max is 1211826176
>>>
>>> ERROR [Thread-14] 2010-04-19 21:23:18,508 CassandraDaemon.java (line 78)
>>>> Fatal exception in thread Thread[Thread-14,5,main]
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> at sun.nio.ch.Util.releaseTemporaryDirectBuffer(Util.java:67)
>>>
>>> at sun.nio.ch.IOUtil.read(IOUtil.java:212)
>>>
>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>>>
>>> at
>>>> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
>>>
>>> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
>>>
>>> at java.io.InputStream.read(InputStream.java:85)
>>>
>>> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:64)
>>>
>>> at java.io.DataInputStream.readInt(DataInputStream.java:370)
>>>
>>> at
>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:70)
>>>
>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:23:18,514
>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>>>
>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>>>> heap space
>>>
>>> at
>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>>>
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>>
>>> at
>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>>>
>>> at
>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>
>>> at java.lang.Thread.run(Thread.java:619)
>>>
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>
>>> INFO [FLUSH-WRITER-POOL:1] 2010-04-19 21:23:25,600 Memtable.java (line
>>>> 162) Completed flushing /m/cassandra/data/Keyspace1/Standard1-623-Data.db
>>>
>>> ERROR [Thread-13] 2010-04-19 21:23:18,514 CassandraDaemon.java (line 78)
>>>> Fatal exception in thread Thread[Thread-13,5,main]
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> ERROR [Thread-15] 2010-04-19 21:23:18,514 CassandraDaemon.java (line 78)
>>>> Fatal exception in thread Thread[Thread-15,5,main]
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:23:18,514 CassandraDaemon.java
>>>> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main]
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>>
>>> and
>>>
>>> INFO 21:00:31,319 GC for ConcurrentMarkSweep: 1417 ms, 206216 reclaimed
>>>> leaving 1094527752 used; max is 1211826176
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> Dumping heap to java_pid28670.hprof ...
>>>
>>> INFO 21:01:23,882 GC for ConcurrentMarkSweep: 2100 ms, 734008 reclaimed
>>>> leaving 1093996648 used; max is 1211826176
>>>
>>> Heap dump file created [1095841554 bytes in 12.960 secs]
>>>
>>> INFO 21:01:45,082 GC for ConcurrentMarkSweep: 2089 ms, 769968 reclaimed
>>>> leaving 1093960776 used; max is 1211826176
>>>
>>> ERROR 21:01:49,559 Fatal exception in thread Thread[Hint delivery,5,main]
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> at java.util.Arrays.copyOf(Arrays.java:2772)
>>>
>>> at java.util.Arrays.copyOf(Arrays.java:2746)
>>>
>>> at java.util.ArrayList.ensureCapacity(ArrayList.java:187)
>>>
>>> at java.util.ArrayList.add(ArrayList.java:378)
>>>
>>> at
>>>> java.util.concurrent.ConcurrentSkipListMap.toList(ConcurrentSkipListMap.java:2341)
>>>
>>> at
>>>> java.util.concurrent.ConcurrentSkipListMap$Values.toArray(ConcurrentSkipListMap.java:2445)
>>>
>>> at
>>>> org.apache.cassandra.db.Memtable.getSliceIterator(Memtable.java:207)
>>>
>>> at
>>>> org.apache.cassandra.db.filter.SliceQueryFilter.getMemColumnIterator(SliceQueryFilter.java:58)
>>>
>>> at
>>>> org.apache.cassandra.db.filter.QueryFilter.getMemColumnIterator(QueryFilter.java:53)
>>>
>>> at
>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:816)
>>>
>>> at
>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
>>>
>>> at
>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
>>>
>>> at
>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:175)
>>>
>>> at
>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
>>>
>>> at
>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
>>>
>>> at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>
>>> at java.lang.Thread.run(Thread.java:636)
>>>
>>> INFO 21:01:56,123 GC for ConcurrentMarkSweep: 2115 ms, 893240 reclaimed
>>>> leaving 1093862712 used; max is 1211826176
>>>
>>>
>>> and
>>>
>>> ERROR [Hint delivery] 2010-04-19 21:57:07,089 CassandraDaemon.java (line
>>>> 78) Fatal exception in thread Thread[Hint delivery,5,main]
>>>
>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>>>> heap space
>>>
>>> at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>>
>>> at java.lang.Thread.run(Thread.java:636)
>>>
>>> Caused by: java.lang.RuntimeException:
>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>>>> heap space
>>>
>>> at
>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:209)
>>>
>>> at
>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
>>>
>>> at
>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
>>>
>>> at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>
>>> ... 1 more
>>>
>>> Caused by: java.util.concurrent.ExecutionException:
>>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> at
>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>>>
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>>
>>> at
>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:205)
>>>
>>> ... 4 more
>>>
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>
>>> ERROR [MESSAGE-DESERIALIZER-POOL:1] 2010-04-19 21:57:07,089
>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>>>
>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>>>> heap space
>>>
>>> at
>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>>>
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>>
>>> at
>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>
>>> at java.lang.Thread.run(Thread.java:636)
>>>
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>
>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:57:07,089
>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>>>
>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>>>> heap space
>>>
>>> at
>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>>>
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>>
>>> at
>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>>>
>>> at
>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
>>>
>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>
>>> at java.lang.Thread.run(Thread.java:636)
>>>
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>
>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:56:29,572 CassandraDaemon.java
>>>> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main]
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> at java.util.HashMap.<init>(HashMap.java:226) at
>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
>>>> at java.util.TimerThread.mainLoop(Timer.java:534) at
>>>> java.util.TimerThread.run(Timer.java:484)
>>>
>>>
>>>
>>>
>>
>>
>
