JVM heap is full and GC take no effect ,so many NonBlockingHashMap objects in
heap
------------------------------------------------------------------------------------
Key: CASSANDRA-3537
URL: https://issues.apache.org/jira/browse/CASSANDRA-3537
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.0.0
Environment: 当前堆大小:
5,815,955 Kb
堆大小的最大值:
6,045,696 Kb
分配的内存:
6,045,696 Kb
暂挂结束操作:
0 个对象
垃圾收集器:
Name = 'ParNew', Collections = 3,294, Total time spent = 2 minutes
垃圾收集器:
Name = 'ConcurrentMarkSweep', Collections = 5,909, Total time spent = 2 hours
17 minutes
操作系统:
Linux 2.6.32.12-0.7-default
体系结构:
amd64
处理器的数目:
16
分配的虚拟内存:
42,748,416 Kb
物理内存总量:
24,568,836 Kb
可用物理内存:
7,136,380 Kb
交换空间总量:
2,104,472 Kb
可用交换空间:
1,970,800 Kb
VM 参数:
-ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
-Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=9000
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dpasswd.properties=/opt/obs/cassandra/conf/passwd.properties -Dpasswd.mode=MD5
-Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true
类路径:
/opt/obs/cassandra/conf:/opt/obs/cassandra/build/classes/main:/opt/obs/cassandra/build/classes/thrift:/opt/obs/cassandra/lib/antlr-3.2.jar:/opt/obs/cassandra/lib/apache-cassandra-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-clientutil-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-thrift-1.0.0.jar:/opt/obs/cassandra/lib/avro-1.4.0-fixes.jar:/opt/obs/cassandra/lib/avro-1.4.0-sources-fixes.jar:/opt/obs/cassandra/lib/cassandra_simple_authentication.jar:/opt/obs/cassandra/lib/commons-cli-1.1.jar:/opt/obs/cassandra/lib/commons-codec-1.2.jar:/opt/obs/cassandra/lib/commons-lang-2.4.jar:/opt/obs/cassandra/lib/compress-lzf-0.8.4.jar:/opt/obs/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/obs/cassandra/lib/guava-r08.jar:/opt/obs/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/obs/cassandra/lib/jackson-core-asl-1.4.0.jar:/opt/obs/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/obs/cassandra/lib/jamm-0.2.5.jar:/opt/obs/cassandra/lib/jline-0.9.94.jar:/opt/obs/cassandra/lib/json-simple-1.1.jar:/opt/obs/cassandra/lib/libthrift-0.6.jar:/opt/obs/cassandra/lib/log4j-1.2.16.jar:/opt/obs/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/obs/cassandra/lib/slf4j-api-1.6.1.jar:/opt/obs/cassandra/lib/slf4j-log4j12-1.6.1.jar:/opt/obs/cassandra/lib/snakeyaml-1.6.jar:/opt/obs/cassandra/lib/snappy-java-1.0.3.jar
Reporter: MaHaiyang
I have 4 cassandra nodes ,and I put about 30G data to db for every nodes . It's
just 4 days before I start the cluster ,but now every 4 nodes have the same
problem ,JVM heap is full ,and GC take no effect ,There must be some memory
leak . Jmap the memory as follow:
Object Histogram:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 15793606 758093088 java.nio.HeapByteBuffer
2: 2153811 320138208 java.lang.Object[]
3: 6163192 197222144 org.apache.cassandra.db.Column
4: 2543836 175890256 int[]
5: 2168816 155397192 long[]
6: 2078123 116374888
org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
7: 1847111 73884440 java.math.BigInteger
8: 1234243 59243664 java.util.Hashtable
9: 1770829 58233000 char[]
10: 1770627 56660064 java.lang.String
11: 1665886 39981264 org.apache.cassandra.db.DecoratedKey
12: 692706 38791536
org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
13: 1234274 37172088 java.util.Hashtable$Entry[]
14: 1133541 36273312 java.net.Inet4Address
15: 738528 35449344
org.apache.cassandra.service.ReadCallback
16: 2078118 33249888 org.cliffc.high_scale_lib.Counter
17: 1373886 32973264 org.apache.cassandra.db.ReadResponse
18: 1234023 29616552 org.apache.cassandra.net.Message
19: 1234019 29616456 org.apache.cassandra.net.Header
20: 1846185 29538960 org.apache.cassandra.dht.BigIntegerToken
21: 891378 28524096
org.apache.cassandra.utils.ExpiringMap$CacheableObject
22: 692706 27708240
org.cliffc.high_scale_lib.NonBlockingHashMap
23: 1148252 27558048 java.util.Collections$SynchronizedSet
24: 541977 26014896
org.apache.cassandra.db.SliceFromReadCommand
25: 998001 23952024
java.util.concurrent.ConcurrentSkipListMap$Node
26: 928792 22291008 java.util.ArrayList
27: 692715 22166880
java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
28: 891378 21393072 org.apache.cassandra.net.CallbackInfo
29: 1148247 18371952 java.util.Hashtable$KeySet
30: 731859 17564616 org.apache.cassandra.db.Row
31: 529991 16959712
org.apache.cassandra.db.ArrayBackedSortedColumns
32: 691425 16594200
org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
33: 648580 15565920 org.apache.cassandra.db.filter.QueryPath
34: 648338 15560112
org.apache.cassandra.service.RowDigestResolver
35: 971376 15542016
java.util.concurrent.atomic.AtomicInteger
36: 837418 13398688
org.apache.cassandra.utils.SimpleCondition
37: 535614 12854736 org.apache.cassandra.db.ColumnFamily
38: 725634 11610144
java.util.concurrent.atomic.AtomicReference
39: 195117 9365616 org.apache.cassandra.db.ThreadSafeSortedColumns
40: 281921 9021472
java.util.concurrent.ConcurrentSkipListMap$HeadIndex
41: 277679 8885728
java.util.concurrent.locks.ReentrantLock$NonfairSync
42: 314424 7546176 java.util.concurrent.ConcurrentSkipListMap$Index
43: 275186 6604464
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
44: 270280 6486720 java.util.concurrent.LinkedBlockingQueue$Node
45: 219553 5269272
org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
46: 106436 5108928 java.util.TreeMap
47: 122185 4887400 org.apache.cassandra.db.ExpiringColumn
48: 189968 4559232 org.apache.cassandra.db.SuperColumn
49: 275659 4410544 java.util.concurrent.locks.ReentrantLock
50: 90213 4330224 java.util.concurrent.LinkedBlockingQueue
51: 107026 4281040 java.util.TreeMap$Entry
52: 30501 4222056 * ConstMethodKlass
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
