[
https://issues.apache.org/jira/browse/CASSANDRA-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161363#comment-13161363
]
MaHaiyang commented on CASSANDRA-3537:
--------------------------------------
After I set jamm correctly , I do a new test , and haven't find the memory
problem in 20 hours .But I could't confirm it's "jamm not specified as java
agent" result the heap problem . I analysised the dump memory info of the node
which had the heap problem , they nearly have the same thing : too many
Message objects and ralated objects like below,although I have stoped reading
and writing data for a few minutes and empty all memtables .
#instances #bytes Class description
---------------------------------------------
900654 21615696 org.apache.cassandra.net.Message
900653 21615672 org.apache.cassandra.net.Header
341351 16384848 org.apache.cassandra.service.ReadCallback
338502 13540080 org.apache.cassandra.db.RangeSliceCommand
338500 10832000 org.apache.cassandra.thrift.SliceRange
338502 8124048 org.apache.cassandra.thrift.SlicePredicate
343328 5493248 org.apache.cassandra.utils.SimpleCondition
Is there someone can tell me ,whether there are some relationship between "jamm
not specified as java agent" and so many
objects like above ?
> JVM heap is full and GC take no effect ,so many NonBlockingHashMap objects
> in heap
> ------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3537
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3537
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Environment: 当前堆大小:
> 5,815,955 Kb
> 堆大小的最大值:
> 6,045,696 Kb
> 分配的内存:
> 6,045,696 Kb
> 暂挂结束操作:
> 0 个对象
> 垃圾收集器:
> Name = 'ParNew', Collections = 3,294, Total time spent = 2 minutes
> 垃圾收集器:
> Name = 'ConcurrentMarkSweep', Collections = 5,909, Total time spent = 2 hours
> 17 minutes
>
> 操作系统:
> Linux 2.6.32.12-0.7-default
> 体系结构:
> amd64
> 处理器的数目:
> 16
> 分配的虚拟内存:
> 42,748,416 Kb
> 物理内存总量:
> 24,568,836 Kb
> 可用物理内存:
> 7,136,380 Kb
> 交换空间总量:
> 2,104,472 Kb
> 可用交换空间:
> 1,970,800 Kb
>
> VM 参数:
> -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
> -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.port=9000
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dpasswd.properties=/opt/obs/cassandra/conf/passwd.properties
> -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true
> 类路径:
> /opt/obs/cassandra/conf:/opt/obs/cassandra/build/classes/main:/opt/obs/cassandra/build/classes/thrift:/opt/obs/cassandra/lib/antlr-3.2.jar:/opt/obs/cassandra/lib/apache-cassandra-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-clientutil-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-thrift-1.0.0.jar:/opt/obs/cassandra/lib/avro-1.4.0-fixes.jar:/opt/obs/cassandra/lib/avro-1.4.0-sources-fixes.jar:/opt/obs/cassandra/lib/cassandra_simple_authentication.jar:/opt/obs/cassandra/lib/commons-cli-1.1.jar:/opt/obs/cassandra/lib/commons-codec-1.2.jar:/opt/obs/cassandra/lib/commons-lang-2.4.jar:/opt/obs/cassandra/lib/compress-lzf-0.8.4.jar:/opt/obs/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/obs/cassandra/lib/guava-r08.jar:/opt/obs/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/obs/cassandra/lib/jackson-core-asl-1.4.0.jar:/opt/obs/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/obs/cassandra/lib/jamm-0.2.5.jar:/opt/obs/cassandra/lib/jline-0.9.94.jar:/opt/obs/cassandra/lib/json-simple-1.1.jar:/opt/obs/cassandra/lib/libthrift-0.6.jar:/opt/obs/cassandra/lib/log4j-1.2.16.jar:/opt/obs/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/obs/cassandra/lib/slf4j-api-1.6.1.jar:/opt/obs/cassandra/lib/slf4j-log4j12-1.6.1.jar:/opt/obs/cassandra/lib/snakeyaml-1.6.jar:/opt/obs/cassandra/lib/snappy-java-1.0.3.jar
> Reporter: MaHaiyang
>
> I have 4 cassandra nodes ,and I put about 30G data to db for every nodes .
> It's just 4 days before I start the cluster ,but now every 4 nodes have the
> same problem ,JVM heap is full ,and GC take no effect ,There must be some
> memory leak . Jmap the memory as follow:
> Object Histogram:
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 15793606 758093088 java.nio.HeapByteBuffer
> 2: 2153811 320138208 java.lang.Object[]
> 3: 6163192 197222144 org.apache.cassandra.db.Column
> 4: 2543836 175890256 int[]
> 5: 2168816 155397192 long[]
> 6: 2078123 116374888
> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> 7: 1847111 73884440 java.math.BigInteger
> 8: 1234243 59243664 java.util.Hashtable
> 9: 1770829 58233000 char[]
> 10: 1770627 56660064 java.lang.String
> 11: 1665886 39981264 org.apache.cassandra.db.DecoratedKey
> 12: 692706 38791536
> org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
> 13: 1234274 37172088 java.util.Hashtable$Entry[]
> 14: 1133541 36273312 java.net.Inet4Address
> 15: 738528 35449344
> org.apache.cassandra.service.ReadCallback
> 16: 2078118 33249888 org.cliffc.high_scale_lib.Counter
> 17: 1373886 32973264 org.apache.cassandra.db.ReadResponse
> 18: 1234023 29616552 org.apache.cassandra.net.Message
> 19: 1234019 29616456 org.apache.cassandra.net.Header
> 20: 1846185 29538960 org.apache.cassandra.dht.BigIntegerToken
> 21: 891378 28524096
> org.apache.cassandra.utils.ExpiringMap$CacheableObject
> 22: 692706 27708240
> org.cliffc.high_scale_lib.NonBlockingHashMap
> 23: 1148252 27558048 java.util.Collections$SynchronizedSet
> 24: 541977 26014896
> org.apache.cassandra.db.SliceFromReadCommand
> 25: 998001 23952024
> java.util.concurrent.ConcurrentSkipListMap$Node
> 26: 928792 22291008 java.util.ArrayList
> 27: 692715 22166880
> java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
> 28: 891378 21393072 org.apache.cassandra.net.CallbackInfo
> 29: 1148247 18371952 java.util.Hashtable$KeySet
> 30: 731859 17564616 org.apache.cassandra.db.Row
> 31: 529991 16959712
> org.apache.cassandra.db.ArrayBackedSortedColumns
> 32: 691425 16594200
> org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
> 33: 648580 15565920 org.apache.cassandra.db.filter.QueryPath
> 34: 648338 15560112
> org.apache.cassandra.service.RowDigestResolver
> 35: 971376 15542016
> java.util.concurrent.atomic.AtomicInteger
> 36: 837418 13398688
> org.apache.cassandra.utils.SimpleCondition
> 37: 535614 12854736 org.apache.cassandra.db.ColumnFamily
> 38: 725634 11610144
> java.util.concurrent.atomic.AtomicReference
> 39: 195117 9365616 org.apache.cassandra.db.ThreadSafeSortedColumns
> 40: 281921 9021472
> java.util.concurrent.ConcurrentSkipListMap$HeadIndex
> 41: 277679 8885728
> java.util.concurrent.locks.ReentrantLock$NonfairSync
> 42: 314424 7546176 java.util.concurrent.ConcurrentSkipListMap$Index
> 43: 275186 6604464
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
> 44: 270280 6486720 java.util.concurrent.LinkedBlockingQueue$Node
> 45: 219553 5269272
> org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
> 46: 106436 5108928 java.util.TreeMap
> 47: 122185 4887400 org.apache.cassandra.db.ExpiringColumn
> 48: 189968 4559232 org.apache.cassandra.db.SuperColumn
> 49: 275659 4410544 java.util.concurrent.locks.ReentrantLock
> 50: 90213 4330224 java.util.concurrent.LinkedBlockingQueue
> 51: 107026 4281040 java.util.TreeMap$Entry
> 52: 30501 4222056 * ConstMethodKlass
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
