[
https://issues.apache.org/jira/browse/CASSANDRA-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-3537:
--------------------------------------
Attachment: 3537-v2.txt
v2 attached that address this for DSTPE in general. Also fixes a double-log of
RuntimeExceptions from ThreadPoolExecutor in our DTPE afterExecute code.
The culprit for this last part is this fragment from TPE.runTask:
{code}
. try {
task.run();
ran = true;
afterExecute(task, null);
++completedTasks;
} catch (RuntimeException ex) {
if (!ran)
afterExecute(task, ex);
throw ex;
}
{code}
That is, for "raw" exceptions thrown by the Task (and not wrapped inside
FutureTask), runTask will already re-throw the exception, allowing our default
uncaught exception handler to log it. So we don't want to double-log that in
DTPE.
> ExpiringMap timer is not exception-proof
> ----------------------------------------
>
> Key: CASSANDRA-3537
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3537
> Project: Cassandra
> Issue Type: Bug
> Environment: 当前堆大小:
> 5,815,955 Kb
> 堆大小的最大值:
> 6,045,696 Kb
> 分配的内存:
> 6,045,696 Kb
> 暂挂结束操作:
> 0 个对象
> 垃圾收集器:
> Name = 'ParNew', Collections = 3,294, Total time spent = 2 minutes
> 垃圾收集器:
> Name = 'ConcurrentMarkSweep', Collections = 5,909, Total time spent = 2 hours
> 17 minutes
>
> 操作系统:
> Linux 2.6.32.12-0.7-default
> 体系结构:
> amd64
> 处理器的数目:
> 16
> 分配的虚拟内存:
> 42,748,416 Kb
> 物理内存总量:
> 24,568,836 Kb
> 可用物理内存:
> 7,136,380 Kb
> 交换空间总量:
> 2,104,472 Kb
> 可用交换空间:
> 1,970,800 Kb
>
> VM 参数:
> -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
> -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.port=9000
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dpasswd.properties=/opt/obs/cassandra/conf/passwd.properties
> -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true
> 类路径:
> /opt/obs/cassandra/conf:/opt/obs/cassandra/build/classes/main:/opt/obs/cassandra/build/classes/thrift:/opt/obs/cassandra/lib/antlr-3.2.jar:/opt/obs/cassandra/lib/apache-cassandra-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-clientutil-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-thrift-1.0.0.jar:/opt/obs/cassandra/lib/avro-1.4.0-fixes.jar:/opt/obs/cassandra/lib/avro-1.4.0-sources-fixes.jar:/opt/obs/cassandra/lib/cassandra_simple_authentication.jar:/opt/obs/cassandra/lib/commons-cli-1.1.jar:/opt/obs/cassandra/lib/commons-codec-1.2.jar:/opt/obs/cassandra/lib/commons-lang-2.4.jar:/opt/obs/cassandra/lib/compress-lzf-0.8.4.jar:/opt/obs/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/obs/cassandra/lib/guava-r08.jar:/opt/obs/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/obs/cassandra/lib/jackson-core-asl-1.4.0.jar:/opt/obs/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/obs/cassandra/lib/jamm-0.2.5.jar:/opt/obs/cassandra/lib/jline-0.9.94.jar:/opt/obs/cassandra/lib/json-simple-1.1.jar:/opt/obs/cassandra/lib/libthrift-0.6.jar:/opt/obs/cassandra/lib/log4j-1.2.16.jar:/opt/obs/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/obs/cassandra/lib/slf4j-api-1.6.1.jar:/opt/obs/cassandra/lib/slf4j-log4j12-1.6.1.jar:/opt/obs/cassandra/lib/snakeyaml-1.6.jar:/opt/obs/cassandra/lib/snappy-java-1.0.3.jar
> Reporter: MaHaiyang
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 1.0.9
>
> Attachments: 3537-v2.txt, 3537.txt
>
>
> I have 4 cassandra nodes ,and I put about 30G data to db for every nodes .
> It's just 4 days before I start the cluster ,but now every 4 nodes have the
> same problem ,JVM heap is full ,and GC take no effect ,There must be some
> memory leak . Jmap the memory as follow:
> Object Histogram:
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 15793606 758093088 java.nio.HeapByteBuffer
> 2: 2153811 320138208 java.lang.Object[]
> 3: 6163192 197222144 org.apache.cassandra.db.Column
> 4: 2543836 175890256 int[]
> 5: 2168816 155397192 long[]
> 6: 2078123 116374888
> org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> 7: 1847111 73884440 java.math.BigInteger
> 8: 1234243 59243664 java.util.Hashtable
> 9: 1770829 58233000 char[]
> 10: 1770627 56660064 java.lang.String
> 11: 1665886 39981264 org.apache.cassandra.db.DecoratedKey
> 12: 692706 38791536
> org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
> 13: 1234274 37172088 java.util.Hashtable$Entry[]
> 14: 1133541 36273312 java.net.Inet4Address
> 15: 738528 35449344
> org.apache.cassandra.service.ReadCallback
> 16: 2078118 33249888 org.cliffc.high_scale_lib.Counter
> 17: 1373886 32973264 org.apache.cassandra.db.ReadResponse
> 18: 1234023 29616552 org.apache.cassandra.net.Message
> 19: 1234019 29616456 org.apache.cassandra.net.Header
> 20: 1846185 29538960 org.apache.cassandra.dht.BigIntegerToken
> 21: 891378 28524096
> org.apache.cassandra.utils.ExpiringMap$CacheableObject
> 22: 692706 27708240
> org.cliffc.high_scale_lib.NonBlockingHashMap
> 23: 1148252 27558048 java.util.Collections$SynchronizedSet
> 24: 541977 26014896
> org.apache.cassandra.db.SliceFromReadCommand
> 25: 998001 23952024
> java.util.concurrent.ConcurrentSkipListMap$Node
> 26: 928792 22291008 java.util.ArrayList
> 27: 692715 22166880
> java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
> 28: 891378 21393072 org.apache.cassandra.net.CallbackInfo
> 29: 1148247 18371952 java.util.Hashtable$KeySet
> 30: 731859 17564616 org.apache.cassandra.db.Row
> 31: 529991 16959712
> org.apache.cassandra.db.ArrayBackedSortedColumns
> 32: 691425 16594200
> org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
> 33: 648580 15565920 org.apache.cassandra.db.filter.QueryPath
> 34: 648338 15560112
> org.apache.cassandra.service.RowDigestResolver
> 35: 971376 15542016
> java.util.concurrent.atomic.AtomicInteger
> 36: 837418 13398688
> org.apache.cassandra.utils.SimpleCondition
> 37: 535614 12854736 org.apache.cassandra.db.ColumnFamily
> 38: 725634 11610144
> java.util.concurrent.atomic.AtomicReference
> 39: 195117 9365616 org.apache.cassandra.db.ThreadSafeSortedColumns
> 40: 281921 9021472
> java.util.concurrent.ConcurrentSkipListMap$HeadIndex
> 41: 277679 8885728
> java.util.concurrent.locks.ReentrantLock$NonfairSync
> 42: 314424 7546176 java.util.concurrent.ConcurrentSkipListMap$Index
> 43: 275186 6604464
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
> 44: 270280 6486720 java.util.concurrent.LinkedBlockingQueue$Node
> 45: 219553 5269272
> org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
> 46: 106436 5108928 java.util.TreeMap
> 47: 122185 4887400 org.apache.cassandra.db.ExpiringColumn
> 48: 189968 4559232 org.apache.cassandra.db.SuperColumn
> 49: 275659 4410544 java.util.concurrent.locks.ReentrantLock
> 50: 90213 4330224 java.util.concurrent.LinkedBlockingQueue
> 51: 107026 4281040 java.util.TreeMap$Entry
> 52: 30501 4222056 * ConstMethodKlass
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
