OOME found on the same cluster as mentioned in
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-td9443.html
Recap the setup here:
We have 2 machines (M1, M2)
In M1 running a Ignite server (I1) and a Tomcat server (T1), which hosted 2
Ignite clients (C1a, C1b),
similarly in M2 running a Ignite server (I2) and a Tomcat server (T2), which
hosted 2 Ignite clients (C2a, C2b).
After restarting I1, I2, T1, T2 on 2016-12-07, we got an OOME from T2 on
2016-12-10.
However the heap dump seems show a different pattern.
The histogram shows the memory were held by some
java.util.concurrent.LinkedBlockingDeque objects
Class Name
| Objects | Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------
| | |
java.util.concurrent.LinkedBlockingDeque
| 17 | 680 | >= 1,787,746,904
java.util.concurrent.LinkedBlockingDeque$Node
| 50,954 | 1,222,896 | >= 1,426,229,816
java.util.ArrayList
| 254,999 | 6,119,976 | >= 1,339,620,328
java.lang.Object[]
| 257,817 | 69,919,176 | >= 1,339,571,872
java.util.HashMap
| 127,003 | 6,096,144 | >= 1,300,721,992
java.util.HashMap$Node[]
| 83,427 | 163,804,808 | >= 1,297,201,592
java.util.HashMap$Node
| 19,930,286 | 637,769,152 | >= 1,249,981,888
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker
| 2 | 112 | >= 893,476,648
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker
| 2 | 112 | >= 893,322,888
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
| 21,656 | 3,464,960 | >= 743,337,608
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
| 21,656 | 3,464,960 | >= 742,465,888
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$5|
768 | 18,432 | >= 450,735,512
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$5|
768 | 18,432 | >= 450,735,512
org.apache.ignite.internal.processors.cache.CacheAffinityChangeMessage
| 768 | 30,720 | >= 450,717,088
org.apache.ignite.internal.processors.cache.CacheAffinityChangeMessage
| 768 | 30,720 | >= 450,717,088
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage
| 768 | 49,152 | >= 450,594,256
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage
| 768 | 49,152 | >= 450,594,256
java.util.Collections$UnmodifiableMap
| 15,084 | 482,688 | >= 371,291,912
java.lang.String
| 3,517,909 | 84,429,816 | >= 353,946,616
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap
| 4,613 | 295,232 | >= 307,249,264
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap
| 4,613 | 295,232 | >= 307,249,264
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap2
| 9,226 | 369,040 | >= 306,363,640
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap2
| 9,226 | 369,040 | >= 306,363,640
char[]
| 3,506,511 | 276,762,784 | >= 276,762,784
org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode
| 7,495 | 839,440 | >= 194,134,936
org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode
| 7,495 | 839,440 | >= 193,971,624
java.lang.Integer
| 11,436,504 | 182,984,064 | >= 182,984,688
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$6|
7,305,804 | 175,339,296 | >= 175,339,304
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$6|
7,305,804 | 175,339,296 | >= 175,339,304
org.apache.ignite.internal.processors.cache.DynamicCacheChangeRequest
| 6,697 | 375,032 | >= 53,361,672
org.apache.ignite.internal.processors.cache.DynamicCacheChangeRequest
| 6,697 | 375,032 | >= 53,361,672
org.apache.ignite.configuration.CacheConfiguration
| 6,708 | 1,931,904 | >= 51,855,840
org.apache.ignite.configuration.CacheConfiguration
| 6,708 | 1,931,904 | >= 51,855,840
org.apache.ignite.internal.processors.cache.CacheDefaultBinaryAffinityKeyMapper
| 6,704 | 214,528 | >= 47,625,800
org.apache.ignite.internal.processors.cache.CacheDefaultBinaryAffinityKeyMapper
| 6,704 | 214,528 | >= 47,625,448
org.apache.ignite.internal.util.GridReflectionCache
| 6,713 | 214,816 | >= 47,474,920
org.apache.ignite.internal.util.GridReflectionCache
| 6,713 | 214,816 | >= 47,474,568
org.apache.ignite.internal.util.GridBoundedConcurrentLinkedHashMap
| 13,435 | 1,074,800 | >= 47,450,888
org.apache.ignite.internal.util.GridBoundedConcurrentLinkedHashMap
| 13,435 | 1,074,800 | >= 47,438,264
java.lang.Class
| 20,731 | 163,864 | >= 46,959,328
org.jsr166.ConcurrentLinkedHashMap$Segment[]
| 13,443 | 1,077,664 | >= 45,515,200
org.jsr166.ConcurrentLinkedHashMap$Segment[]
| 13,443 | 1,077,664 | >= 45,502,648
org.jsr166.ConcurrentLinkedHashMap$Segment
| 215,644 | 12,076,064 | >= 44,437,544
org.jsr166.ConcurrentLinkedHashMap$Segment
| 215,644 | 12,076,064 | >= 44,424,992
org.apache.catalina.loader.WebappClassLoader
| 8 | 1,088 | >= 37,026,856
long[]
| 2,947 | 33,261,336 | >= 33,261,336
java.util.concurrent.ConcurrentLinkedQueue
| 6,448 | 154,752 | >= 31,993,720
java.util.concurrent.ConcurrentLinkedQueue$Node
| 12,223 | 293,352 | >= 31,838,960
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
| 432,200 | 20,745,600 | >= 27,660,984
java.util.concurrent.ConcurrentHashMap
| 2,030 | 129,920 | >= 21,635,032
java.util.concurrent.ConcurrentHashMap$Node[]
| 1,409 | 708,096 | >= 21,518,264
java.util.concurrent.ConcurrentHashMap$Node
| 57,416 | 1,837,312 | >= 20,772,768
org.HdrHistogram.Histogram
| 1,000 | 168,000 | >= 16,992,000
org.HdrHistogram.Histogram
| 1,000 | 168,000 | >= 16,992,000
com.netflix.hystrix.metric.CachedValuesHistogram
| 10 | 1,280 | >= 15,996,376
com.netflix.hystrix.metric.CachedValuesHistogram
| 14 | 1,792 | >= 15,588,504
java.lang.ThreadLocal$ThreadLocalMap
| 5,056 | 121,344 | >= 12,869,816
java.lang.ThreadLocal$ThreadLocalMap$Entry[]
| 5,056 | 482,176 | >= 12,748,464
org.apache.ignite.events.DiscoveryEvent
| 14,960 | 957,440 | >= 12,736,312
org.apache.ignite.events.DiscoveryEvent
| 14,960 | 957,440 | >= 12,736,240
java.util.HashSet
| 65,432 | 1,046,912 | >= 11,882,408
Total: 61 of 20,722 entries; 20,661 more
| 59,227,063 | 2,020,763,584 |
-------------------------------------------------------------------------------------------------------------------------------------------------------
We found two LinkedBlockingDeque together holding 88% of memory, here's one
of them:
Class Name | Shallow
Heap | Retained Heap | Percentage
---------------------------------------------------------------------------------------------------------
|
| |
java.util.concurrent.LinkedBlockingDeque @ 0x86cfb670 |
40 | 893,476,328 | 44.21%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0x87b09890|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0x928e0180|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xb2081670|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xb907d988|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xba6525e0|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xbd065878|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xbe074aa0|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xbed16790|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xc0a3aaa8|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xc174ab78|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xc34a3628|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xc417f950|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xc5122198|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xc609da10|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xc6d414a0|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xc6d414d0|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xc7a5ab60|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xca0dce98|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xca6780b0|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xd0e8ce98|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xd17a4530|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xd34d3308|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xd42ed4b0|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xd4bd1a68|
24 | 598,240 | 0.03%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0xd57ed438|
24 | 598,240 | 0.03%
'- Total: 25 of 35,432 entries; 35,407 more |
| |
---------------------------------------------------------------------------------------------------------
I found there are many GridDhtPartitionMap2 objects holding Integer objects
though a HashMap, each consume some 55,424 retained heap, here is one
example:
(the dominator tree with more nodes expanded
https://drive.google.com/open?id=0BwY2dxDlRYhBcUtNY01RN3M4YnM)
Class Name
| Shallow Heap | Retained Heap | Percentage
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| | |
java.util.concurrent.LinkedBlockingDeque @ 0x86cfb670
| 40 | 893,476,328 | 44.21%
|- java.util.concurrent.LinkedBlockingDeque$Node @ 0x87b09890
| 24 | 598,240 | 0.03%
| '-
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
@ 0x87b098a8 | 160 | 598,216 |
0.03%
| |- java.util.ArrayList @ 0x87b09d98
| 24 | 596,704 | 0.03%
| | '- java.lang.Object[366] @ 0x8adf7290
| 1,480 | 596,680 | 0.03%
| | |-
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$5
@ 0x87b09df8 | 24 | 586,896 | 0.03%
| | | '-
org.apache.ignite.internal.processors.cache.CacheAffinityChangeMessage @
0x87b09e10 | 40 |
586,872 | 0.03%
| | | |-
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsFullMessage
@ 0x87b09eb0 | 64 | 586,712 | 0.03%
| | | | |- java.util.HashMap @ 0x87b09f18
| 48 | 400,176 | 0.02%
| | | | | '- java.util.HashMap$Node[16] @ 0x87b09f48
| 80 | 400,128 | 0.02%
| | | | | |- java.util.HashMap$Node @ 0x87b09f98
| 32 | 111,088 | 0.01%
| | | | | | |-
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap
@ 0x87b09fc8 | 64 | 111,040 | 0.01%
| | | | | | | '- java.util.HashMap$Node[4] @
0x87b0a008
| 32 | 110,976 | 0.01%
| | | | | | | |- java.util.HashMap$Node @
0x87b0a028
| 32 | 55,488 | 0.00%
| | | | | | | | |-
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap2
@ 0x87b0a068| 40 | 55,424 | 0.00%
| | | | | | | | | |- java.util.HashMap @
0x87b0a0a8
| 48 | 55,360 | 0.00%
| | | | | | | | | | '-
java.util.HashMap$Node[2048] @ 0x87b0a528
| 8,208 | 55,312 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a0d8
| 32 | 48 | 0.00%
| | | | | | | | | | | '- java.lang.Integer
@ 0x87b0a0f8 1023 |
16 | 16 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a108
| 32 | 48 | 0.00%
| | | | | | | | | | | '- java.lang.Integer
@ 0x87b0a128 1022 |
16 | 16 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a138
| 32 | 48 | 0.00%
| | | | | | | | | | | '- java.lang.Integer
@ 0x87b0a158 1021 |
16 | 16 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a168
| 32 | 48 | 0.00%
| | | | | | | | | | | '- java.lang.Integer
@ 0x87b0a188 1020 |
16 | 16 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a198
| 32 | 48 | 0.00%
| | | | | | | | | | | '- java.lang.Integer
@ 0x87b0a1b8 1019 |
16 | 16 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a1c8
| 32 | 48 | 0.00%
| | | | | | | | | | | '- java.lang.Integer
@ 0x87b0a1e8 1018 |
16 | 16 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a1f8
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a228
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a258
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a288
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a2b8
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a2e8
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a318
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a348
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a378
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a3a8
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a3d8
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a408
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a438
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a468
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a498
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a4c8
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0a4f8
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0c538
| 32 | 48 | 0.00%
| | | | | | | | | | |-
java.util.HashMap$Node @ 0x87b0c568
| 32 | 48 | 0.00%
| | | | | | | | | | '- Total: 25 of 1,024
entries; 999 more |
| |
| | | | | | | | | |-
org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion @
0x87b0a090 | 24 | 24 | 0.00%
| | | | | | | | | '- Total: 2 entries
| | |
| | | | | | | | |- java.util.UUID @ 0x87b0a048
| 32 | 32 | 0.00%
| | | | | | | | '- Total: 2 entries
| | |
| | | | | | | |- java.util.HashMap$Node @
0x87b1c8b8
| 32 | 55,456 | 0.00%
| | | | | | | | '-
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap2
@ 0x87b1c8f8| 40 | 55,424 | 0.00%
| | | | | | | | |- java.util.HashMap @
0x87b1c938
| 48 | 55,360 | 0.00%
| | | | | | | | | '-
java.util.HashMap$Node[2048] @ 0x87b1d448
| 8,208 | 55,312 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1c968
| 32 | 48 | 0.00%
| | | | | | | | | | '- java.lang.Integer
@ 0x87b1c988 1023 |
16 | 16 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1c998
| 32 | 48 | 0.00%
| | | | | | | | | | '- java.lang.Integer
@ 0x87b1c9b8 1022 |
16 | 16 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1c9c8
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1c9f8
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1ca28
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1ca58
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1ca88
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cab8
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cae8
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cb18
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cb48
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cb78
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cba8
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cbd8
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cc08
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cc38
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cc68
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cc98
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1ccc8
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1ccf8
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cd28
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cd58
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cd88
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cdb8
| 32 | 48 | 0.00%
| | | | | | | | | |-
java.util.HashMap$Node @ 0x87b1cde8
| 32 | 48 | 0.00%
| | | | | | | | | '- Total: 25 of 1,024
entries; 999 more |
| |
| | | | | | | | |-
org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion @
0x87b1c920 | 24 | 24 | 0.00%
| | | | | | | | '- Total: 2 entries
| | |
| | | | | | | '- Total: 2 entries
| | |
| | | | | | |- java.lang.Integer @ 0x87b09fb8
-1933864410
| 16 | 16 | 0.00%
| | | | | | '- Total: 2 entries
| | |
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-GridDhtPartitionMap2-tp9504.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
