We have 2 machines (M1, M2)
In M1 running a Ignite server (I1) and a Tomcat server (T1), which hosted 2
Ignite clients (C1a, C1b),
similarly in M2 running a Ignite server (I2) and a Tomcat server (T2), which
hosted 2 Ignite clients (C2a, C2b).
OutOfMemoryError were found in both T1 and T2 yesterday, we have a heap dump
from T1, but failed to get a heap dump from T2.
After the error, by using visor we found only T1, C1a, and C1b were still in
the cluster, all Ignite nodes on M2 were disconnected.
We've lost the Tomcat server log due to some reason.
>From the heap dump of T1 we found 6
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor holding
total 1.2GB memory:
Class Name
| Shallow Heap | Retained Heap | Percentage
------------------------------------------------------------------------------------------------------------------------------------------
| | |
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8 | 56 | 227,306,848 | 14.05%
'- java.util.HashMap @ 0x8423c790
| 48 | 227,306,792 | 14.05%
'- java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744 | 14.05%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x8449bed0 | 56 | 227,299,144 | 14.05%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81ae5c10 | 56 | 227,071,080 | 14.04%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x8449f158 | 56 | 227,063,384 | 14.04%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81adfa38 | 56 | 226,599,544 | 14.01%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x8449f090 | 56 | 226,591,864 | 14.01%
org.apache.catalina.loader.WebappClassLoader @ 0x80167cd8
| 136 | 29,521,360 | 1.83%
org.apache.catalina.loader.WebappClassLoader @ 0x8363a078
| 136 | 28,471,928 | 1.76%
com.cityline.cps.admin.api.controller.LoginController @ 0x819c54d0
| 48 | 21,042,952 | 1.30%
org.apache.ignite.spi.discovery.tcp.ClientImpl @ 0x84477b20
| 96 | 14,456,360 | 0.89%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0xade21488 | 152 | 10,462,696 | 0.65%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0xae8ec420 | 152 | 9,800,264 | 0.61%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x86aa66a0 | 152 | 9,677,848 | 0.60%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x86ac6b58 | 152 | 9,634,288 | 0.60%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x83013720 | 152 | 6,075,032 | 0.38%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x86008f08 | 152 | 5,967,304 | 0.37%
class java.beans.ThreadGroupContext @ 0x81070c30 System Class
| 8 | 2,634,888 | 0.16%
org.hibernate.internal.SessionFactoryImpl @ 0x811ec9f0
| 136 | 2,406,176 | 0.15%
org.hibernate.internal.SessionFactoryImpl @ 0x85de4c98
| 136 | 2,082,128 | 0.13%
org.hibernate.internal.SessionFactoryImpl @ 0x83b30890
| 136 | 2,036,976 | 0.13%
org.apache.ignite.internal.GridKernalContextImpl @ 0x841d4090
| 248 | 1,599,680 | 0.10%
org.apache.ignite.internal.GridKernalContextImpl @ 0x818f6900
| 248 | 1,596,328 | 0.10%
java.net.URLClassLoader @ 0x800fe890
| 80 | 1,451,640 | 0.09%
org.apache.catalina.webresources.JarResourceSet @ 0x8019d4d0
| 72 | 1,302,400 | 0.08%
org.apache.ignite.internal.processors.cache.GridCacheSharedContext @
0x8427bbc0 | 72 | 1,242,936 | 0.08%
org.apache.ignite.internal.processors.cache.GridCacheSharedContext @
0x819c5e18 | 72 | 1,239,464 | 0.08%
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager
@ 0x819d54e8 | 88 | 1,052,272 | 0.07%
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager
@ 0x842cc828 | 88 | 1,052,272 | 0.07%
org.apache.tomcat.util.net.NioEndpoint @ 0x80780e38
| 288 | 1,052,136 | 0.07%
org.apache.tomcat.util.net.NioEndpoint @ 0x80666fb0
| 288 | 1,051,584 | 0.07%
org.springframework.beans.factory.support.DefaultListableBeanFactory @
0x80fe9e50 | 208 | 882,424 | 0.05%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x82fb0d28 | 152 | 847,280 | 0.05%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x85fa7cd0 | 152 | 834,120 | 0.05%
org.springframework.beans.factory.support.DefaultListableBeanFactory @
0x857e33d0 | 208 | 725,632 | 0.04%
org.springframework.beans.factory.support.DefaultListableBeanFactory @
0x83b2fbb8 | 208 | 695,544 | 0.04%
org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource
@ 0x81367d80| 24 | 583,376 | 0.04%
Total: 36 of 252,650 entries; 252,614 more
| | |
------------------------------------------------------------------------------------------------------------------------------------------
drilling into one of DynamicCacheDescriptor, found the memory was held by a
HashMap (java.util.HashMap @ 0x8423c790, java.util.HashMap$Node[65536] @
0xcd657f20), which seems to contain a bunch of CacheConfiguration objects:
Class Name
| Shallow Heap | Retained Heap | Percentage
-----------------------------------------------------------------------------------------------------------------------------------------------------------
| | |
java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744 | 14.05%
|- java.util.HashMap$Node @ 0xc559b860
| 32 | 38,520 | 0.00%
| |- java.util.HashMap$Node @ 0xcad62418
| 32 | 30,816 | 0.00%
| | |- java.util.HashMap$Node @ 0xcb6d56b8
| 32 | 23,112 | 0.00%
| | | |- java.util.HashMap$Node @ 0xce1be088
| 32 | 15,408 | 0.00%
| | | | |- java.util.HashMap$Node @ 0xd4e3dbe8
| 32 | 7,704 | 0.00%
| | | | | '- org.apache.ignite.configuration.CacheConfiguration @
0xd4e3be10 | 288 | 7,672 |
0.00%
| | | | | |-
org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper
@ 0xd4e3bfe0| 24 | 7,096 | 0.00%
| | | | | |- java.lang.String @ 0xd4e38260
ignite-marshaller-sys-cache | 24 |
96 | 0.00%
| | | | | |-
org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction @
0xd4e3bf90 | 40 | 80 | 0.00%
| | | | | |- java.util.HashSet @ 0xd4e3bf50
| 16 | 64 | 0.00%
| | | | | |-
javax.cache.configuration.FactoryBuilder$SingletonFactory @ 0xd4e3bf30
| 16 | 32 | 0.00%
| | | | | |-
org.apache.ignite.plugin.CachePluginConfiguration[0] @ 0xd4e3db98
| 16 | 16 | 0.00%
| | | | | '- Total: 6 entries
| | |
| | | | |- org.apache.ignite.configuration.CacheConfiguration @
0xce1be0a8 | 288 | 7,672 |
0.00%
| | | | | |-
org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper
@ 0xce1be2e8 | 24 | 7,096 | 0.00%
| | | | | |- java.lang.String @ 0xce1be228
ignite-marshaller-sys-cache | 24
| 96 | 0.00%
| | | | | |-
org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction @
0xce1be298 | 40 | 80 | 0.00%
| | | | | |- java.util.HashSet @ 0xce1be1c8
| 16 | 64 | 0.00%
| | | | | |-
javax.cache.configuration.FactoryBuilder$SingletonFactory @ 0xce1be208
| 16 | 32 | 0.00%
| | | | | |- org.apache.ignite.plugin.CachePluginConfiguration[0] @
0xce1c2098 | 16 | 16 |
0.00%
| | | | | '- Total: 6 entries
| | |
| | | | '- Total: 2 entries
| | |
| | | |- org.apache.ignite.configuration.CacheConfiguration @
0xcb6d56d8 | 288 | 7,672
| 0.00%
| | | '- Total: 2 entries
| | |
| | |- org.apache.ignite.configuration.CacheConfiguration @ 0xcad62438
| 288 | 7,672 | 0.00%
| | '- Total: 2 entries
| | |
| |- org.apache.ignite.configuration.CacheConfiguration @ 0xc559b880
| 288 | 7,672 | 0.00%
| '- Total: 2 entries
| | |
|- java.util.HashMap$Node @ 0x8e45adf0
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x983d3e70
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x87b1c1c8
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x8a0d6980
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x97143f50
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x9731b858
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x906efbd8
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x9735be90
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x8ebdf958
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x951d6be0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x9061a5a8
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x97b80fa0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x877bad58
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x873bb2d0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x8d47e168
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x89e19d58
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x8711c5f0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x980292b0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0xb25e7ac8
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x9d3ddbe0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x948dd4d0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x8d7eb430
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0xc60fa120
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x9cd18138
| 32 | 30,816 | 0.00%
'- Total: 25 of 23,850 entries; 23,825 more
| | |
-----------------------------------------------------------------------------------------------------------------------------------------------------------
and this is the shortest paths to the accumulation point for
java.util.HashMap @ 0x8423c790
Class Name
| Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| |
java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744
'- table java.util.HashMap @ 0x8423c790
| 48 | 227,306,792
'- rmtCfgs
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8
| 56 | 227,306,848
'- value java.util.HashMap$Node @ 0xe066c018
| 32 | 32
'- [11] java.util.HashMap$Node[16] @ 0xe066bf48
| 80 | 272
'- table java.util.HashMap @ 0xe066bf08
| 48 | 320
'- cachesOnDisconnect
org.apache.ignite.internal.processors.cache.GridCacheProcessor @ 0x81bacb08
| 80 | 2,640
'- cacheProc
org.apache.ignite.internal.GridKernalContextImpl @ 0x818f6900
| 248 | 1,596,328
|- ctx org.apache.ignite.internal.IgniteKernal @
0x818f68a0
| 96 | 200
| |- ignite
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi @ 0x819c19c0
| 240 | 2,000
| | |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker
@ 0x81b97128 tcp-comm-worker-#1%null% Thread| 144 | 2,640
| | |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$3 @ 0x81b97098
| 16 | 16
| | '- Total: 2 entries
| |
| |- ignite
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi @ 0x81968500
| 256 | 104,176
| |- grid
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance @ 0x81967d28
| 88 | 344
| '- Total: 3 entries
| |
|- ctx
org.apache.ignite.internal.processors.clock.GridClockServer @ 0x819680e0
| 32 | 32
|- ctx
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager @
0x81968468 |
128 | 45,944
|- ctx
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor @
0x81b97c98 |
32 | 512
|- ctx
org.apache.ignite.internal.processors.rest.GridRestProcessor @ 0x81bad1a0
| 64 | 1,056
|- ctx
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor @
0x83009a50 | 48
| 2,008
|- ctx
org.apache.ignite.internal.managers.deployment.GridDeploymentManager @
0x818f69f8 |
48 | 2,496
'- Total: 7 entries
| |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
