Hi,
Ignite version : 1.8
One of the ignite node in 3node cluster is down due to full usage of RAM.
At that point of time i can observe the following logs on this node:
[00:32:02,119][INFO
][grid-timeout-worker-#7%CasinoApacheIgniteServices%][IgniteKernal%CasinoApacheIgniteServices]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9f8df386, name=CasinoApacheIgniteServices,
uptime=23:21:45:744]
^-- H/N/C [hosts=8, nodes=8, CPUs=44]
^-- CPU [cur=8.33%, avg=1.6%, GC=0%]
^-- Heap [used=3886MB, free=36.65%, comm=6134MB]
^-- Non heap [used=78MB, free=85.96%, comm=529MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[00:33:24,674][WARN
][exchange-worker-#23%CasinoApacheIgniteServices%][GridCachePartitionExchangeManager]
Failed to wait for partition map exchange [topVer=AffinityTopologyVersion
[topVer=84, minorTopVer=0], node=9f8df386-2886-451f-b1ff-53713878d432].
Dumping pending objects that might be the cause:
[00:33:24,674][WARN
][exchange-worker-#23%CasinoApacheIgniteServices%][GridCachePartitionExchangeManager]
Failed to wait for partition map exchange [topVer=AffinityTopologyVersion
[topVer=84, minorTopVer=0], node=9f8df386-2886-451f-b1ff-53713878d432].
Dumping pending objects that might be the cause:
SAR stats for memory usage on this date:
-- mar 6
12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit
%commit kbactive kbinact kbdirty
12:10:01 PM 170120 16090232 98.95 0 3393384 8222696
45.02 9887268 2088504 60
01:50:01 PM 168176 16092176 98.97 0 2120848 8224724
45.03 10804712 1596792 48
03:10:01 PM 199128 16061224 98.78 0 991832 8224904
45.04 11384652 1241284 436
04:10:01 PM 153060 16107292 99.06 0 229984 8224880
45.04 11255628 1627600 208
04:20:01 PM 165580 16094772 98.98 0 78572 8224828
45.03 11338592 1560944 52
04:30:01 PM 153508 16106844 99.06 0 29740 8224872
45.03 11436544 1579468 44
04:40:01 PM 162184 16098168 99.00 0 33152 8224892
45.04 11606584 1580388 24
11:10:01 PM 370956 15889396 97.72 0 74816 8225312
45.04 11927676 1610828 36
11:20:01 PM 348576 15911776 97.86 0 69012 8225272
45.04 11929820 1602748 48
11:30:01 PM 359132 15901220 97.79 0 27060 8225308
45.04 11912656 1577848 36
11:40:01 PM 340252 15920100 97.91 0 24908 8225272
45.04 11910516 1577668 32
11:50:01 PM 308340 15952012 98.10 0 39208 8242284
45.13 11914564 1589208 48
Average: 253568 16006784 98.44 0 2317289 8226063
45.04 10368276 1955525 142
Please find the attached file for the cache configuration.
ignite-clb-cache-config_dev.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t1753/ignite-clb-cache-config_dev.xml>
Please find the memory snapshot which is captured by app dynamics tool in
the attachment.
memorySnapshot.JPG
<http://apache-ignite-users.70518.x6.nabble.com/file/t1753/memorySnapshot.JPG>
Following is my analysis.
When the data is evicting from on heap to off heap, there is not much space
in off heap.
Due to that off heap memory usage is full and application has become slow
and unresponsive.
Even the data in off heap is not expired because of that there is not much
free memory in RAM.
After i restarted the application on this node, the RAM usage has become to
25% and now it's usage is 45%.
can you please check and suggest once.
Thanks,
Praveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
