Manesh, Start all the nodes with the following parameters: https://apacheignite.readme.io/docs/jvm-and-system-tuning#section-getting-heap-dump-on-out-of-memory-errors
JVM will create a heap dump on failure and you'll be able to see the root cause of the leak. If that's in Ignite then please file a BLOCKER ticket. - Denis On Tue, Jul 30, 2019 at 7:45 AM Mahesh Renduchintala < [email protected]> wrote: > Infact, in the logs you can see that whenever the below print comes up, > memory jumps up by 100-200MB > > >>Full map updating for 873 groups performed in 16 ms > > > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=4c8b23b4, uptime=00:19:13.959] > ^-- H/N/C [hosts=8, nodes=10, CPUs=48] > ^-- CPU [cur=8%, avg=7.55%, GC=0%] > ^-- PageMemory [pages=0] > * ^-- Heap [used=1307MB, free=35.9%, comm=2039MB]* > ^-- Off-heap [used=0MB, free=-1%, comm=0MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=4, qSize=0] > ^-- System thread pool [active=0, idle=2, qSize=0] > 2019-07-30 14:11:48.485 INFO 26 --- [ sys-#167] > .c.d.d.p.GridDhtPartitionsExchangeFuture : Received full message, will > finish exchange [node=9e2951dc-e8ad-44e6-9495-83b0e5337511, > resVer=AffinityTopologyVersion [topVer=1361, minorTopVer=0]] > 2019-07-30 14:11:48.485 INFO 26 --- [ sys-#167] > .c.d.d.p.GridDhtPartitionsExchangeFuture : Received full message, need > merge [curFut=AffinityTopologyVersion [topVer=1357, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=1361, minorTopVer=0]] > 2019-07-30 14:11:48.485 INFO 26 --- [ sys-#167] > .i.p.c.GridCachePartitionExchangeManager : Merge exchange future on finish > [curFut=AffinityTopologyVersion [topVer=1357, minorTopVer=0], > mergedFut=AffinityTopologyVersion [topVer=1358, minorTopVer=0], > evt=NODE_JOINED, evtNode=864571bd-7235-4fe0-9e52-f3a78f35dbb2, > evtNodeClient=false] > 2019-07-30 14:11:48.485 INFO 26 --- [ sys-#167] > .i.p.c.GridCachePartitionExchangeManager : Merge exchange future on finish > [curFut=AffinityTopologyVersion [topVer=1357, minorTopVer=0], > mergedFut=AffinityTopologyVersion [topVer=1359, minorTopVer=0], > evt=NODE_FAILED, evtNode=20eef25d-b7ec-4340-9da8-1a5a35678ba5, > evtNodeClient=false] > 2019-07-30 14:11:48.485 INFO 26 --- [ sys-#167] > .i.p.c.GridCachePartitionExchangeManager : Merge exchange future on finish > [curFut=AffinityTopologyVersion [topVer=1357, minorTopVer=0], > mergedFut=AffinityTopologyVersion [topVer=1360, minorTopVer=0], > evt=NODE_JOINED, evtNode=9c318eb2-dd21-457c-8d1f-e6d4677e1a55, > evtNodeClient=true] > 2019-07-30 14:11:48.486 INFO 26 --- [ sys-#167] > .i.p.c.GridCachePartitionExchangeManager : Merge exchange future on finish > [curFut=AffinityTopologyVersion [topVer=1357, minorTopVer=0], > mergedFut=AffinityTopologyVersion [topVer=1361, minorTopVer=0], > evt=NODE_FAILED, evtNode=864571bd-7235-4fe0-9e52-f3a78f35dbb2, > evtNodeClient=false] > 2019-07-30 14:11:48.861 INFO 26 --- [ sys-#167] > o.a.i.i.p.c.CacheAffinitySharedManager : Affinity applying from full > message performed in 375 ms. > 2019-07-30 14:11:48.864 INFO 26 --- [ sys-#167] > .c.d.d.p.GridDhtPartitionsExchangeFuture : Affinity changes applied in 379 > ms. > 2019-07-30 14:11:48.880 INFO 26 --- [ sys-#167] > .c.d.d.p.GridDhtPartitionsExchangeFuture : Full map updating for 873 groups > performed in 16 ms. > 2019-07-30 14:11:48.880 INFO 26 --- [ sys-#167] > .c.d.d.p.GridDhtPartitionsExchangeFuture : Finish exchange future > [startVer=AffinityTopologyVersion [topVer=1357, minorTopVer=0], > resVer=AffinityTopologyVersion [topVer=1361, minorTopVer=0], err=null] > 2019-07-30 14:11:48.927 INFO 26 --- [ sys-#167] > .c.d.d.p.GridDhtPartitionsExchangeFuture : Detecting lost partitions > performed in 47 ms. > 2019-07-30 14:11:49.280 INFO 26 --- [ sys-#167] > .c.d.d.p.GridDhtPartitionsExchangeFuture : Completed partition exchange > [localNode=4c8b23b4-ce12-4dbb-a7ea-9279711f4008, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=1357, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode > [id=20eef25d-b7ec-4340-9da8-1a5a35678ba5, addrs=[0:0:0:0:0:0:0:1%lo, > 127.0.0.1, 192.168.1.139, 192.168.1.181], sockAddrs=[/192.168.1.181:47500, > /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /192.168.1.139:47500], > discPort=47500, order=1357, intOrder=696, lastExchangeTime=1564495322589, > loc=false, ver=2.7.0#20181130-sha1:256ae401, isClient=false], done=true], > topVer=AffinityTopologyVersion [topVer=1361, minorTopVer=0], > durationFromInit=4411] > 2019-07-30 14:11:49.289 INFO 26 --- [ange-worker-#43] > .i.p.c.GridCachePartitionExchangeManager : Skipping rebalancing (no > affinity changes) [top=AffinityTopologyVersion [topVer=1361, > minorTopVer=0], rebTopVer=AffinityTopologyVersion [topVer=-1, > minorTopVer=0], evt=NODE_JOINED, > evtNode=20eef25d-b7ec-4340-9da8-1a5a35678ba5, client=true] > 2019-07-30 14:11:50.127 INFO 26 --- [eout-worker-#23] > org.apache.ignite.internal.IgniteKernal : > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=4c8b23b4, uptime=00:19:16.964] > ^-- H/N/C [hosts=8, nodes=10, CPUs=48] > ^-- CPU [cur=50.33%, avg=7.59%, GC=22.8%] > ^-- PageMemory [pages=0] > * ^-- Heap [used=1537MB, free=24.64%, comm=2039MB]* > ^-- Off-heap [used=0MB, free=-1%, comm=0MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=4, qSize=0] > >
