Hello! This may be due to some rogue process connecting to Discovery port and writing unexpected input to it.
Regards, -- Ilya Kasnacheev ср, 29 янв. 2020 г. в 05:47, Mahesh Renduchintala < [email protected]>: > Hi, > > We have Ignite data nodes reporting out of memory after many days. > What could be possible reasons? > > > [16:45:04,018][SEVERE][tcp-disco-client-message-worker-#2404][TcpDiscoverySpi] > Runtime error caught during grid runnable execution: GridWorker > [name=tcp-disco-client-message-worker, igniteInstanceName=null, > finished=false, heartbeatTs=1580229848371, hashCode=32546845, > interrupted=true, runner=tcp-disco-client-message-worker-#2404] > *java.lang.OutOfMemoryError: Java heap space* > [16:45:04,018][SEVERE][tcp-disco-client-message-worker-#2404][TcpDiscoverySpi] > Runtime error caught during grid runnable execution: IgniteSpiThread > [name=tcp-disco-client-message-worker-#2404] > java.lang.OutOfMemoryError: Java heap space > > However, the heap memory looks pretty good. > > ^-- Node [id=84a46677, uptime=11 days, 14:01:34.176] > ^-- H/N/C [hosts=5, nodes=5, CPUs=31] > ^-- CPU [cur=100%, avg=5.59%, GC=154.43%] > ^-- PageMemory [pages=1696651] > ^-*- Heap [used=12800MB, free=50%, comm=25600MB]* > ^-- Off-heap [used=6705MB, free=79.72%, comm=33068MB] > ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB] > ^-- default region [used=6704MB, free=79.54%, comm=32768MB] > ^-- metastoreMemPlc region [used=0MB, free=99.45%, comm=100MB] > ^-- TxLog region [used=0MB, free=100%, comm=100MB] > ^-- Ignite persistence [used=22613MB] > ^-- sysMemPlc region [used=0MB] > ^-- default region [used=22613MB] > ^-- metastoreMemPlc region [used=unknown] > ^-- TxLog region [used=0MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=2, qSize=0] > ^-- System thread pool [active=1, idle=3, qSize=0] > > > Mahesh > Aline Consulting > > >
