Ok, there are no exceptions in the ignite logs for the client JVMs but I've attached the log for one of the problem servers. Looks like a few errors but I am unable to determine the root cause. ignite-46073e05.zip <http://apache-ignite-users.70518.x6.nabble.com/file/t2658/ignite-46073e05.zip>
ilya.kasnacheev wrote > Hello! > > This is very strange, since we expect this collection to be cleared on > exchange. > > Please make sure you don't have any stray exceptions during exchange in > your logs. > > Regards, > -- > Ilya Kasnacheev > > > пт, 8 нояб. 2019 г. в 12:49, mvkarp < > liquid_ninja2k@ > >: > >> Hi, >> >> This is not the case. Always only a maximum total of two server nodes. >> One >> JVM server on each. However there are many client JVMs that start and >> stop >> caches with setClientMode=true. It looks like one of the server instances >> is >> immune to the issue, whilst the most newly created one gets the leak, >> with >> a >> lot of partition exchanges happening for EVT_NODE_JOINED and >> EVT_NODE_LEFT >> (one of the nodes don't get any of these partition exchanges, however the >> exact server node that gets this can alternate so its not linked to one >> node >> in particular but seems to be linked to the most newly launched server). >> >> >> ilya.kasnacheev wrote >> > Hello! >> > >> > How many nodes do you have in your cluster? >> > >> > From the dump it seems that the number of server nodes is in thousands. >> Is >> > this the case? >> > >> > Regards, >> > -- >> > Ilya Kasnacheev >> > >> > >> > пт, 8 нояб. 2019 г. в 10:26, mvkarp < >> >> > liquid_ninja2k@ >> >> > >: >> > >> >> Let me know if these help or if you need anything more specific. >> >> recoveryBallotBoxes.zip >> >> < >> >> >> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/recoveryBallotBoxes.zip >> > >> >> >> >> >> >> >> >> ilya.kasnacheev wrote >> >> > Hello! >> >> > >> >> > Can you please check whether there are any especially large objects >> >> inside >> >> > recoveryBallotBoxes object graph? Sorting by retained heap may help >> in >> >> > determining this. It would be nice to know what is the type >> histogram >> >> of >> >> > what's inside recoveryBallotBoxes and where the bulk of heap usage >> >> > resides. >> >> > >> >> > Regards, >> >> > -- >> >> > Ilya Kasnacheev >> >> > >> >> > >> >> > чт, 7 нояб. 2019 г. в 06:23, mvkarp < >> >> >> >> > liquid_ninja2k@ >> >> >> >> > >: >> >> > >> >> >> I've attached another set of screenshots, might be more clear. >> >> >> heap.zip >> >> >> < >> >> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/heap.zip> >> >> >> >> >> >> >> >> >> mvkarp wrote >> >> >> > I've attached some extra screenshots showing what is inside these >> >> >> records >> >> >> > and path to GC roots. heap.zip >> >> >> > < >> >> >> >> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/heap.zip> >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >> >> >> >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
