Hi, This is not the case. Always only a maximum total of two server nodes. One JVM server on each. However there are many client JVMs that start and stop caches with setClientMode=true. It looks like one of the server instances is immune to the issue, whilst the most newly created one gets the leak, with a lot of partition exchanges happening for EVT_NODE_JOINED and EVT_NODE_LEFT (one of the nodes don't get any of these partition exchanges, however the exact server node that gets this can alternate so its not linked to one node in particular but seems to be linked to the most newly launched server).
ilya.kasnacheev wrote > Hello! > > How many nodes do you have in your cluster? > > From the dump it seems that the number of server nodes is in thousands. Is > this the case? > > Regards, > -- > Ilya Kasnacheev > > > пт, 8 нояб. 2019 г. в 10:26, mvkarp < > liquid_ninja2k@ > >: > >> Let me know if these help or if you need anything more specific. >> recoveryBallotBoxes.zip >> < >> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/recoveryBallotBoxes.zip> >> >> >> >> ilya.kasnacheev wrote >> > Hello! >> > >> > Can you please check whether there are any especially large objects >> inside >> > recoveryBallotBoxes object graph? Sorting by retained heap may help in >> > determining this. It would be nice to know what is the type histogram >> of >> > what's inside recoveryBallotBoxes and where the bulk of heap usage >> > resides. >> > >> > Regards, >> > -- >> > Ilya Kasnacheev >> > >> > >> > чт, 7 нояб. 2019 г. в 06:23, mvkarp < >> >> > liquid_ninja2k@ >> >> > >: >> > >> >> I've attached another set of screenshots, might be more clear. >> >> heap.zip >> >> < >> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/heap.zip> >> >> >> >> >> >> mvkarp wrote >> >> > I've attached some extra screenshots showing what is inside these >> >> records >> >> > and path to GC roots. heap.zip >> >> > < >> >> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/heap.zip> >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >> >> >> >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
