Hello! This is very strange, since we expect this collection to be cleared on exchange.
Please make sure you don't have any stray exceptions during exchange in your logs. Regards, -- Ilya Kasnacheev пт, 8 нояб. 2019 г. в 12:49, mvkarp <liquid_ninj...@hotmail.com>: > Hi, > > This is not the case. Always only a maximum total of two server nodes. One > JVM server on each. However there are many client JVMs that start and stop > caches with setClientMode=true. It looks like one of the server instances > is > immune to the issue, whilst the most newly created one gets the leak, with > a > lot of partition exchanges happening for EVT_NODE_JOINED and EVT_NODE_LEFT > (one of the nodes don't get any of these partition exchanges, however the > exact server node that gets this can alternate so its not linked to one > node > in particular but seems to be linked to the most newly launched server). > > > ilya.kasnacheev wrote > > Hello! > > > > How many nodes do you have in your cluster? > > > > From the dump it seems that the number of server nodes is in thousands. > Is > > this the case? > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > пт, 8 нояб. 2019 г. в 10:26, mvkarp < > > > liquid_ninja2k@ > > > >: > > > >> Let me know if these help or if you need anything more specific. > >> recoveryBallotBoxes.zip > >> < > >> > http://apache-ignite-users.70518.x6.nabble.com/file/t2658/recoveryBallotBoxes.zip > > > >> > >> > >> > >> ilya.kasnacheev wrote > >> > Hello! > >> > > >> > Can you please check whether there are any especially large objects > >> inside > >> > recoveryBallotBoxes object graph? Sorting by retained heap may help in > >> > determining this. It would be nice to know what is the type histogram > >> of > >> > what's inside recoveryBallotBoxes and where the bulk of heap usage > >> > resides. > >> > > >> > Regards, > >> > -- > >> > Ilya Kasnacheev > >> > > >> > > >> > чт, 7 нояб. 2019 г. в 06:23, mvkarp < > >> > >> > liquid_ninja2k@ > >> > >> > >: > >> > > >> >> I've attached another set of screenshots, might be more clear. > >> >> heap.zip > >> >> < > >> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/heap.zip> > >> >> > >> >> > >> >> mvkarp wrote > >> >> > I've attached some extra screenshots showing what is inside these > >> >> records > >> >> > and path to GC roots. heap.zip > >> >> > < > >> >> > http://apache-ignite-users.70518.x6.nabble.com/file/t2658/heap.zip> > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> -- > >> >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > >> >> > >> > >> > >> > >> > >> > >> -- > >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > >> > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
