Hi Denis, I think I've found the issue. Ignite skips references to Ignite-Caches. Moving away from the Ignite-Cache, but using the Ignite-Marshaller caused the issue. So it isn't an issue Ignite can prevent. Removing those references did it.
Thanks for listening, Peter 2017-01-22 6:15 GMT+01:00 Denis Magda <[email protected]>: > Sorry, didn’t pay attention that the size of every object is 1 KB. Then I > would suggest to share your test/heap dump to see what happens. > > — > Denis > > On Jan 21, 2017, at 9:10 PM, Denis Magda <[email protected]> wrote: > > Hello Peter, > > This behavior is expected. Look, you serialize 2000 objects * 1000 KB > which gives ~ 2 GB in total. After that you hold the references to these > objects by putting them into an ordinary hash map. The objects will not be > removed from the heap until the map keeps them. > > One the other side, when the same objects are placed into an off-heap > cache they will be collected from the heap because at the end there will be > no reachable references to them. > > — > Denis > > On Jan 21, 2017, at 2:46 PM, Peter Schmitt <[email protected]> > wrote: > > Hello Ignite-Community! > > I would like to use the Marshaller from Ignite for de-/serialising objects. > I would like to re-use it, because it's very fast and I'm using Ignite > anyway. > > I tried it with: > > //every thread gets the Marshaller via: > GridKernalContext ctx = ((IgniteKernal) ignite).context(); > Marshaller marshaller = ctx.grid().configuration().getMarshaller(); > > -> > > byte serializedResult[] = marshaller.marshal(obj); > or > Object deserializedResult = marshaller.unmarshal(serializedResult, > ctx.getClass().getClassLoader()); > > In a simple test it worked very well (and fast). > However, once I'm using larger Objects (e.g. 2000 instances each ~1kb > after marshalling) and store the results in a map, the Heap consumption > grows a lot (almost 2gb in few seconds). > If I store the same objects in an Ignite OffHeap-Cache, the (OffHeap) > memory-consumption is minimal. > In the debugger I saw that the heap grows once those 2000 instances get > processed. > (A manually triggered GC can just release few mb.) > > I even tried to re-use the marshaller, but nothing changed. > At runtime BinaryMarshaller is used which wraps GridBinaryMarshaller. > > Any hint to get rid of the leak is appreciated, > Peter > > > >
