On Tue, Mar 15, 2016 at 2:29 PM, 'Bill Hart' via julia-users <[email protected]> wrote: > > > On Tuesday, 15 March 2016 19:02:41 UTC+1, Yichao Yu wrote: >> >> On Tue, Mar 15, 2016 at 1:45 PM, Yichao Yu <[email protected]> wrote: >> > >> > On Mar 15, 2016 11:56 AM, "'Bill Hart' via julia-users" >> > <[email protected]> wrote: >> >> >> >> We have been trying to understand the garbage collector behaviour, >> >> since >> >> we had some code for which our machine is running out of memory in a >> >> matter >> >> of an hour. >> >> >> >> We already realised that Julia isn't responsible for memory we allocate >> >> on >> >> the C side unless we use jl_gc_counted_malloc, which we now do >> >> everywhere. >> >> But it still uses masses of memory where we were roughly expecting no >> >> growth >> >> in memory usage (lots of short-lived objects and nothing much else). >> >> >> >> The behaviour of the gc on my machine seems to be to allocate objects >> >> until 23mb of memory is allocated, then do a jl_gc_collect. However, >> >> even >> >> after reading as much of the GC code in C as I can, I still can't >> >> determine >> >> why we are observing the behaviour we are seeing. >> >> >> >> Here is a concrete example whose behaviour I don't understand: >> >> >> >> function doit2(n::Int) >> >> s = >> >> BigInt(2234567876543456789876545678987654567898765456789876545678) >> >> for i = 1:n >> >> s += i >> >> end >> >> return s >> >> end >> >> >> >> doit(10000000000) >> >> Another note is that adding finalizers will (currently) extend the >> lifetime of an object. https://github.com/JuliaLang/julia/pull/13995 >> should solve this problem but I'm holding on to it before we finish >> some other GC rework. > > > This at least sounds like it might be related to the problem we are seeing. > > I guess it means an object with a finalizer cannot be immediately cleaned up > if it is short lived. The only way for such an object to get cleaned up is > in long lived territory, which the state diagram seems to suggest is the > only place a finalizer can even be scheduled. Is that about correct (I have
I don't think the state diagram covers finalizer. The issue with the finalizer is that they are arbitrary julia code and cannot be run in the GC. They need to run after the GC finishes and therefore must survive the sweep. This will make them survive one more GC than what they would have been and make them likely to end up in the old gen. (If most of your temporaries don't survive more than a GC this shouldn't be an issue.) > to explain it to someone else, so I'm hoping to not tell them too many wrong > things). > > Bill. One way to get an idea of what's causing the memory issue is to call `gc(true)` or `gc(false)` from time to time (you can call it multiple times in a row but don't call it in every iteration, it'll be terribly slow).
