Issue created : https://github.com/JuliaLang/julia/issues/8912
On Thu, Nov 6, 2014 at 7:56 AM, Seth Yuan <[email protected]> wrote: > I don't think it's an OSX only issue, as I have duplicated the issue on a > Centos too. > > Perhaps you guys can run the demo code I gave, test it on your own > machines to better appreciate the problem. > > Here a modified code without module dependency. > > function test_allocation() > a = drand(10000, 10000) > b = map(x->x+one(x), a) > readline() > end > > test_allocation() > @everywhere gc() > @everywhere Base.flush_gc_msgs() > @everywhere gc() > readline() > > > On Thursday, November 6, 2014 10:05:21 AM UTC+8, Elliot Saba wrote: >> >> @Stefan, they do, but we've had a longstanding issue >> <https://github.com/JuliaLang/julia/issues/1339> regarding whether we're >> releasing it as aggressively as we might need to in order to really shrink >> the number of pages we've got allocated. >> >> In short; calling gc() may not actually reduce the "Memory" column on >> OSX, however once your process starts to page to disk, OSX seems to come to >> its wits and start unmapping pages. >> -E >> >> On Wed, Nov 5, 2014 at 5:58 PM, Seth Yuan <[email protected]> wrote: >> >>> I tried this as you recommended, sadly, it doesn't do better. :( >>> >>> On Wednesday, November 5, 2014 4:48:10 PM UTC+8, Amit Murthy wrote: >>>> >>>> Could you try: >>>> >>>> @everywhere gc() >>>> @everywhere Base.flush_gc_msgs() >>>> @everywhere gc() >>>> >>>> >>>> >>>> On Wed, Nov 5, 2014 at 2:07 PM, Seth Yuan <[email protected]> wrote: >>>> >>>>> Hi Amit, >>>>> >>>>> Thanks for the response, I tried Base.flush_gc_msgs(), the result is >>>>> shown below: >>>>> >>>>> >>>>> <https://lh5.googleusercontent.com/-bEVLUYu-DPk/VFnhFUh28CI/AAAAAAAABk0/WNl3ukSetUk/s1600/mem.png> >>>>> It did free some memory on some workers, but not quite. I think the >>>>> acceptable number for workers would be around 100MB (like the master is). >>>>> >>>>> So what's wrong with DArrays? >>>>> >>>>> >>>>> On Wednesday, November 5, 2014 2:32:42 PM UTC+8, Amit Murthy wrote: >>>>>> >>>>>> Distributed objects are currently gc'ed asynchronously. >>>>>> >>>>>> You may force this using an internal function flush_gc_msgs like this: >>>>>> >>>>>> @everywhere Base.flush_gc_msgs() >>>>>> @everywhere gc() >>>>>> >>>>>> >>>>>> The above is an internal function and is not meant for general use. >>>>>> It will be great if you could open an issue on github requesting a >>>>>> feature >>>>>> to force gc of distributed objects. >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Nov 5, 2014 at 9:01 AM, Seth Yuan <[email protected]> wrote: >>>>>> >>>>>>> I did a test on deallocations, and the behavior is not what I >>>>>>> expected. >>>>>>> >>>>>>> The code I ran is: >>>>>>> >>>>>>> @everywhere using Ops >>>>>>> >>>>>>> function test_allocation() >>>>>>> a = drand(10000, 10000) >>>>>>> b = 2a >>>>>>> readline() >>>>>>> end >>>>>>> >>>>>>> test_allocation() >>>>>>> @everywhere gc() >>>>>>> readline() >>>>>>> >>>>>>> Ops module defined the '*' operator (where a new DArray is created). >>>>>>> I can see that master's memory went down after the gc() call, but the >>>>>>> workers' memory remained unchanged. >>>>>>> >>>>>>> >>>>>>> <https://lh5.googleusercontent.com/-qo7SKwkoya0/VFmZrIWL0EI/AAAAAAAABkk/FctwkEcQWnk/s1600/mem.png> >>>>>>> So the question is: "how to really free a DArray? Both master and >>>>>>> workers!" >>>>>>> >>>>>>> >>>>>>> On Thursday, October 16, 2014 10:18:42 PM UTC+8, Amit Murthy wrote: >>>>>>>> >>>>>>>> meant to say "all references from all processes are removed" >>>>>>>> >>>>>>>> On Thu, Oct 16, 2014 at 7:47 PM, Amit Murthy <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes, it is gc'ed just like any other object, when all references >>>>>>>>> to it from any process is removed. >>>>>>>>> >>>>>>>>> On Thu, Oct 16, 2014 at 2:59 PM, Seth Yuan <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Now, I know by reading the documentation how a DArray is created, >>>>>>>>>> but I'm not sure when a DArray is deallocated. >>>>>>>>>> >>>>>>>>>> Is it GC able when the master looses all references of it? Does >>>>>>>>>> somebody know the answer to this question? >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >>
