I don't think it's an OSX only issue, as I have duplicated the issue on a Centos too.
Perhaps you guys can run the demo code I gave, test it on your own machines to better appreciate the problem. Here a modified code without module dependency. function test_allocation() a = drand(10000, 10000) b = map(x->x+one(x), a) readline() end test_allocation() @everywhere gc() @everywhere Base.flush_gc_msgs() @everywhere gc() readline() On Thursday, November 6, 2014 10:05:21 AM UTC+8, Elliot Saba wrote: > > @Stefan, they do, but we've had a longstanding issue > <https://github.com/JuliaLang/julia/issues/1339> regarding whether we're > releasing it as aggressively as we might need to in order to really shrink > the number of pages we've got allocated. > > In short; calling gc() may not actually reduce the "Memory" column on OSX, > however once your process starts to page to disk, OSX seems to come to its > wits and start unmapping pages. > -E > > On Wed, Nov 5, 2014 at 5:58 PM, Seth Yuan <[email protected] <javascript:> > > wrote: > >> I tried this as you recommended, sadly, it doesn't do better. :( >> >> On Wednesday, November 5, 2014 4:48:10 PM UTC+8, Amit Murthy wrote: >>> >>> Could you try: >>> >>> @everywhere gc() >>> @everywhere Base.flush_gc_msgs() >>> @everywhere gc() >>> >>> >>> >>> On Wed, Nov 5, 2014 at 2:07 PM, Seth Yuan <[email protected]> wrote: >>> >>>> Hi Amit, >>>> >>>> Thanks for the response, I tried Base.flush_gc_msgs(), the result is >>>> shown below: >>>> >>>> >>>> <https://lh5.googleusercontent.com/-bEVLUYu-DPk/VFnhFUh28CI/AAAAAAAABk0/WNl3ukSetUk/s1600/mem.png> >>>> It did free some memory on some workers, but not quite. I think the >>>> acceptable number for workers would be around 100MB (like the master is). >>>> >>>> So what's wrong with DArrays? >>>> >>>> >>>> On Wednesday, November 5, 2014 2:32:42 PM UTC+8, Amit Murthy wrote: >>>>> >>>>> Distributed objects are currently gc'ed asynchronously. >>>>> >>>>> You may force this using an internal function flush_gc_msgs like this: >>>>> >>>>> @everywhere Base.flush_gc_msgs() >>>>> @everywhere gc() >>>>> >>>>> >>>>> The above is an internal function and is not meant for general use. It >>>>> will be great if you could open an issue on github requesting a feature >>>>> to >>>>> force gc of distributed objects. >>>>> >>>>> >>>>> >>>>> On Wed, Nov 5, 2014 at 9:01 AM, Seth Yuan <[email protected]> wrote: >>>>> >>>>>> I did a test on deallocations, and the behavior is not what I >>>>>> expected. >>>>>> >>>>>> The code I ran is: >>>>>> >>>>>> @everywhere using Ops >>>>>> >>>>>> function test_allocation() >>>>>> a = drand(10000, 10000) >>>>>> b = 2a >>>>>> readline() >>>>>> end >>>>>> >>>>>> test_allocation() >>>>>> @everywhere gc() >>>>>> readline() >>>>>> >>>>>> Ops module defined the '*' operator (where a new DArray is created). >>>>>> I can see that master's memory went down after the gc() call, but the >>>>>> workers' memory remained unchanged. >>>>>> >>>>>> >>>>>> <https://lh5.googleusercontent.com/-qo7SKwkoya0/VFmZrIWL0EI/AAAAAAAABkk/FctwkEcQWnk/s1600/mem.png> >>>>>> So the question is: "how to really free a DArray? Both master and >>>>>> workers!" >>>>>> >>>>>> >>>>>> On Thursday, October 16, 2014 10:18:42 PM UTC+8, Amit Murthy wrote: >>>>>>> >>>>>>> meant to say "all references from all processes are removed" >>>>>>> >>>>>>> On Thu, Oct 16, 2014 at 7:47 PM, Amit Murthy <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes, it is gc'ed just like any other object, when all references to >>>>>>>> it from any process is removed. >>>>>>>> >>>>>>>> On Thu, Oct 16, 2014 at 2:59 PM, Seth Yuan <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Now, I know by reading the documentation how a DArray is created, >>>>>>>>> but I'm not sure when a DArray is deallocated. >>>>>>>>> >>>>>>>>> Is it GC able when the master looses all references of it? Does >>>>>>>>> somebody know the answer to this question? >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>> >
