Your concern about the OS capability to do memory management (or show info about it) is misleading people!
I did the same experiment on a CentOS 6.5, and the memory usage shown by 'top' is: <https://lh5.googleusercontent.com/-1kr66hoaQJ0/VFrYDJv-QLI/AAAAAAAABlE/jZEcQQKhAgA/s1600/mem.png> As you can see, the exact problem is been duplicated on a Linux. On Wednesday, November 5, 2014 7:38:25 PM UTC+8, Stefan Karpinski wrote: > > This looks like an OS X screenshot – do processes on OS X ever relinquish > memory back to the kernel? I'm not sure that they do. > > On Wed, Nov 5, 2014 at 9:47 AM, Amit Murthy <[email protected] > <javascript:>> wrote: > >> Could you try: >> >> @everywhere gc() >> @everywhere Base.flush_gc_msgs() >> @everywhere gc() >> >> >> >> On Wed, Nov 5, 2014 at 2:07 PM, Seth Yuan <[email protected] >> <javascript:>> wrote: >> >>> Hi Amit, >>> >>> Thanks for the response, I tried Base.flush_gc_msgs(), the result is >>> shown below: >>> >>> >>> <https://lh5.googleusercontent.com/-bEVLUYu-DPk/VFnhFUh28CI/AAAAAAAABk0/WNl3ukSetUk/s1600/mem.png> >>> It did free some memory on some workers, but not quite. I think the >>> acceptable number for workers would be around 100MB (like the master is). >>> >>> So what's wrong with DArrays? >>> >>> >>> On Wednesday, November 5, 2014 2:32:42 PM UTC+8, Amit Murthy wrote: >>>> >>>> Distributed objects are currently gc'ed asynchronously. >>>> >>>> You may force this using an internal function flush_gc_msgs like this: >>>> >>>> @everywhere Base.flush_gc_msgs() >>>> @everywhere gc() >>>> >>>> >>>> The above is an internal function and is not meant for general use. It >>>> will be great if you could open an issue on github requesting a feature to >>>> force gc of distributed objects. >>>> >>>> >>>> >>>> On Wed, Nov 5, 2014 at 9:01 AM, Seth Yuan <[email protected]> wrote: >>>> >>>>> I did a test on deallocations, and the behavior is not what I expected. >>>>> >>>>> The code I ran is: >>>>> >>>>> @everywhere using Ops >>>>> >>>>> function test_allocation() >>>>> a = drand(10000, 10000) >>>>> b = 2a >>>>> readline() >>>>> end >>>>> >>>>> test_allocation() >>>>> @everywhere gc() >>>>> readline() >>>>> >>>>> Ops module defined the '*' operator (where a new DArray is created). I >>>>> can see that master's memory went down after the gc() call, but the >>>>> workers' memory remained unchanged. >>>>> >>>>> >>>>> <https://lh5.googleusercontent.com/-qo7SKwkoya0/VFmZrIWL0EI/AAAAAAAABkk/FctwkEcQWnk/s1600/mem.png> >>>>> So the question is: "how to really free a DArray? Both master and >>>>> workers!" >>>>> >>>>> >>>>> On Thursday, October 16, 2014 10:18:42 PM UTC+8, Amit Murthy wrote: >>>>>> >>>>>> meant to say "all references from all processes are removed" >>>>>> >>>>>> On Thu, Oct 16, 2014 at 7:47 PM, Amit Murthy <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Yes, it is gc'ed just like any other object, when all references to >>>>>>> it from any process is removed. >>>>>>> >>>>>>> On Thu, Oct 16, 2014 at 2:59 PM, Seth Yuan <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Now, I know by reading the documentation how a DArray is created, >>>>>>>> but I'm not sure when a DArray is deallocated. >>>>>>>> >>>>>>>> Is it GC able when the master looses all references of it? Does >>>>>>>> somebody know the answer to this question? >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >> >
