I haven't tested this myself, but is there any chance that a reference to the
SharedArray survives in one of the workers? For example, if you did a
rr = put!(RemoteRef(pworker), SA)
then you'd have to do the same tricks in the worker that you did in the driver
process.
But if you've checked the obvious candidates, you might as well file an issue.
It would be helpful if you can produce a small testcase that reproduces the
problem.
--Tim
On Monday, May 19, 2014 10:31:41 AM Gustavo Camilo wrote:
> Hi all,
>
> I've run across a memory problem that I want to know if I'm doing wrong
> before I open an issue:
>
> In the function I optimize I run a large procedure that calls C code
> repeatedly and I end up having a huge (40^5 Float64) matrix, which I want
> to be able to use in a function I pmap, so I run this code to compute and
> then copy into a shared array:
>
>
> capitalsRaw = findZeroDividendsInC(privModel)
> capitals =
> SharedArray(Float64,(length(privModel.vW),3,length(privModel.vK),length(priv
> Model.vQ),
> length(privModel.vQ),length(privModel.vQ)),init=false,pids=workers());
> copy!(capitals,capitalsRaw)
>
> I do some work with this, and then, since my memory usage was increasing, I
> release these arrays like it says in the Performance section of the manual:
>
>
> capitals = 0
> capitalsRaw = 0
> gc()
>
>
> However, after the function has been called a bunch of times:
>
> exception on
> From worker 17: System max size of single shmem segment(MB) : 65536
> From worker 17: System max size of all shmem segments(MB) : 16777216
> From worker 17: Requested size(MB) : 1221
> From worker 17: Please ensure requested size is within system limits.
> From worker 17: If not, increase system limits and try again.
> ERROR: memory mapping failed: Cannot allocate memory
> in systemerror at error.jl:39
> in mmap at mmap.jl:32
> in mmap_array at mmap.jl:107
> in shm_mmap_array at sharedarray.jl:357
> in anonymous at sharedarray.jl:59
> in anonymous at multi.jl:840
> in run_work_thunk at multi.jl:613
> in run_work_thunk at multi.jl:622
> in anonymous at task.jl:6
>
>
> Shouldn't releasing the shared arrays by setting them to zero and calling
> gc() have cleaned up my usage? I don't think there is any other leak in the
> code, and the error is from the shared memory array allocator always, not
> for any other part of the code.
>
> Thanks for any help!
> Gustavo
