mmap consumes virtual memory. Presumably that is counted against the
process's limit. You've got a cap of 4gb total allowed according to that
ulimit output.
On Sat, Apr 11, 2015 at 5:31 PM Marc Williams <[email protected]>
wrote:
> Yes this must be the problem. If I try running locally accessing the files
> on the server through sshfs, everything seems to work fine. Would be
> interested to know what the problem is though? And if there are any
> workarounds?
>
> Thanks
> Marc
>
> On Saturday, 11 April 2015 21:34:04 UTC+1, Marc Williams wrote:
>>
>> I'm working on a managed cluster that runs scientific lunix. I get the
>> following with ulimit -a when I've requested a session with 4gb:
>>
>> core file size (blocks, -c) unlimited
>>
>> data seg size (kbytes, -d) unlimited
>>
>> scheduling priority (-e) 0
>>
>> file size (blocks, -f) unlimited
>>
>> pending signals (-i) 191971
>>
>> max locked memory (kbytes, -l) 137435659880
>>
>> max memory size (kbytes, -m) unlimited
>>
>> open files (-n) 1024
>>
>> pipe size (512 bytes, -p) 8
>>
>> POSIX message queues (bytes, -q) 819200
>>
>> real-time priority (-r) 0
>>
>> stack size (kbytes, -s) unlimited
>>
>> cpu time (seconds, -t) unlimited
>>
>> max user processes (-u) 191971
>>
>> virtual memory (kbytes, -v) 4194304
>>
>> file locks (-x) unlimited
>> Thanks
>> Marc
>>
>> On Saturday, 11 April 2015 21:01:20 UTC+1, Tim Holy wrote:
>>>
>>> I'll bet you're working within a constrained environment. If you're on a
>>> unix
>>> platform, what does 'ulimit -a' say?
>>>
>>> Best,
>>> --Tim
>>>
>>> On Saturday, April 11, 2015 12:04:03 PM Marc Williams wrote:
>>> > So I get the error when I call mmap_array() as follows:
>>> >
>>> > s = open("matrixfile.bin")
>>> >
>>> > m = read(s, Float64)
>>> >
>>> > weight = mmap_array(Float64, (int64(m),int64(m)), s)
>>> >
>>> > close(s)
>>> >
>>> > When my "matrixfile.bin" is small everything works fine, but when I
>>> get to
>>> > a stage where the file size is similar to the amount of RAM available
>>> I get
>>> > the following error:
>>> >
>>> > ERROR: memory mapping failed: Cannot allocate memory
>>> >
>>> > in mmap at mmap.jl:35
>>> >
>>> > in mmap_array at mmap.jl:110
>>> >
>>> > in readinfile at none:4
>>> >
>>> > On Friday, 10 April 2015 15:12:14 UTC+1, tshort wrote:
>>> > > More information would help, especially a concise reproducible
>>> example.
>>> > >
>>> > > On Fri, Apr 10, 2015 at 8:00 AM, Marc Williams <[email protected]
>>> > >
>>> > > <javascript:>> wrote:
>>> > >> Hi,
>>> > >>
>>> > >> I'm doing some analysis where I need to compute some large matrices
>>> (up
>>> > >> to about 50,000 X 50,000) so I quickly run out of RAM, it seems
>>> like
>>> > >> using
>>> > >> a memory mapped array could be a useful approach so I compute the
>>> > >> matrices
>>> > >> and save them to a binary file and then read them in using
>>> mmap_array().
>>> > >> For the smaller matrices everything works fine but when the size of
>>> the
>>> > >> binary file is greater than the amount of RAM available I get the
>>> > >> following
>>> > >> error:
>>> > >>
>>> > >> ERROR: memory mapping failed: Cannot allocate memory
>>> > >>
>>> > >> Once I've read the matrices in using mmap_array() I do some basic
>>> > >> calculation like computing the sum over all the elements, the sum
>>> over
>>> > >> rows
>>> > >> and I access every element a couple of times.
>>> > >>
>>> > >> I've not used memory mapping before so am I using it in the right
>>> way and
>>> > >> is there anything I'm missing that I need to do to make this a
>>> solution
>>> > >> to
>>> > >> my RAM issue?
>>> > >>
>>> > >> Many Thanks
>>> > >> Marc
>>>
>>>
