RAM is not the limit. THE OS makes RAM essentialy invisible to applications. Actively and randomly using an allocation much larger than RAM will work, but because of swapping between page file space and RAM space performance will be much slower than if it fit into RAM.
We have made some progress on your problem. We now know that it has absolutely nothing to do with mapped files. I think you will see this when you realize that your expression newname =: i. bigsize fails with an out of memory error. This means the OS has refused to allocate this amount of memory. This allocation has nothing to do with file mapping. This memory would be be backed by the page file. The OS can also have limits on the largest allocation it will allow. This depends on the OS. Are you running windows? You need to make sure that your page file (settings...) is large enough for these kinds of allocations. That said, you may not want to populate your mapped file with J expressions that require being first created entirely in page file space. Again, I suggest that you do the labs, if you have not already done so. Also, I suggest that you play and experiment, with smaller (say 1 or 2 gig size arrays) before scaling up so much. On Mon, Dec 8, 2008 at 7:23 AM, Matthew Brand <[EMAIL PROTECTED]>wrote: > Has anybody been actively using mapped files greater than the RAM size > in a past or present project? > > The code below successfully creates a mapped file with size 64GB, much > larger than my RAM. However, if I want to fill the file/noun with say > (i.size) or (size $ 'abc') then "out of memory" results because J > tries to evaluate those in RAM before pushing the result into the > mapped file. > > Is there a way to overcome this type of problem other than writing > messy code to break the varname =. i.size problem into smaller pieces? > > Note 'does not work' > load 'jmf' > size =. <. 2^36 NB. size of file 64GB. > filename =. jpath '~temp\big.jmf' > createjmf_jmf_ filename;size > map_jmf_ 'varname';filename > varname =. i.size NB. this line gives error "out of memory" > unmap_jmf_ 'varname' > ) > > When J creates its temporary variables does it explicitly request that > the memory is allocated in physical RAM rather than anywhere else in > the VAS? Is there a configuration option to allow all variables to be > created anywhere in the VAS rather than only in physical RAM? > > I don't understand how J or any other application can limit itself to > physical RAM. My understanding, which is likely erroneous, is: > that the OS only shows the application the VAS which has 8 %~ 2^64 > locations of size 8 bytes each; > that an application makes requests to the OS for working space to be > allocated; > that the OS returns a pointer to a working space it reserves in the VAS; > that the OS then handles, in the background, all of the mapping > between the VAS segment allocated to the application and the actual > memory addresses in the physical storage devices that are utilized > (RAM and hard disk); > that it is entirely up to the OS whether a given VAS address is mapped > to a hard disk location or a physical RAM location; > that the OS may alter the mapping dynamically especially if the > requested working space does not fit into the RAM all at once, i.e. it > might use swap files. > > I guess my question boils down to: > If the above is correct then why does J ever return "out of memory" in > a 64-bit VAS, or what is incorrect? > > Thanks, > Matthew. > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
