Has anybody been actively using mapped files greater than the RAM size in a past or present project?
The code below successfully creates a mapped file with size 64GB, much larger than my RAM. However, if I want to fill the file/noun with say (i.size) or (size $ 'abc') then "out of memory" results because J tries to evaluate those in RAM before pushing the result into the mapped file. Is there a way to overcome this type of problem other than writing messy code to break the varname =. i.size problem into smaller pieces? Note 'does not work' load 'jmf' size =. <. 2^36 NB. size of file 64GB. filename =. jpath '~temp\big.jmf' createjmf_jmf_ filename;size map_jmf_ 'varname';filename varname =. i.size NB. this line gives error "out of memory" unmap_jmf_ 'varname' ) When J creates its temporary variables does it explicitly request that the memory is allocated in physical RAM rather than anywhere else in the VAS? Is there a configuration option to allow all variables to be created anywhere in the VAS rather than only in physical RAM? I don't understand how J or any other application can limit itself to physical RAM. My understanding, which is likely erroneous, is: that the OS only shows the application the VAS which has 8 %~ 2^64 locations of size 8 bytes each; that an application makes requests to the OS for working space to be allocated; that the OS returns a pointer to a working space it reserves in the VAS; that the OS then handles, in the background, all of the mapping between the VAS segment allocated to the application and the actual memory addresses in the physical storage devices that are utilized (RAM and hard disk); that it is entirely up to the OS whether a given VAS address is mapped to a hard disk location or a physical RAM location; that the OS may alter the mapping dynamically especially if the requested working space does not fit into the RAM all at once, i.e. it might use swap files. I guess my question boils down to: If the above is correct then why does J ever return "out of memory" in a 64-bit VAS, or what is incorrect? Thanks, Matthew. ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
