Hi Eric, On Mon, Dec 8, 2008 at 2:58 PM, Eric Iverson <[EMAIL PROTECTED]> wrote: > RAM is not the limit. THE OS makes RAM essentialy invisible to applications. > Actively and randomly using an allocation much larger than RAM will work, > but because of swapping between page file space and RAM space performance > will be much slower than if it fit into RAM. > > We have made some progress on your problem. We now know that it has > absolutely nothing to do with mapped files. I think you will see this when > you realize that your expression > newname =: i. bigsize > fails with an out of memory error. >
I agree, the mapped files are working perfectly. > This means the OS has refused to allocate this amount of memory. This > allocation has nothing to do with file mapping. This memory would be be > backed by the page file. The OS can also have limits on the largest > allocation it will allow. This depends on the OS. Are you running windows? > You need to make sure that your page file (settings...) is large enough for > these kinds of allocations. I am using Ubunto 64-bit as a virtual machine in VMWare Fusion on OSX 10.5.5 with 2Gb RAM on a MacBook Pro. > > That said, you may not want to populate your mapped file with J expressions > that require being first created entirely in page file space. > > Again, I suggest that you do the labs, if you have not already done so. > I have looked at the two labs and cannot find any examples of how to populate a mapped noun which is larger than the RAM other than nounname =. tempvariable. In-place amend will suffer equally because I would have to generate the data and the indices. Is there another way to populate the mapped noun? > Also, I suggest that you play and experiment, with smaller (say 1 or 2 gig > size arrays) before scaling up so much. I have done lots of testing with small data sets which work and am at the stage where I need to run on the whole data set. I can do the job in manageable chucks and get a result out (it is messy and slow but works), but it would be much neater if the memory management was done in the background, e.g. with the swap file. Thanks, Matthew. ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
