Forget about mapped files for a bit. What you need to figure out is what to do to get newname=: i. bigsize to work (that is, not give out of memory).
You need to make your page file big enough and any limits on memory allocation big enough to allow that sentence to run. Your configuration is complicated so this may take a bit of digging. You need to dig into unbuntu, vmware, osx. There are likely required steps at every level. It might be interesting to first determine what the current limit is. Write a loop that does something like: erase 'newname' newname=: i. bigsize bigsize=: bigsize + more When you get the out of memory error, bigsize will tell you how much the layers of OS allow you to allocate. Then you can start the search of how to increase that limit in the OS. This is an issue of the OS, not J. On Mon, Dec 8, 2008 at 4:32 PM, Matthew Brand <[EMAIL PROTECTED]>wrote: > Hi Eric, > > On Mon, Dec 8, 2008 at 2:58 PM, Eric Iverson <[EMAIL PROTECTED]> > wrote: > > RAM is not the limit. THE OS makes RAM essentialy invisible to > applications. > > Actively and randomly using an allocation much larger than RAM will work, > > but because of swapping between page file space and RAM space performance > > will be much slower than if it fit into RAM. > > > > We have made some progress on your problem. We now know that it has > > absolutely nothing to do with mapped files. I think you will see this > when > > you realize that your expression > > newname =: i. bigsize > > fails with an out of memory error. > > > > I agree, the mapped files are working perfectly. > > > This means the OS has refused to allocate this amount of memory. This > > allocation has nothing to do with file mapping. This memory would be be > > backed by the page file. The OS can also have limits on the largest > > allocation it will allow. This depends on the OS. Are you running > windows? > > You need to make sure that your page file (settings...) is large enough > for > > these kinds of allocations. > > I am using Ubunto 64-bit as a virtual machine in VMWare Fusion on OSX > 10.5.5 with 2Gb RAM on a MacBook Pro. > > > > > That said, you may not want to populate your mapped file with J > expressions > > that require being first created entirely in page file space. > > > > Again, I suggest that you do the labs, if you have not already done so. > > > > I have looked at the two labs and cannot find any examples of how to > populate a mapped noun which is larger than the RAM other than > nounname =. tempvariable. In-place amend will suffer equally because I > would have to generate the data and the indices. > > Is there another way to populate the mapped noun? > > > Also, I suggest that you play and experiment, with smaller (say 1 or 2 > gig > > size arrays) before scaling up so much. > > I have done lots of testing with small data sets which work and am at > the stage where I need to run on the whole data set. I can do the job > in manageable chucks and get a result out (it is messy and slow but > works), but it would be much neater if the memory management was done > in the background, e.g. with the swap file. > > Thanks, > Matthew. > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
