You can get the exact amount of the maximum virtual address space (VAS) using command ulimit -v at the shell.
The result is likely related to the size of the available swap space (file or partition) set when VMWare installed the virtual machine. ~ Gilles ---------- Original Message ----------- From: "Eric Iverson" <[EMAIL PROTECTED]> To: "Programming forum" <[email protected]> Sent: Mon, 8 Dec 2008 22:02:07 -0500 Subject: Re: [Jprogramming] Large mapped files and VAS again > Forget about mapped files for a bit. What you need to figure out is > what to do to get newname=: i. bigsize to work (that is, not give > out of memory). > > You need to make your page file big enough and any limits on memory > allocation big enough to allow that sentence to run. Your > configuration is complicated so this may take a bit of digging. You > need to dig into unbuntu, vmware, osx. There are likely required > steps at every level. > > It might be interesting to first determine what the current limit > is. Write a loop that does something like: > > erase 'newname' > newname=: i. bigsize > bigsize=: bigsize + more > > When you get the out of memory error, bigsize will tell you how much > the layers of OS allow you to allocate. Then you can start the > search of how to increase that limit in the OS. This is an issue of > the OS, not J. > > On Mon, Dec 8, 2008 at 4:32 PM, Matthew Brand <[EMAIL PROTECTED]>wrote: > > > Hi Eric, > > > > On Mon, Dec 8, 2008 at 2:58 PM, Eric Iverson <[EMAIL PROTECTED]> > > wrote: > > > RAM is not the limit. THE OS makes RAM essentialy invisible to > > applications. > > > Actively and randomly using an allocation much larger than RAM will work, > > > but because of swapping between page file space and RAM space performance > > > will be much slower than if it fit into RAM. > > > > > > We have made some progress on your problem. We now know that it has > > > absolutely nothing to do with mapped files. I think you will see this > > when > > > you realize that your expression > > > newname =: i. bigsize > > > fails with an out of memory error. > > > > > > > I agree, the mapped files are working perfectly. > > > > > This means the OS has refused to allocate this amount of memory. This > > > allocation has nothing to do with file mapping. This memory would be be > > > backed by the page file. The OS can also have limits on the largest > > > allocation it will allow. This depends on the OS. Are you running > > windows? > > > You need to make sure that your page file (settings...) is large enough > > for > > > these kinds of allocations. > > > > I am using Ubunto 64-bit as a virtual machine in VMWare Fusion on OSX > > 10.5.5 with 2Gb RAM on a MacBook Pro. > > > > > > > > That said, you may not want to populate your mapped file with J > > expressions > > > that require being first created entirely in page file space. > > > > > > Again, I suggest that you do the labs, if you have not already done so. > > > > > > > I have looked at the two labs and cannot find any examples of how to > > populate a mapped noun which is larger than the RAM other than > > nounname =. tempvariable. In-place amend will suffer equally because I > > would have to generate the data and the indices. > > > > Is there another way to populate the mapped noun? > > > > > Also, I suggest that you play and experiment, with smaller (say 1 or 2 > > gig > > > size arrays) before scaling up so much. > > > > I have done lots of testing with small data sets which work and am at > > the stage where I need to run on the whole data set. I can do the job > > in manageable chucks and get a result out (it is messy and slow but > > works), but it would be much neater if the memory management was done > > in the background, e.g. with the swap file. > > > > Thanks, > > Matthew. > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ------- End of Original Message ------- ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
