Thanks for all the help. It was the swap file size that was causing the problem. I did not realise that the swap file had such a limited size. Ubuntu makes it very small by default so I installed it again using custom setting for the partitions. I will write a step by step of how to get j64 working on the mac hopefully by the weekend.
On 12/9/08, Gilles Kirouac <[EMAIL PROTECTED]> wrote: > You can get the exact amount of the maximum virtual address space (VAS) > using command ulimit -v at the shell. > > The result is likely related to the size of the available swap space > (file or partition) set when VMWare installed the virtual machine. > > ~ Gilles > > ---------- Original Message ----------- > From: "Eric Iverson" <[EMAIL PROTECTED]> > To: "Programming forum" <[email protected]> > Sent: Mon, 8 Dec 2008 22:02:07 -0500 > Subject: Re: [Jprogramming] Large mapped files and VAS again > >> Forget about mapped files for a bit. What you need to figure out is >> what to do to get newname=: i. bigsize to work (that is, not give >> out of memory). >> >> You need to make your page file big enough and any limits on memory >> allocation big enough to allow that sentence to run. Your >> configuration is complicated so this may take a bit of digging. You >> need to dig into unbuntu, vmware, osx. There are likely required >> steps at every level. >> >> It might be interesting to first determine what the current limit >> is. Write a loop that does something like: >> >> erase 'newname' >> newname=: i. bigsize >> bigsize=: bigsize + more >> >> When you get the out of memory error, bigsize will tell you how much >> the layers of OS allow you to allocate. Then you can start the >> search of how to increase that limit in the OS. This is an issue of >> the OS, not J. >> >> On Mon, Dec 8, 2008 at 4:32 PM, Matthew Brand >> <[EMAIL PROTECTED]>wrote: >> >> > Hi Eric, >> > >> > On Mon, Dec 8, 2008 at 2:58 PM, Eric Iverson <[EMAIL PROTECTED]> >> > wrote: >> > > RAM is not the limit. THE OS makes RAM essentialy invisible to >> > applications. >> > > Actively and randomly using an allocation much larger than RAM will >> > > work, >> > > but because of swapping between page file space and RAM space >> > > performance >> > > will be much slower than if it fit into RAM. >> > > >> > > We have made some progress on your problem. We now know that it has >> > > absolutely nothing to do with mapped files. I think you will see this >> > when >> > > you realize that your expression >> > > newname =: i. bigsize >> > > fails with an out of memory error. >> > > >> > >> > I agree, the mapped files are working perfectly. >> > >> > > This means the OS has refused to allocate this amount of memory. This >> > > allocation has nothing to do with file mapping. This memory would be >> > > be >> > > backed by the page file. The OS can also have limits on the largest >> > > allocation it will allow. This depends on the OS. Are you running >> > windows? >> > > You need to make sure that your page file (settings...) is large >> > > enough >> > for >> > > these kinds of allocations. >> > >> > I am using Ubunto 64-bit as a virtual machine in VMWare Fusion on OSX >> > 10.5.5 with 2Gb RAM on a MacBook Pro. >> > >> > > >> > > That said, you may not want to populate your mapped file with J >> > expressions >> > > that require being first created entirely in page file space. >> > > >> > > Again, I suggest that you do the labs, if you have not already done >> > > so. >> > > >> > >> > I have looked at the two labs and cannot find any examples of how to >> > populate a mapped noun which is larger than the RAM other than >> > nounname =. tempvariable. In-place amend will suffer equally because I >> > would have to generate the data and the indices. >> > >> > Is there another way to populate the mapped noun? >> > >> > > Also, I suggest that you play and experiment, with smaller (say 1 or 2 >> > gig >> > > size arrays) before scaling up so much. >> > >> > I have done lots of testing with small data sets which work and am at >> > the stage where I need to run on the whole data set. I can do the job >> > in manageable chucks and get a result out (it is messy and slow but >> > works), but it would be much neater if the memory management was done >> > in the background, e.g. with the swap file. >> > >> > Thanks, >> > Matthew. >> > ---------------------------------------------------------------------- >> > For information about J forums see http://www.jsoftware.com/forums.htm >> > >> ---------------------------------------------------------------------- >> For information about J forums see http://www.jsoftware.com/forums.htm > ------- End of Original Message ------- > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
