If you have not already studied them in detail I suggest you go through the labs 'Mapped Files' and 'Mapped Files Database'.
RAM is not a limit to file mapping. And page file size is also not a limit to file mapping (you don't need file space in the page file because the file itself is the required page space (for swapping as required between RAM and disk). What platform are you using? There were some problems with release code for file mapping in J64 at some stage. You should be sure to have the latest JAL update installed. If the total space in your files exceeds the VAS available to your application then you will need to map/unmap as required. A dead simple example of your problem would be easier for the forum to deal with than a higher level overview. For example: *** On Windows Vista 64 with J64 sentence(s) ... to create a junk file of size x (jmf format or just data) sentence(s) ... fail (when I expected it to work) *** 2008/12/6 Matthew Brand <[EMAIL PROTECTED]> > Hi All, > > Sorry to go on about the VAS / mapped file issue but I am still a bit > stuck on this. > > I took the paragraph below from the page: > http://www.jsoftware.com/help/learning/28.htm > > "Since mapped files may be very large, further advantages are: > > Larger arrays are possible. Ordinarily, the size of a J array is > limited by the size of available RAM and swap-file. Large amounts of > additional virtual memory can be provided for the array by mapping it > to a file. > > Larger files may be handled. Ordinarily, a very large file must be > dealt with piecemeal (because of the limit on size of arrays). By > mapping, it can be handled as a single variable." > > Maybe I have got my wires cross but it sounds like this might not > actually be the case at the moment and that we can only use files and > data that does fit into the RAM. That would be a serious flaw for a > language designed to act quickly on huge amounts of data - but it > sounds like it would not be too difficult to fix, it just requires a > modification of the mapped files code? > > What happens in JDB when a column does not fit in RAM, will it also > fall over? Does q/KDB suffer from this issue? > > The main problem I encounter using J at the moment is that I can > express what I want to do in very aesthetically pleasing J code, but > when I press return it runs out of memory. So I end up writing lots of > extra code which goes very slowly and is cumbersome (maybe even > buggy). It separates my arrays into a separate file for each column of > what should be one large array. I end up with over 200,000 files and a > lot of overhead. To add to the pain, the OS then kicks in and tries to > index them... > > I could try to cut down the number of files by using rectangular > slices of data for each file, but I think I am just programming around > the problem that J is not utilizing the VAS? I don't want to go there. > > Can J use the VAS for variables that do not fit in the RAM? If not > then how does it avoid it? Because I thought that the VAS usage was > transparent to the application (i.e. J) and handled by the OS... i.e. > the application does not know whether an "address" it refers to is > pointing at RAM or a swap file on the disk or even if it refers to > some other device like a sound or graphics card. > > I am confused about VAS, mapped files and running out of memory on > 64-bit machines. At the moment I spend a lot of time in J essentially > doing my own "swap files" very badly. > > Is it possible for somebody to make the changes to the mapped files > code so that we can access the whole VAS? > > > Thanks, > Matthew. > > > > On Tue, Nov 18, 2008 at 4:27 PM, bill lam <[EMAIL PROTECTED]> wrote: > > In the article appeared in Vector, online version at > > http://www.jsoftware.com/papers/mmf.htm > > it said: > > Memory-mapped files can be much larger than system RAM, yet they can > > be accessed without page thrashing. > > > > The content of mapped files are never duplicated in physical ram so > > that it appeared that the restriction is that the total size of view > > of all mapped file must fit into vas. Even the view is the entire > > file, it should pose no problem for 64-bit system. > > > > I'm puzzled too. > > > > On Tue, 18 Nov 2008, Don Guinn wrote: > > > >> After reading this thread, I'm a little confused on how mapped files are > >> really handled. Years ago I worked on CDC mainframes(205) and on them > mapped > >> files were nothing unusual. When a file was opened as mapped it simply > >> became an extension of the page (swap) file and was assigned a virtual > >> address range to cover the file. So references to the file simply were > done > >> as virtual addresses and were paged into and out of real memory as > needed > >> directly from and to the file. There were no restrictions on the size of > >> real memory other than performance considerations and limits of the > virtual > >> address range. I thought that mapped files on the PC (Windows, at least) > >> worked the same way. > >> > >> It seems to me that the only real problem with large mapped files in > >> J should be the ease in which an entire file can be processed. We have > to go > >> to a lot of trouble to break files into pieces. Most other languages > process > >> files a record at a time so for them there is seldom a need of keeping > an > >> entire mapped file in real memory. > >> > >> Are you saying that mapping a file on a PC copies the entire file into > real > >> memory then to the swap file as needed? If so then it must make a second > >> write to the file itself for updates. Sounds pretty awkward. > > > > -- > > regards, > > ==================================================== > > GPG key 1024D/4434BAB3 2008-08-24 > > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 > > 唐��253 崔�� �L干行二首之一 > > 君家何��住 妾住在�M塘 停船�航��� 或恐是同�l > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm >
---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
