Hi All, Sorry to go on about the VAS / mapped file issue but I am still a bit stuck on this.
I took the paragraph below from the page: http://www.jsoftware.com/help/learning/28.htm "Since mapped files may be very large, further advantages are: Larger arrays are possible. Ordinarily, the size of a J array is limited by the size of available RAM and swap-file. Large amounts of additional virtual memory can be provided for the array by mapping it to a file. Larger files may be handled. Ordinarily, a very large file must be dealt with piecemeal (because of the limit on size of arrays). By mapping, it can be handled as a single variable." Maybe I have got my wires cross but it sounds like this might not actually be the case at the moment and that we can only use files and data that does fit into the RAM. That would be a serious flaw for a language designed to act quickly on huge amounts of data - but it sounds like it would not be too difficult to fix, it just requires a modification of the mapped files code? What happens in JDB when a column does not fit in RAM, will it also fall over? Does q/KDB suffer from this issue? The main problem I encounter using J at the moment is that I can express what I want to do in very aesthetically pleasing J code, but when I press return it runs out of memory. So I end up writing lots of extra code which goes very slowly and is cumbersome (maybe even buggy). It separates my arrays into a separate file for each column of what should be one large array. I end up with over 200,000 files and a lot of overhead. To add to the pain, the OS then kicks in and tries to index them... I could try to cut down the number of files by using rectangular slices of data for each file, but I think I am just programming around the problem that J is not utilizing the VAS? I don't want to go there. Can J use the VAS for variables that do not fit in the RAM? If not then how does it avoid it? Because I thought that the VAS usage was transparent to the application (i.e. J) and handled by the OS... i.e. the application does not know whether an "address" it refers to is pointing at RAM or a swap file on the disk or even if it refers to some other device like a sound or graphics card. I am confused about VAS, mapped files and running out of memory on 64-bit machines. At the moment I spend a lot of time in J essentially doing my own "swap files" very badly. Is it possible for somebody to make the changes to the mapped files code so that we can access the whole VAS? Thanks, Matthew. On Tue, Nov 18, 2008 at 4:27 PM, bill lam <[EMAIL PROTECTED]> wrote: > In the article appeared in Vector, online version at > http://www.jsoftware.com/papers/mmf.htm > it said: > Memory-mapped files can be much larger than system RAM, yet they can > be accessed without page thrashing. > > The content of mapped files are never duplicated in physical ram so > that it appeared that the restriction is that the total size of view > of all mapped file must fit into vas. Even the view is the entire > file, it should pose no problem for 64-bit system. > > I'm puzzled too. > > On Tue, 18 Nov 2008, Don Guinn wrote: > >> After reading this thread, I'm a little confused on how mapped files are >> really handled. Years ago I worked on CDC mainframes(205) and on them mapped >> files were nothing unusual. When a file was opened as mapped it simply >> became an extension of the page (swap) file and was assigned a virtual >> address range to cover the file. So references to the file simply were done >> as virtual addresses and were paged into and out of real memory as needed >> directly from and to the file. There were no restrictions on the size of >> real memory other than performance considerations and limits of the virtual >> address range. I thought that mapped files on the PC (Windows, at least) >> worked the same way. >> >> It seems to me that the only real problem with large mapped files in >> J should be the ease in which an entire file can be processed. We have to go >> to a lot of trouble to break files into pieces. Most other languages process >> files a record at a time so for them there is seldom a need of keeping an >> entire mapped file in real memory. >> >> Are you saying that mapping a file on a PC copies the entire file into real >> memory then to the swap file as needed? If so then it must make a second >> write to the file itself for updates. Sounds pretty awkward. > > -- > regards, > ==================================================== > GPG key 1024D/4434BAB3 2008-08-24 > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 > 唐��253 崔�� �L干行二首之一 > 君家何��住 妾住在�M塘 停船�航��� 或恐是同�l > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm >
---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
