Email sent by Matthew can not be decoded correctly by J forum online. May be Matthew should reconfigure his email client. Hopefully this email is now in the correct format.
On Sat, 06 Dec 2008, Eric Iverson wrote: > If you have not already studied them in detail I suggest you go through the > labs 'Mapped Files' and 'Mapped Files Database'. > > RAM is not a limit to file mapping. And page file size is also not a limit > to file mapping (you don't need file space in the page file because the file > itself is the required page space (for swapping as required between RAM and > disk). > > What platform are you using? > > There were some problems with release code for file mapping in J64 at some > stage. You should be sure to have the latest JAL update installed. > > If the total space in your files exceeds the VAS available to your > application then you will need to map/unmap as required. > > A dead simple example of your problem would be easier for the forum to deal > with than a higher level overview. For example: > *** > On Windows Vista 64 with J64 > sentence(s) ... to create a junk file of size x (jmf format or just data) > sentence(s) ... fail (when I expected it to work) > *** > > > 2008/12/6 Matthew Brand <[EMAIL PROTECTED]> > > > Hi All, > > > > Sorry to go on about the VAS / mapped file issue but I am still a bit > > stuck on this. > > > > I took the paragraph below from the page: > > http://www.jsoftware.com/help/learning/28.htm > > > > "Since mapped files may be very large, further advantages are: > > > > Larger arrays are possible. Ordinarily, the size of a J array is > > limited by the size of available RAM and swap-file. Large amounts of > > additional virtual memory can be provided for the array by mapping it > > to a file. > > > > Larger files may be handled. Ordinarily, a very large file must be > > dealt with piecemeal (because of the limit on size of arrays). By > > mapping, it can be handled as a single variable." > > > > Maybe I have got my wires cross but it sounds like this might not > > actually be the case at the moment and that we can only use files and > > data that does fit into the RAM. That would be a serious flaw for a > > language designed to act quickly on huge amounts of data - but it > > sounds like it would not be too difficult to fix, it just requires a > > modification of the mapped files code? > > > > What happens in JDB when a column does not fit in RAM, will it also > > fall over? Does q/KDB suffer from this issue? > > > > The main problem I encounter using J at the moment is that I can > > express what I want to do in very aesthetically pleasing J code, but > > when I press return it runs out of memory. So I end up writing lots of > > extra code which goes very slowly and is cumbersome (maybe even > > buggy). It separates my arrays into a separate file for each column of > > what should be one large array. I end up with over 200,000 files and a > > lot of overhead. To add to the pain, the OS then kicks in and tries to > > index them... > > > > I could try to cut down the number of files by using rectangular > > slices of data for each file, but I think I am just programming around > > the problem that J is not utilizing the VAS? I don't want to go there. > > > > Can J use the VAS for variables that do not fit in the RAM? If not > > then how does it avoid it? Because I thought that the VAS usage was > > transparent to the application (i.e. J) and handled by the OS... i.e. > > the application does not know whether an "address" it refers to is > > pointing at RAM or a swap file on the disk or even if it refers to > > some other device like a sound or graphics card. > > > > I am confused about VAS, mapped files and running out of memory on > > 64-bit machines. At the moment I spend a lot of time in J essentially > > doing my own "swap files" very badly. > > > > Is it possible for somebody to make the changes to the mapped files > > code so that we can access the whole VAS? > > > > > > Thanks, > > Matthew. > > > > > > > > On Tue, Nov 18, 2008 at 4:27 PM, bill lam <[EMAIL PROTECTED]> wrote: > > > In the article appeared in Vector, online version at > > > http://www.jsoftware.com/papers/mmf.htm > > > it said: > > > Memory-mapped files can be much larger than system RAM, yet they can > > > be accessed without page thrashing. > > > > > > The content of mapped files are never duplicated in physical ram so > > > that it appeared that the restriction is that the total size of view > > > of all mapped file must fit into vas. Even the view is the entire > > > file, it should pose no problem for 64-bit system. > > > > > > I'm puzzled too. > > > > > > On Tue, 18 Nov 2008, Don Guinn wrote: > > > > > >> After reading this thread, I'm a little confused on how mapped files are > > >> really handled. Years ago I worked on CDC mainframes(205) and on them > > mapped > > >> files were nothing unusual. When a file was opened as mapped it simply > > >> became an extension of the page (swap) file and was assigned a virtual > > >> address range to cover the file. So references to the file simply were > > done > > >> as virtual addresses and were paged into and out of real memory as > > needed > > >> directly from and to the file. There were no restrictions on the size of > > >> real memory other than performance considerations and limits of the > > virtual > > >> address range. I thought that mapped files on the PC (Windows, at least) > > >> worked the same way. > > >> > > >> It seems to me that the only real problem with large mapped files in > > >> J should be the ease in which an entire file can be processed. We have > > to go > > >> to a lot of trouble to break files into pieces. Most other languages > > process > > >> files a record at a time so for them there is seldom a need of keeping > > an > > >> entire mapped file in real memory. > > >> > > >> Are you saying that mapping a file on a PC copies the entire file into > > real > > >> memory then to the swap file as needed? If so then it must make a second > > >> write to the file itself for updates. Sounds pretty awkward. > > > > > > -- > > > regards, > > > ==================================================== > > > GPG key 1024D/4434BAB3 2008-08-24 > > > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 > > > 唐詩253 崔顥 長干行二首之一 > > > 君家何處住 妾住在橫塘 停船暫借問 或恐是同鄉 > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm -- regards, ==================================================== GPG key 1024D/4434BAB3 2008-08-24 gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 唐詩002 張九齡 感遇四首之二 蘭葉春葳蕤 桂華秋皎潔 欣欣此生意 自爾為佳節 誰知林棲者 聞風坐相悅 草木有本心 何求美人折 ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
