Dear all, I just posted a message to the programming forum but it appears to be just random ascii characters, which is odd because it does not look like that in my sent folder:
http://www.jsoftware.com/pipermail/programming/2008-December/013139.html This is the text of the message I wanted to posted: Hi All, Sorry to go on about the VAS / mapped file issue but I am still a bit stuck on this. I took the paragraph below from the page: http://www.jsoftware.com/help/learning/28.htm "Since mapped files may be very large, further advantages are: Larger arrays are possible. Ordinarily, the size of a J array is limited by the size of available RAM and swap-file. Large amounts of additional virtual memory can be provided for the array by mapping it to a file. Larger files may be handled. Ordinarily, a very large file must be dealt with piecemeal (because of the limit on size of arrays). By mapping, it can be handled as a single variable." Maybe I have got my wires cross but it sounds like this might not actually be the case at the moment and that we can only use files and data that does fit into the RAM. That would be a serious flaw for a language designed to act quickly on huge amounts of data - but it sounds like it would not be too difficult to fix, it just requires a modification of the mapped files code? What happens in JDB when a column does not fit in RAM, will it also fall over? Does q/KDB suffer from this issue? The main problem I encounter using J at the moment is that I can express what I want to do in very aesthetically pleasing J code, but when I press return it runs out of memory. So I end up writing lots of extra code which goes very slowly and is cumbersome (maybe even buggy). It separates my arrays into a separate file for each column of what should be one large array. I end up with over 200,000 files and a lot of overhead. To add to the pain, the OS then kicks in and tries to index them... I could try to cut down the number of files by using rectangular slices of data for each file, but I think I am just programming around the problem that J is not utilizing the VAS? I don't want to go there. Can J use the VAS for variables that do not fit in the RAM? If not then how does it avoid it? Because I thought that the VAS usage was transparent to the application (i.e. J) and handled by the OS... i.e. the application does not know whether an "address" it refers to is pointing at RAM or a swap file on the disk or even if it refers to some other device like a sound or graphics card. I am confused about VAS, mapped files and running out of memory on 64-bit machines. At the moment I spend a lot of time in J essentially doing my own "swap files" very badly. Is it possible for somebody to make the changes to the mapped files code so that we can access the whole VAS? Thanks, Matthew. ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
